AviSynth+ thread Vol.2

[jpsdr]

Unfortunately i don't have time right now, can quick test could be this :

static void Sobel_16(const unsigned char *psrc,unsigned char *pdst,const int32_t src_pitch, const int32_t dst_pitch,
const int32_t src_height,int32_t dst_row_size, int32_t thresh,uint8_t bit_pixel)
{
const int32_t i = (dst_row_size + 3) >> 2;
const int32_t i0 = (dst_row_size + 3-2) >> 2;

dst_row_size >>= 1;
thresh <<= (bit_pixel-8);

if (aWarpSharp_Enable_AVX)
{
for (int32_t y=0; y<src_height; y++)
{
uint16_t *dst=(uint16_t *)pdst;

if (y==0) JPSDR_Sobel_16_AVX(psrc+2,pdst+2,src_pitch,y,src_height,i0,thresh);
else
{
if (y==src_height-1) JPSDR_Sobel_16_AVX(psrc,pdst,src_pitch,y,src_height,i0,thresh);
else JPSDR_Sobel_16_AVX(psrc,pdst,src_pitch,y,src_height,i,thresh);
}
dst[0]=dst[1];
dst[dst_row_size-1]=dst[dst_row_size-2];

psrc += src_pitch;
pdst += dst_pitch;
}
}
else
{
for (int32_t y=0; y<src_height; y++)
{
uint16_t *dst=(uint16_t *)pdst;

if (y==0) JPSDR_Sobel_16_SSE2(psrc+2,pdst+2,src_pitch,y,src_height,i0,thresh);
else
{
if (y==src_height-1) JPSDR_Sobel_16_SSE2(psrc,pdst,src_pitch,y,src_height,i0,thresh);
else JPSDR_Sobel_16_SSE2(psrc,pdst,src_pitch,y,src_height,i,thresh);
}
dst[0]=dst[1];
dst[dst_row_size-1]=dst[dst_row_size-2];

psrc += src_pitch;
pdst += dst_pitch;
}
}
}


You have to put threads=1 in aWarpSharp2 call.

The

dst[0]=dst[1];
dst[dst_row_size-1]=dst[dst_row_size-2];

may fill the missing pixels.

[GMJCZP]

Thanks pinterf.

With frames = 2 it improved noticeably but still doesn't even match Prefetch 1:

Log file created with: AVSMeter 3.0.9.0 (x86)
Script file: Prueba.avs
Command line switches: -log

[OS/Hardware info]
Operating system: Windows 7 (x86) Service Pack 1.0 (Build 7601)

CPU: Pentium(R) Dual-Core CPU E5800 @ 3.20GHz / Wolfdale (Core 2 Duo) 2M
MMX, SSE, SSE2, SSE3, SSSE3
2 physical cores / 2 logical cores


[Avisynth info]
VersionString: AviSynth+ 3.7.0 (r3382, 3.7, i386)
VersionNumber: 2.60
File / Product version: 3.7.0.0 / 3.7.0.0
Interface Version: 8
Multi-threading support: Yes
Avisynth.dll location: C:\Windows\system32\avisynth.dll
Avisynth.dll time stamp: 2021-01-11, 20:46:40 (UTC)
PluginDir2_5 (HKLM, x86): C:\Program Files\AviSynth+\plugins
PluginDir+ (HKLM, x86): C:\Program Files\AviSynth+\plugins+


[Clip info]
Number of frames: 162
Length (hh:mm:ss.ms): 00:00:06.757
Frame width: 640
Frame height: 480
Framerate: 23.976 (24000/1001)
Colorspace: YUV420P12
Audio channels: n/a
Audio bits/sample: n/a
Audio sample rate: n/a
Audio samples: n/a


[Runtime info]
Frames processed: 162 (0 - 161)
FPS (min | max | average): 0.971 | 107491 | 29.03
Process memory usage (max): 37 MiB
Thread count: 8
CPU usage (average): 64.9%

Time (elapsed): 00:00:05.581


[Script]

LWLibavVideoSource("Sample2.mp4")
AssumeFPS("ntsc_film")
Prefetch(2,frames=2)

I include the test with FFMS2:

Log file created with: AVSMeter 3.0.9.0 (x86)
Script file: Prueba.avs
Command line switches: -log

[OS/Hardware info]
Operating system: Windows 7 (x86) Service Pack 1.0 (Build 7601)

CPU: Pentium(R) Dual-Core CPU E5800 @ 3.20GHz / Wolfdale (Core 2 Duo) 2M
MMX, SSE, SSE2, SSE3, SSSE3
2 physical cores / 2 logical cores


[Avisynth info]
VersionString: AviSynth+ 3.7.0 (r3382, 3.7, i386)
VersionNumber: 2.60
File / Product version: 3.7.0.0 / 3.7.0.0
Interface Version: 8
Multi-threading support: Yes
Avisynth.dll location: C:\Windows\system32\avisynth.dll
Avisynth.dll time stamp: 2021-01-11, 20:46:40 (UTC)
PluginDir2_5 (HKLM, x86): C:\Program Files\AviSynth+\plugins
PluginDir+ (HKLM, x86): C:\Program Files\AviSynth+\plugins+


[Clip info]
Number of frames: 162
Length (hh:mm:ss.ms): 00:00:06.757
Frame width: 640
Frame height: 480
Framerate: 23.976 (24000/1001)
Colorspace: YUV420P16
Audio channels: n/a
Audio bits/sample: n/a
Audio sample rate: n/a
Audio samples: n/a


[Runtime info]
Frames processed: 162 (0 - 161)
FPS (min | max | average): 0.756 | 239787 | 5.697
Process memory usage (max): 35 MiB
Thread count: 7
CPU usage (average): 81.5%

Time (elapsed): 00:00:28.437


[Script]
FFVideoSource("Sample2.mp4")
AssumeFPS("ntsc_film")
Prefetch(2)

My Avs+ system is suffering of "mono-nucleosis".
I need your help.

[StainlessS]

@Wonkey,

Avisynth is a bit laxly specified, you can even have a variable with same name as a function, its not until
it tries to evaluate an expression that it can figure out what it is, and whether the result of that expression is
assigned to some variable or used in some way as an argument in another expression. It may not be used
for anyhting at all, eg just plonk a "123456" on a line by itself somewhere, with or without the double quotes.

I doubt whether avisynth script lnaguage could be properly described in Backus–Naur [used in Kernighan & Ritchie,
"The C Programming Language", at the back of the book to describe Std/ISO C]. Or in those language describing tools
derived from Unix "Lex", and the like.

Making "\" line continuation optional, is just making it even more lax than it already is, and is bound to come with a bundle of trip wires, safer to forget any changes at this stage in the life of AVS.
I guess Ben implemented the language to the point that it could get the job done, and no further.

Backus–Naur form:- https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form
The C Programming Language:- https://en.wikipedia.org/wiki/The_C_Programming_Language
Lex:- https://en.wikipedia.org/wiki/Lex_(software)
Lexx [The bestest Sci-Fi ever created, weird mix of Canadian-German humour]:- https://en.wikipedia.org/wiki/Lexx

EDIT:
plonk a "123456" on a line
Except the last line. [the entire script must evaluate to a clip]

EDIT: Fixed the Lex link, the trailing ")" of the url is wrongly appended as simple text after the remaining part of the link by the vBulletin url insertion thingy.

[jpsdr]

Ok, i've tested the following change :

static void Sobel_8(const unsigned char *psrc,unsigned char *pdst,const int32_t src_pitch, const int32_t dst_pitch,
const int32_t src_height,const int32_t dst_row_size, int32_t thresh)
{
const int32_t i = (dst_row_size-2 + 3) >> 2;

if (aWarpSharp_Enable_AVX)
{
for (int32_t y=0; y<src_height; y++)
{
JPSDR_Sobel_8_AVX(psrc+1,pdst+1,src_pitch,y,src_height,i,thresh);
pdst[0] = pdst[1];
pdst[dst_row_size-1] = pdst[dst_row_size-2];
psrc += src_pitch;
pdst += dst_pitch;
}
}
else
{
for (int32_t y=0; y<src_height; y++)
{
JPSDR_Sobel_8_SSE2(psrc+1,pdst+1,src_pitch,y,src_height,i,thresh);
pdst[0] = pdst[1];
pdst[dst_row_size-1] = pdst[dst_row_size-2];
psrc += src_pitch;
pdst += dst_pitch;
}
}
}

tested with :

a=AVISource("SP_IVTC.avi",False,"YV12").SetPlanarLegacyAlignment(True)
b=aWarpSharp2(a,chroma=3,threads=1)
c=aWarpSharp2(a,chroma=3,threads=0)

Subtract(b,c).Levels(127, 1, 129, 0, 255)

As i was hopping for, it seems it produces the same result.
So can you test if still crashing with previous change, and following change :

static void Sobel_16(const unsigned char *psrc,unsigned char *pdst,const int32_t src_pitch, const int32_t dst_pitch,
const int32_t src_height,int32_t dst_row_size, int32_t thresh,uint8_t bit_pixel)
{
const int32_t i = (dst_row_size-4 + 3) >> 2;

dst_row_size >>= 1;
thresh <<= (bit_pixel-8);

if (aWarpSharp_Enable_AVX)
{
for (int32_t y=0; y<src_height; y++)
{
uint16_t *dst=(uint16_t *)pdst;

JPSDR_Sobel_16_AVX(psrc+2,pdst+2,src_pitch,y,src_height,i,thresh);
dst[0]=dst[1];
dst[dst_row_size-1]=dst[dst_row_size-2];

psrc += src_pitch;
pdst += dst_pitch;
}
}
else
{
for (int32_t y=0; y<src_height; y++)
{
uint16_t *dst=(uint16_t *)pdst;

JPSDR_Sobel_16_SSE2(psrc+2,pdst+2,src_pitch,y,src_height,i,thresh);
dst[0]=dst[1];
dst[dst_row_size-1]=dst[dst_row_size-2];

psrc += src_pitch;
pdst += dst_pitch;
}
}
}

with the following :

aWarpSharp2(a,chroma=3,threads=1)


Edit
.... I realised too late, should have put this on my aWarpsharp thread... :(

[pinterf]

Ok, i've tested the following change :
...
with the following :

aWarpSharp2(a,chroma=3,threads=1)


Thanks, neither Sobel_8 not Sobel_16 is working (I don't have AVX in this test machine), because there is a new crash at e.g. in JPSDR_Sobel_8_SSE2
movntdq XMMWORD ptr[rsi+rdi],xmm2

I think changing the destination start by +1 (8 bit) or +2 (16 bit) will make the storage unaligned :(

Unfortunately one have to treat the left and rightmost loads specially when we want to keep the aligned access in the middle.

[jpsdr]

Argh... Forgot this. For purpose testing right not, does replacing "movntdq" with "movdqu" make it work ?
If yes, will create a specific asm function with "movdqu" for 1rst and last line.

Euh.... Why Sobel_8 worked for me on my Windows7 x86 avs2.60... .... Ah... picture was grey as expected in VDub and probably didn't notice an error message displayed in the bottom information line. If there is no pop-up crash, displayed picture was grey as expected, and i didn't realise... :(

I'll redo the test.

Edit
Indeed, didn't notice the error message in the bottom line of VDub...

Edit2
Replacing with "movdqu" solved, but if it seems to produce the same result for the 1 line pixel, it seems not for the last line pixel.
Can you confirm that with "movdqu" there is also no crash with CUDA ?

[pinterf]

I'll look into it tomorrow, this bug however is not Cuda aware build specific; pure luck if it did not cause troubles. Movnt is the streaming version of mov mnemonic and requires aligned access. Replacing it with .u (unaligned) will surely solve the problem.

[GMJCZP]

Friends, I am extremely worried, I made a great effort to buy my "new" card with a dual core processor and I have not been able to take advantage of its two cores, I feel like I have a Celeron, if someone gave a light, I am trying to update and improve all the scripts that I have posted but this situation is out of my hands.

[tormento]

Let me dream about at least MVTools2 on CUDA. :)

[Gavino]

Making "" line continuation optional, is just making it even more lax than it already is, and is bound to come with a bundle of trip wires, safer to forget any changes at this stage in the life of AVS.

In the AviSynth language, newline is a statement terminator (except when using the \ escape), but the parser can also recognise it has reached the end of a statement when the next symbol is not a valid continuation of what it already has. Therefore in most cases, the newline is not strictly necessary (though recommended for readability).
However, there are some cases where a newline is required (see this post and this one), so it is not possible to have the parser ignore them altogether.
I doubt whether avisynth script lnaguage could be properly described in Backus–Naur
Well, we do have the formal Avisynth grammar (http://avisynth.nl/index.php/Formal_AviSynth_grammar) in Extended Backus-Naur Form (EBNF) (contributed a long time ago by gzardakas and possibly slightly out-of-date).

[DJATOM]

So true...
https://i.imgur.com/S2eXJek.png

[pinterf]

So true...
https://i.imgur.com/S2eXJek.png
Nekopanda extracted and rewrote some stuff what was needed for his project. Actually KTGMC filter is the beginning of what you are searching for.
https://github.com/pinterf/AviSynthCUDAFilters/blob/master/KTGMC/MV.cpp#L5416

[DJATOM]

Interesting. To actually use that, I have to build Avs+ WIP or 3.7 will work?

[kedautinh12]

Interesting. To actually use that, I have to build Avs+ WIP or 3.7 will work?

Here had built x64 CUDA
https://drive.google.com/uc?export=download&id=1CpFdkqbNRDwtuHCtWiQ4x49ZCt8W2jkS

[DJATOM]

I already did my own build, now building boost libs since it's the only one dependency that I didn't yet resolved.

[DJATOM]

Apparently it doesn't work.
ClearAutoloadDirs()
AddAutoloadDir("C:\avsCuda\scripts")
AddAutoloadDir("C:\avsCuda\plugins")
DGSource("NCOP.dgi").onCPU()
KTGMC()
gives me Error: [KMasktoolFilterBase] CUDAГtГМБ[ГАВЁУ№Ч═В╡В─ВнВ╛В│Вв occurred while reading frame 0. while KTGMC().onCUDA() doesn't crash, but process doing nothing - it simply waiting for something.

[pinterf]

I don't remember now, have you built masktools as well? (cuda branch in my masktools2 repo)

[DJATOM]

I went through some steps of your building manual with minor changes (using cuda 11.3 and setting cuda arch 7.5 where it was set to lower version, also fixed afxres.h to windows.h in nnedi3 since it doesn't build on latest msvc).

[pinterf]

Ehh, that build manual is rather a 'log of my adventures on doing something I've never encountered before'. O.K., in a somewhat polished version, but I was happy that it worked as is for me. Anyway it can be a good start for someone who really wants to be involved in the project. (Probably not me, it requires weeks or months to have an active knowledge on it, though programming CUDA is a very interesting topic).

[pinterf]

My Avs+ system is suffering of "mono-nucleosis".
I need your help.
The problem is that you are trying to use multithreading on a simple script where is source filter is MT_SERIALIZED (just seen in the source). So it cannot be requested in a parallel way.

Such filters cannot be called again for a new frame until the previous frame is ready. There is blocking, new requests are getting into a queue. Prefetch(2) is starting threads with prefetching 4 frames in advance.

It is possible that the calls in this scenario are producing nonlinear frame access. In this case the somewhat slower execution is blocking the other frame requests, we are seeing a negative feedback.

This is not something which is debuggable easily. There is probably an 1/10000sec delay in timing conditions which results in the first out-of-sequence access to LWLibavVideoSource. Debug build is quick. But release build is getting into this state after some ten frames. When I put a simple line in the source code around the delay (cout::stdout << frame_number - really not a time consuming operation) which is writing the actual frame number to the standard output - the problem disappears.
Pretty much an the observer effect: the disturbance of an observed system by the act of observation.

I'd like to understand how it begins and how this chaotic internal state can be healed but it is not easy at all.

[tormento]

Would a moderator/contributor please start a new thread with tested CUDA filter builds?

(And perhaps AVS+ CUDA builds on top)

[real.finder]

Friends, I am extremely worried, I made a great effort to buy my "new" card with a dual core processor and I have not been able to take advantage of its two cores, I feel like I have a Celeron, if someone gave a light, I am trying to update and improve all the scripts that I have posted but this situation is out of my hands.

LWLibavVideoSource("Sample2.mp4")
Prefetch(1)
AssumeFPS("ntsc_film")
Prefetch(2) #or Prefetch(4,frames = 2)

this seems do it

edit: you can also try RequestLinear()

LWLibavVideoSource("Sample2.mp4")
RequestLinear(clim=100)
#~ Prefetch(1) you can also uncomment it
AssumeFPS("ntsc_film")
Prefetch(8)

[GMJCZP]

Thanks real.finder and pinterf. I have tried all the cocktails that have been recommended to me and I cannot beat Prefetch 1. In fact, I complicated the script and in AvsMeter with RequestLinear activated I got the following error message:

RequestLinear: Internal error (Frame not cached!).

[real.finder]

Thanks real.finder and pinterf. I have tried all the cocktails that have been recommended to me and I cannot beat Prefetch 1. In fact, I complicated the script and in AvsMeter with RequestLinear activated I got the following error message:

RequestLinear: Internal error (Frame not cached!).

did you have uptodate https://github.com/pinterf/TIVTC/releases ? it should be fixed https://github.com/pinterf/TIVTC/issues/20

[pinterf]

did you have uptodate https://github.com/pinterf/TIVTC/releases ? it should be fixed https://github.com/pinterf/TIVTC/issues/20
It is not 100%, just a trivial case was fixed. So the likelyhood you are seeing this error was decreased. There still can be circumstances in the frame request pattern when you get this error.

[GMJCZP]

I have TIVTC 1.0.26.

[pinterf]

Spent some hours on the topic. Finally here are some statistics.

Out of order frames have extreme penalty for this source filter / video file. The bigger the frame number to longer it takes.

Source filters are running in MT_SERIALIZED mode.

LWLibavVideoSource("Sample2.mp4", threads = 1, seek_threshold = 10 )
Prefetch(2)

FrameNo : time for GetFrame.
124 : 0.0127242 sec. Locktime: 1.77e-05 sec
126 : 0.0034897 sec. Locktime: 7.2e-06 sec
127 : 0.0062913 sec. Locktime: 6.5e-06 sec
128 : 0.0038575 sec. Locktime: 6.8e-06 sec
125 : 0.569717 sec. Locktime: 0.0231566 sec
129 : 0.0145048 sec. Locktime: 1.64e-05 sec
131 : 0.0042856 sec. Locktime: 8.1e-06 sec
132 : 0.0065563 sec. Locktime: 7.1e-06 sec
133 : 0.0035983 sec. Locktime: 6.6e-06 sec
130 : 0.572035 sec. Locktime: 0.0242634 sec

Lock time is a small overhead but it can be more if there are other slow GetFrames in queue

[GMJCZP]

I have also tried another video with Avisource and I notice that without Prefetch or Prefetch 1 is better than Prefetch 2 (Prefetch (4,2) is just slightly better). IMHO I still think that certain features of the E5800 collide with the Avs+ MT.

[Dogway]

Is there a way to run a for loop with floats? I tried converting to int but I think there's a limited set of supported int values.

Workaround (bisection method).

a = 0.000001
b = 0.1

a = int(a*pow(10,6))
b = int(b*pow(10,6))

function poly_beta (float a) { (10*a-10/pow(a,0.45-1)+1-4.5*a)*pow(10,6) }

a_n = a
b_n = b

for (i=a, b, 1) {
m_n = (a_n + b_n)/2
f_m_n = poly_beta(m_n)
if (poly_beta(a_n)*f_m_n < 0) {
a_n = a_n
b_n = m_n }
else if (poly_beta(b_n)*f_m_n < 0) {
a_n = m_n
b_n = b_n }
else if (f_m_n == 0) {
s_n = m_n }
else { Assert (false, "Failed") } }
s_n = ((a_n + b_n)/2 ) / pow(10,6)

[StainlessS]

For(i=1,10) {
f=i/10.0
...
}

Somethinkg like above, only way.
Mobile:

[FranceBB]

I know that it's probably not something totally important, but today I was working with a BT2020 content in PQ, I had to apply a LUT to convert to HLG that was working in Studio RGB (aka Limited Range RGB as output) and I realized that in Convert the matrix PC.2020 is missing...

So this works:

ConverttoYUV422(matrix="Rec2020")

but this doesn't:

ConverttoYUV422(matrix="PC.2020")

Of course I used AVSResize and everything went through correctly so that instead of getting the Limited range conversion twice (https://i.imgur.com/8lsww1a.png) I got the result I wanted (https://i.imgur.com/CZ6laKK.png), but still, I think it's something we should add in the core...

[pinterf]

.. realized that in Convert the matrix PC.2020 is missing...
Cruel world :)
Anyway, here is an actual snapshot build:
Avisynth+ 3.7.1 - 20210610 (https://drive.google.com/uc?export=download&id=1SRnFC53zrctCcHCbLF8hQ5AlEKud7-Sx) including XP and CUDA-aware builds
You can also find the link in the first post.
20210610 WIP
------------
- Add "PC.2020" to YUV-RGB conversion matrix set
- ColorBarsHD: use BT.709-2 for +I (Pattern 2), not BT.601
These are from the SMPTE RP 219-1:2014, but those are also on Wikipedia now: https://en.wikipedia.org/wiki/SMPTE_color_bars
Former values used BT.601 matrix coeff., which is wrong.
Also fixed Pattern 1 Green.Y to conform to SMPTE RP 219-1:2014 (133, not 134).
ColorBars: fixed studio RGB values for -I and +Q for rgb pixel types
- Speedup: Overlay mode "multiply": overlay clip is not converted to 4:4:4 internally when 420 or 422 subsampled format
(since only Y is used from that clip)
- Speedup: Overlay mode "multiply": SSE4.1 and AVX2 code (was: C only), Proper rounding in internal calculations
- Fix: ConvertAudio integer 32-to-8 bits C code garbage (regression in 3.7)
- ConvertAudio: Add direct Float from/to 8/16 conversions (C,SSE2,AVX2)
- Fix: ConvertAudio: float to 32 bit integer conversion max value glitch (regression in 3.7)
- Fix: Crash in ColorBars very first frame when followed by ResampleAudio
- Fix: frame property access from C interface (for more info see readme.txt)
- Fix: StackVertical and packed RGB formats: get audio and parity from the first and not the last clip
- RGBAdjust: analyse=true 32 bit float support
- experimental! Fix CUDA support on specific builds (apply lost-during-merge differences from Nekopanda branch), add CMake support for the option.
- Fixes for building the core as a static library

[tormento]

Avisynth+ 3.7.1 - 20210610
In dll properties I see 3.7.0 while in the previous CUDA enable version (3.7.1·3396) was 3.7.1. :)

[kedautinh12]

Thank pinterf

[GMJCZP]

pinterf, will there be good news in this snapshot regarding problems with my e5800?

[pinterf]

pinterf, will there be good news in this snapshot regarding problems with my e5800?
No, it is not specific to your e5800. I was able to reproduce it on my i7 as well, since I posted here exact timing data. It's like the source filter behaves ultra-slow when encountering out-of-sequence frame request. Cannot help with it.
I don't know if this is also related to the actual video file encoding (e.g. decoding the 100th frame requires to decode all the preceeding frames) because it seems that it becomes slooower and slooooower when the frame number increases.

[pinterf]

In dll properties I see 3.7.0 while in the previous CUDA enable version (3.7.1·3396) was 3.7.1. :)
Arrgh. Thanks. I'll replace them soon.
EDIT: the file behind the link has been replaced.

[kedautinh12]

Thanks

[tormento]

Thank pinterf
Thanks

Much better the Forum admin (if any still alive) could implement a Thanks! button. :p

Most of the times I don't thank on the Forum but in my mind only because I know the cluttering it should result.

[GMJCZP]

No, it is not specific to your e5800. I was able to reproduce it on my i7 as well, since I posted here exact timing data. It's like the source filter behaves ultra-slow when encountering out-of-sequence frame request. Cannot help with it.
I don't know if this is also related to the actual video file encoding (e.g. decoding the 100th frame requires to decode all the preceeding frames) because it seems that it becomes slooower and slooooower when the frame number increases.

See please the post 1028 (https://forum.doom9.org/showthread.php?p=1944278#post1944278).

I quote this because, Imho, it is not a source problem, but at least my e5800 is not alone in this world.

Edit: once using BeHappy I was modifying the option "Parallel jobs" to encode more than one song at a time and I noticed that some tracks were cut off, as if the program was choking on so many songs. This problem with Prefetch reminded me of it.

[kedautinh12]

See please the post 1028 (https://forum.doom9.org/showthread.php?p=1944278#post1944278).

I quote this because, Imho, it is not a source problem, but at least my e5800 is not alone in this world.

I think you need buy new cpus for modern 😂😂😂

[real.finder]

See please the post 1028 (https://forum.doom9.org/showthread.php?p=1944278#post1944278).

I quote this because, Imho, it is not a source problem, but at least my e5800 is not alone in this world.

Edit: once using BeHappy I was modifying the option "Parallel jobs" to encode more than one song at a time and I noticed that some tracks were cut off, as if the program was choking on so many songs. This problem with Prefetch reminded me of it.

I think this HDD/SSD limit

I did some tests with lossless avi

Video
ID : 0
Format : YUV
Codec ID : YUY2
Codec ID/Info : YUV 4:2:2 as for UYVY but with different component ordering within the u_int32 macropixel
Duration : 2 min 54 s
Bit rate : 165 Mb/s
Width : 720 pixels
Height : 480 pixels
Display aspect ratio : 3:2
Frame rate : 29.970 (30000/1001) FPS
Standard : NTSC
Color space : YUV
Chroma subsampling : 4:2:2
Compression mode : Lossless
Bits/(Pixel*Frame) : 15.936
Stream size : 3.36 GiB (99%)

Audio
ID : 1
Format : PCM
Format settings : Little / Signed
Codec ID : 1
Duration : 2 min 54 s
Bit rate mode : Constant
Bit rate : 1 536 kb/s
Channel(s) : 2 channels
Sampling rate : 48.0 kHz
Bit depth : 16 bits
Stream size : 32.0 MiB (1%)
Alignment : Aligned on interleaves
Interleave, duration : 10 ms (0.30 video frame)
Interleave, preload duratio : 49 ms

AVISource("output.avi")
AssumeFPS("ntsc_film")
about 135 fps


AVISource("output.avi")
RequestLinear(clim=100)
AssumeFPS("ntsc_film")
Prefetch(8)


and


AVISource("output.avi")
Prefetch(1)
AssumeFPS("ntsc_film")
Prefetch(2)


both about same as 1st above (about 135 fps)

AVISource("output.avi")
AssumeFPS("ntsc_film")
Prefetch(8)

about 99 fps

AVISource("output.avi")
Prefetch(1)
AssumeFPS("ntsc_film")
Prefetch(8)

about 105 fps

AVISource("output.avi")
AssumeFPS("ntsc_film")
Prefetch(2)

about 132 fps

[Boulder]

I think this issue is the same I ran into with the Avisynth version of Zopti some time ago.
https://forum.doom9.org/showthread.php?p=1940286#post1940286

[GMJCZP]

real.finder:

As I have two HDDs operating I changed the location of the video source (from a WD1600AAJS, the system one, to a WD10EZEX) to check what you say and here are the results of some tests:

[Script]

LWLibavVideoSource("video.mp4")
a=trim(0,6)
b=trim(138,0)
a+b
trim (0,1500)
RescueFrame("C03_187.bmp",187)
RescueFrame("C03_1171.bmp",1171)
RescueFrame("C03_1390.bmp",1390)
Small_Deflicker(preset=2,rep=true,cnr=false)
Prefetch(1)

[Runtime info]
Frames processed: 1501 (0 - 1500)
FPS (min | max | average): 10.19 | 222658 | 40.73
Process memory usage (max): 73 MiB
Thread count: 7
CPU usage (average): 77.1%

Time (elapsed): 00:00:36.849


With Prefetch(2):

[Runtime info]
Frames processed: 1501 (0 - 1500)
FPS (min | max | average): 12.77 | 101.3 | 50.47
Process memory usage (max): 80 MiB
Thread count: 8
CPU usage (average): 95.5%

Time (elapsed): 00:00:29.743


With Prefetch(4,2):

[Runtime info]
Frames processed: 1501 (0 - 1500)
FPS (min | max | average): 33.49 | 91.06 | 51.89
Process memory usage (max): 76 MiB
Thread count: 10
CPU usage (average): 94.9%

Time (elapsed): 00:00:28.925

It could also be that this HDD, being the latter faster, has helped in the improvement. But if you comment that there is a limit, how could this be solved? According to Crystal DiskInfo both HDDs are in good condition.

Edit: To clarify more, the 160 GB HDD has two partitions, C and D, in D is the video source and I copied it to the other HDD for testing.

[GMJCZP]

I think this issue is the same I ran into with the Avisynth version of Zopti some time ago.
https://forum.doom9.org/showthread.php?p=1940286#post1940286

Thanks for the information.
That is why I have emphasized post #1028.

[real.finder]

It could also be that this HDD, being the latter faster, has helped in the improvement. But if you comment that there is a limit, how could this be solved? According to Crystal DiskInfo both HDDs are in good condition.

It depends on many things like where is the file stored on the hard disk media (near the edge or near the center) and also fragment

anyway, I think Prefetch need add some Frame Cache method for source call filters to avoid this problem (Depending on threads)

[real.finder]

[Script]

LWLibavVideoSource("video.mp4")
a=trim(0,6)
b=trim(138,0)
a+b
trim (0,1500)
RescueFrame("C03_187.bmp",187)
RescueFrame("C03_1171.bmp",1171)
RescueFrame("C03_1390.bmp",1390)
Small_Deflicker(preset=2,rep=true,cnr=false)
Prefetch(1)



can you test with

LWLibavVideoSource("video.mp4")
Prefetch(1)
a=trim(0,6)
b=trim(138,0)
a+b
trim (0,1500)
RescueFrame("C03_187.bmp",187)
RescueFrame("C03_1171.bmp",1171)
RescueFrame("C03_1390.bmp",1390)
Small_Deflicker(preset=2,rep=true,cnr=false)
Prefetch(2)

or better

LWLibavVideoSource("video.mp4")
a=trim(0,6)
b=trim(138,0)
a+b
trim (0,1500)
RescueFrame("C03_187.bmp",187)
RescueFrame("C03_1171.bmp",1171)
RescueFrame("C03_1390.bmp",1390)
Prefetch(1)
Small_Deflicker(preset=2,rep=true,cnr=false)
Prefetch(2)

since Prefetch(4,2) is less random access than Prefetch(2) because Prefetch(2) = Prefetch(2,4) (http://avisynth.nl/index.php/SetFilterMTMode#Prefetch)

[GMJCZP]

Before doing the tests that you suggest I inform you that I obtained the same results with the video source in partition D, for Prefetch 1 and 2, so that, with respect to the HDDs, we return to the starting point, anyway I have defragmented C and D.

[real.finder]

Before doing the tests that you suggest I inform you that I obtained the same results with the video source in partition D, for Prefetch 1 and 2, so that, with respect to the HDDs, we return to the starting point, anyway I have defragmented C and D.

yes, in your compressed in hevc mp4 HDD speed didn't matter much, Actually, you raised two problems, the mp4 problem resulting from nature of hevc compressing and LWLibavVideoSource as said here https://forum.doom9.org/showpost.php?p=1944004&postcount=991

and the avi problem that have a close relationship with HDD speed, but both cases have the same reason (Prefetch case random access that case slowdown in them) and both can have one fix

[Boulder]

The issue has little to do with HDD speed, at least in my case. The file is small enough to fit in the Windows read cache and the file is located on an SSD. To me it seems that all source filters suffer from it, just tested encoding my analysis clip into a lossless HEVC file and used DGSource to decode it. GPU usage was ~100% all the time, in a normal situation it's only a few percent. Besides, the Vapoursynth Zopti works much better with multithreading with the exact same file and setup.

[GMJCZP]

I clarify that by mistake I deleted the AssumeFps line from the previous results of the scripts but now I include it.

[Script]

LWLibavVideoSource("video.mp4")
Prefetch(1)
AssumeFPS("ntsc_film")
a=trim(0,6)
b=trim(138,0)
a+b
trim (0,1500)
RescueFrame("C03_187.bmp",187)
RescueFrame("C03_1171.bmp",1171)
RescueFrame("C03_1390.bmp",1390)
Small_Deflicker(preset=2,rep=true,cnr=false)
Prefetch(2)

[Runtime info]
Frames processed: 1501 (0 - 1500)
FPS (min | max | average): 30.67 | 97.94 | 51.43
Process memory usage (max): 79 MiB
Thread count: 9
CPU usage (average): 98.0%

Time (elapsed): 00:00:29.183


[Script]

LWLibavVideoSource("video.mp4")
AssumeFPS("ntsc_film")
a=trim(0,6)
b=trim(138,0)
a+b
trim (0,1500)
RescueFrame("C03_187.bmp",187)
RescueFrame("C03_1171.bmp",1171)
RescueFrame("C03_1390.bmp",1390)
Prefetch(1)
Small_Deflicker(preset=2,rep=true,cnr=false)
Prefetch(2)


[Runtime info]
Frames processed: 1501 (0 - 1500)
FPS (min | max | average): 7.330 | 389652 | 50.67
Process memory usage (max): 83 MiB
Thread count: 9
CPU usage (average): 96.4%

Time (elapsed): 00:00:29.625

[GMJCZP]

The issue has little to do with HDD speed, at least in my case. The file is small enough to fit in the Windows read cache and the file is located on an SSD. To me it seems that all source filters suffer from it, just tested encoding my analysis clip into a lossless HEVC file and used DGSource to decode it. GPU usage was ~100% all the time, in a normal situation it's only a few percent. Besides, the Vapoursynth Zopti works much better with multithreading with the exact same file and setup.

I see, that could be, a source filter problem.

[Boulder]

Out of interest, how does it work if you put something CPU intensive there, like SMDegrain with some heavier settings? Is the multithreaded version still very slow compared to a non-multithreaded run?

[GMJCZP]

Out of interest, how does it work if you put something CPU intensive there, like SMDegrain with some heavier settings? Is the multithreaded version still very slow compared to a non-multithreaded run?

What I have noticed is that the difference between Prefetch 1 and 2 is even more noticeable when the script is simpler, as you can see in the post #982. (https://forum.doom9.org/showthread.php?p=1943778#post1943778)

[Boulder]

What I have noticed is that the difference between Prefetch 1 and 2 is even more noticeable when the script is simpler, as you can see in the post #982. (https://forum.doom9.org/showthread.php?p=1943778#post1943778)

Just as I assumed. I've only seen the behaviour if the script is rather lightweight. Some MVTools-based stuff etc. already makes the multithreaded version faster, probably because the source filter will not get as frequent requests in general.

[FranceBB]

here is an actual snapshot build:
Avisynth+ 3.7.1 - 20210610 (https://drive.google.com/uc?export=download&id=1OLfGmsQro-fdrzMEi8AyfbHHLq3DkHsg) including XP and CUDA-aware builds

Wow!
This is like the fastest feature-request -> feature-implementation ever! :D
Thanks! ;)


x86 doesn't work on XP, though... :(

https://i.imgur.com/NfqrizJ.png

Here's Dependency Walker: https://i.imgur.com/ZyiXtkb.png

[real.finder]

maybe the time of winxp is over for new avs+, the avs+ rival vs even dropped win7 months ago

[pinterf]

What I have noticed is that the difference between Prefetch 1 and 2 is even more noticeable when the script is simpler, as you can see in the post #982. (https://forum.doom9.org/showthread.php?p=1943778#post1943778)
AssumeFPS does nothing with the clip, zero frame request or processing occurs, it just sets the frame rate in the VideoInfo structure.

With such a single script you are basically trying to test whether forcing multithreading to an intentionally single-threaded (MT_SERIALIZED) source filter works or not. No, it won't work, surely you will have zero speed gain even in the best case. But because of the internal timing conditions sometimes it will stuck in such a state that we'll encounter out-of-sequence frame request. This is when the linearized internal "Prefetch" is overtook by the consumer process's frame requests.

An example of this:
128 (P), 129 (P), 131 (C), 132 (C), 133(C), 130 (P)
where P=frame req by internal prefetch, C=by the external consumer (avsmeter, x264, etc))

Since the out-of-order frame generation is extremely slow, it took half _second_ on my i7 (!) and it was only around the 100th frame number, it will stuck in this state, it will never get out of this sequence (it is trying to escape but then falls back again periodically)

A possible solution can be that when such source filter detects serialized frame requests (usually this is true) and detects a jump (e.g. by two) it generates and caches inside that omitted frame.
E.g. in a frame number 5, 6, 7, 8, 10 it will still generate frame 9 and store it for future use.

MT_SERIALIZED mt mode is not something like MT_LINEARIZED (which does not exist)

I wonder what happens when you are using "Reverse" on this clip.

[pinterf]

Wow!
This is like the fastest feature-request -> feature-implementation ever! :D
Thanks! ;)


x86 doesn't work on XP, though... :(

https://i.imgur.com/NfqrizJ.png

Here's Dependency Walker: https://i.imgur.com/ZyiXtkb.png
I'm quite sure I built it with xp-on settings, anyway I'll check it for you later. This is all I can do, next Visual Studio will completely drop the feature.

[Boulder]

AssumeFPS does nothing with the clip, zero frame request or processing occurs, it just sets the frame rate in the VideoInfo structure.

With such a single script you are basically trying to test whether forcing multithreading to an intentionally single-threaded (MT_SERIALIZED) source filter works or not. No, it won't work, surely you will have zero speed gain even in the best case. But because of the internal timing conditions sometimes it will stuck in such a state that we'll encounter out-of-sequence frame request. This is when the linearized internal "Prefetch" is overtook by the consumer process's frame requests.

An example of this:
128 (P), 129 (P), 131 (C), 132 (C), 133(C), 130 (P)
where P=frame req by internal prefetch, C=by the external consumer (avsmeter, x264, etc))

Since the out-of-order frame generation is extremely slow, it took half _second_ on my i7 (!) and it was only around the 100th frame number, it will stuck in this state, it will never get out of this sequence (it is trying to escape but then falls back again periodically)

A possible solution can be that when such source filter detects serialized frame requests (usually this is true) and detects a jump (e.g. by two) it generates and caches inside that omitted frame.
E.g. in a frame number 5, 6, 7, 8, 10 it will still generate frame 9 and store it for future use.

MT_SERIALIZED mt mode is not something like MT_LINEARIZED (which does not exist)

I wonder what happens when you are using "Reverse" on this clip.

Is this behaviour something that comes from the basic Avisynth architecture, since it seems to affect all decoders? I somehow thought that Prefetch meant a read-ahead, so that the next x frames would be already decoded and in the cache whenever they are requested by the filters, and out of order requests would not be a problem.

[pinterf]

Yes, they are in the cache - optimally. Which is usually true, until it gets slowing down. When the request overtakes the prefetch process there is nothing in the cache.

[pinterf]

Avisynth is adaptive, it can detect the request pattern, it can assume and lock on a 2-4-6-8-10 pattern (delta = 2), even detects a 9-8-7-6-5 (delta = -1) negative pattern and is using the prefetch delta accordingly.

[Boulder]

Yes, they are in the cache - optimally. Which is usually true, until it gets slowing down. When the request overtakes the prefetch process there is nothing in the cache.

So a solution could be just increasing the number of frames in Prefetch? The faster the script is, the more frames you probably need to prefetch to avoid the issue?

[pinterf]

So a solution could be just increasing the number of frames in Prefetch? The faster the script is, the more frames you probably need to prefetch to avoid the issue?
This is where you have to test the effect of multiple Prefetches.
Prefetch(1,bignumber) after the source filter, and an ordinary Prefetch at the end of the script? These are only ideas if someone would like to test it.
I guess the actual benefits depend on the script. But I would not test the issue-originating simple script vs. Prefetch any further because it is just a zero-process shortcut between the consumer and an MT_SERIALIZED source.

[Boulder]

This is where you have to test the effect of multiple Prefetches.
Prefetch(1,bignumber) after the source filter, and an ordinary Prefetch at the end of the script? These are only ideas if someone would like to test it.
I guess the actual benefits depend on the script. But I would not test the issue-originating simple script vs. Prefetch any further because it is just a zero-process shortcut between the consumer and an MT_SERIALIZED source.

Thank you, I will try to find some time to test it (thanks to the football EC, not much time :D)
In my case, the script is slightly more complicated and does resizing + Expr stuff in the metrics calculation so it should be a nice case to try it on.

[pinterf]

Thank you, I will try to find some time to test it (thanks to the football EC, not much time :D)
OFF Yep, my son bought tickets for three matches, and yesterday when I did my usual run on an island in the city and saw the lurking fans and the police around the hotel when C.Ronaldo and the Portugals have their HQ during the EC.

[GMJCZP]

This is where you have to test the effect of multiple Prefetches.
Prefetch(1,bignumber) after the source filter, and an ordinary Prefetch at the end of the script? These are only ideas if someone would like to test it.
I guess the actual benefits depend on the script. But I would not test the issue-originating simple script vs. Prefetch any further because it is just a zero-process shortcut between the consumer and an MT_SERIALIZED source.

Continuing testing from post #982 (https://forum.doom9.org/showthread.php?p=1943778#post1943778), it is now with Prefetch (1,100):

Log file created with: AVSMeter 3.0.9.0 (x86)
Script file: Prueba.avs
Command line switches: -log

[OS/Hardware info]
Operating system: Windows 7 (x86) Service Pack 1.0 (Build 7601)

CPU: Pentium(R) Dual-Core CPU E5800 @ 3.20GHz / Wolfdale (Core 2 Duo) 2M
MMX, SSE, SSE2, SSE3, SSSE3
2 physical cores / 2 logical cores


[Avisynth info]
VersionString: AviSynth+ 3.7.0 (r3382, 3.7, i386)
VersionNumber: 2.60
File / Product version: 3.7.0.0 / 3.7.0.0
Interface Version: 8
Multi-threading support: Yes
Avisynth.dll location: C:\Windows\system32\avisynth.dll
Avisynth.dll time stamp: 2021-01-11, 20:46:40 (UTC)
PluginDir2_5 (HKLM, x86): C:\Program Files\AviSynth+\plugins
PluginDir+ (HKLM, x86): C:\Program Files\AviSynth+\plugins+


[Clip info]
Number of frames: 162
Length (hh:mm:ss.ms): 00:00:06.757
Frame width: 640
Frame height: 480
Framerate: 23.976 (24000/1001)
Colorspace: YUV420P12
Audio channels: n/a
Audio bits/sample: n/a
Audio sample rate: n/a
Audio samples: n/a


[Runtime info]
Frames processed: 162 (0 - 161)
FPS (min | max | average): 0.114 | 283384 | 9.439
Process memory usage (max): 174 MiB
Thread count: 9
CPU usage (average): 89.1%

Time (elapsed): 00:00:17.163


[Script]

LWLibavVideoSource("Sample2.mp4")
Prefetch(1,100)
AssumeFPS("ntsc_film")
Prefetch(2)

[FranceBB]

my son bought tickets for three matches, and yesterday when I did my usual run on an island in the city and saw the lurking fans and the police around the hotel when C.Ronaldo and the Portugals have their HQ during the EC.

Well, you might not be an overpaid football superstar, but you're one of the reason many of us are able to air stories and short documentaries about those, so you're some kind of superstar yourself eheheh
Keep up the good work and thank you for what you're doing for Avisynth (which I'm pretty sure will be used everywhere around the world to work on Euro2020 contents as well). ;)

This should cheer you up: https://i.imgur.com/XG8Axie.png

[Boulder]

OK, I tested running AVSMeter with this script. The analysis clip contains 200 frames, hence the first Prefetch call for the whole range.

With the first Prefetch enabled, everything ran smoothly until frame 196 when the processing seemed to pause. It finished by itself after a while but the average framerate went down considerably because of this.

With the first Prefetch call commented out, everything went the same way until frame 196, and then processing got completely stuck.

It would seem that at that frame 196, the deadlock kind of situation that you described happens.

orig = DGSource("c:\zopti\universe_s01e01.dgi").Prefetch(threads=1, frames=200)

b = -75/100.0 # optimize b = _n_/100.0 | -150..50 | b
c = 15/100.0 # optimize c = _n_/100.0 | -100..100 | c

downscaled_width = 1280
downscaled_height = 720

alternate = BicubicResize(orig, downscaled_width, downscaled_height, b=b, c=c).Lanczos4Resize(orig.width(),orig.height())

GMSD(alternate, orig, show=true)

# per frame logging (gmsd, time)
global delimiter = "; "
global resultFile = "perFrameResults.txt" # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"

# write "stop" at the last frame to tell the optimizer that the script has finished
global frame_count = FrameCount()

WriteFileIf(resultFile, function() {
current_frame == frame_count-1
}, function() {
gmsd = 0.0
str = ""
for (i = 0, frame_count-1) {
value = propGetFloat("_PlaneGMSD", offset = -i)
gmsd = gmsd + value
if (i>0) { str = str + e"\n" }
str = str + string(current_frame - i) + delimiter + string(value) + delimiter + string(avstimer)
}
return str + e"\nstop " + string(gmsd)
}, append=false)

Prefetch(threads=24, frames=10)

[pinterf]

Wow!
This is like the fastest feature-request -> feature-implementation ever! :D
Thanks! ;)
x86 doesn't work on XP, though... :(

Please redownload, there was a glitch in cmake build settings. I hope it works now.

[kedautinh12]

Thanks

[GMJCZP]

OK, I tested running AVSMeter with this script. The analysis clip contains 200 frames, hence the first Prefetch call for the whole range.

With the first Prefetch enabled, everything ran smoothly until frame 196 when the processing seemed to pause. It finished by itself after a while but the average framerate went down considerably because of this.

With the first Prefetch call commented out, everything went the same way until frame 196, and then processing got completely stuck.

It would seem that at that frame 196, the deadlock kind of situation that you described happens.

orig = DGSource("c:\zopti\universe_s01e01.dgi").Prefetch(threads=1, frames=200)

b = -75/100.0 # optimize b = _n_/100.0 | -150..50 | b
c = 15/100.0 # optimize c = _n_/100.0 | -100..100 | c

downscaled_width = 1280
downscaled_height = 720

alternate = BicubicResize(orig, downscaled_width, downscaled_height, b=b, c=c).Lanczos4Resize(orig.width(),orig.height())

GMSD(alternate, orig, show=true)

# per frame logging (gmsd, time)
global delimiter = "; "
global resultFile = "perFrameResults.txt" # output out1="gmsd: MIN(float)" out2="time: MIN(time) ms" file="perFrameResults.txt"

# write "stop" at the last frame to tell the optimizer that the script has finished
global frame_count = FrameCount()

WriteFileIf(resultFile, function() {
current_frame == frame_count-1
}, function() {
gmsd = 0.0
str = ""
for (i = 0, frame_count-1) {
value = propGetFloat("_PlaneGMSD", offset = -i)
gmsd = gmsd + value
if (i>0) { str = str + e"\n" }
str = str + string(current_frame - i) + delimiter + string(value) + delimiter + string(avstimer)
}
return str + e"\nstop " + string(gmsd)
}, append=false)

Prefetch(threads=24, frames=10)

Out of curiosity, can you recreate the test I did in post #982?

[kedautinh12]

Please redownload, there was a glitch in cmake build settings. I hope it works now.

File redownloaded, time modified same with old ver :confused: :confused: :confused:

[pinterf]

File redownloaded, time modified same with old ver :confused: :confused: :confused:
Files are different inside. Have you tried them on XP and they did not work? On VirtualBox or on a real PC?

[Boulder]

Out of curiosity, can you recreate the test I did in post #982?

The sample file seems to be gone. If you reupload it, I can give it a go.

[GMJCZP]

Corrected the link, Boulder.

[Boulder]

With Prefetch(1), "Script runtime is too short for meaningful measurements" :D
With Prefetch(2), definitely slower but that was expected per pinterf's explanation.


Frames processed: 162 (0 - 161)
FPS (min | max | average): 4.002 | 303033 | 152.2
Process memory usage (max): 88 MiB
Thread count: 46
CPU usage (average): 5.6%
Time (elapsed): 00:00:01.064

[GMJCZP]

With Prefetch(1), "Script runtime is too short for meaningful measurements" :D
With Prefetch(2), definitely slower but that was expected per pinterf's explanation.


Frames processed: 162 (0 - 161)
FPS (min | max | average): 4.002 | 303033 | 152.2
Process memory usage (max): 88 MiB
Thread count: 46
CPU usage (average): 5.6%
Time (elapsed): 00:00:01.064

In my case the difference is brutal. Will this have a solution?

[Boulder]

With a simple script like that, Avisynth multithreading is useless.

[GMJCZP]

With a simple script like that, Avisynth multithreading is useless.

I think this question becomes imperative, will it be possible to do this test in an equivalent way in Vapoursynth?

[Boulder]

Yes, it should be possible to test it in VS. The amount of threads can be set in the script.
I think the question is: what are you looking to multithread? With that script, you are just reading and decoding the source so multithreading it doesn't make any sense - setting the FPS just changes the clip properties. Some decoders can do internal multithreading, I think FFMS2 does at least in some cases.

[zorr]

Some decoders can do internal multithreading, I think FFMS2 does at least in some cases.

You can set the number of threads with parameter threads of FFVideoSource. On some formats that can be very useful, for example for some reason FFV1 decoding takes a huge number of threads for optimal performance.

https://i.postimg.cc/VNTxgWdw/ffv1-performance.png

You can see my ticket about it here (https://trac.ffmpeg.org/ticket/8694).

[GMJCZP]

All this is as if it takes 2 men to lift a large stone and they succeed, but if later we take the same men to move a pebble they both start arguing which of the two will do it, because four hands will get in the way to grab some of it. so small size, and all these the pebble is still on the ground waiting for the conclusion of the discussion.

[real.finder]

indeed you can't make the decode speed of source call faster with avs+ Prefetch, but the problem is MT_SERIALIZED not suitable for source call as said here https://forum.doom9.org/showthread.php?p=1944724#post1944724 maybe we should wait to see if pinterf add MT_LINEARIZED for this case

[GMJCZP]

I wonder what happens when you are using "Reverse" on this clip.

I have done new tests including Reverse(). I have left Performance data because at the beginning of the test with Prefetch 1 it was like stuck:

PREFETCH 1

[Script]

LWLibavVideoSource("Sample2.mp4")
AssumeFPS("ntsc_film")
Reverse()
Prefetch(1)


[Runtime info]
Frames processed: 162 (0 - 161)
FPS (min | max | average): 0.838 | 25.19 | 1.555
Process memory usage (max): 35 MiB
Thread count: 10
CPU usage (average): 89.3%

Time (elapsed): 00:01:44.162


[Performance data]
Frame Frames/sec Time/frame(ms) CPU(%) Threads Memory(MiB)
1 0.868 1152.605727 92.6 7 32
2 0.860 1163.214254 91.3 7 33
3 0.859 1163.807085 93.2 7 34
4 0.871 1147.891894 93.9 7 35
5 0.869 1150.160631 94.6 10 35
6 0.866 1154.832738 91.2 10 35
7 0.854 1170.400746 89.3 10 35
8 0.866 1154.755398 92.6 10 35
9 0.871 1148.168415 91.8 10 35
10 0.901 1109.587501 91.7 10 35
11 0.889 1124.645850 90.3 10 35
12 0.905 1104.950360 91.4 10 35
13 0.900 1111.412557 91.7 10 35
14 0.911 1097.943521 92.9 10 35
15 0.907 1102.785271 91.5 10 35
16 0.922 1084.399055 94.2 10 35
17 0.918 1089.068591 91.4 10 35
18 0.838 1193.176204 82.5 10 35
19 0.868 1152.698459 86.3 10 35
20 0.887 1127.143238 88.4 10 35
21 0.903 1107.059022 86.6 10 35
22 0.891 1121.785248 88.2 10 35
23 0.904 1106.133519 86.6 10 35
24 0.956 1046.068052 94.8 10 35
25 0.968 1033.054167 90.9 10 35
26 0.944 1059.561132 89.7 10 35
27 0.964 1037.391365 92.4 10 35
28 0.958 1043.598816 88.8 10 35
29 0.968 1033.550756 90.2 10 35
30 1.005 994.713861 89.1 10 35
31 0.984 1016.456976 92.3 10 35
32 0.986 1014.562660 86.2 10 35
33 0.992 1008.083794 89.2 10 35
34 1.019 980.900602 92.9 10 35
35 1.011 989.204094 91.3 10 35
36 1.028 972.877384 93.7 10 35
37 1.034 966.910498 93.5 10 35
38 1.019 981.714723 90.3 10 35
39 1.053 949.681402 91.0 10 35
40 1.032 969.054089 91.9 10 35
41 1.093 914.576683 93.2 10 35
42 1.057 946.248242 91.8 10 35
43 1.075 930.512359 91.5 10 35
44 1.082 924.285279 94.2 10 35
45 1.094 913.881815 93.1 10 35
46 1.090 917.400964 89.8 10 35
47 1.086 920.875541 89.0 10 35
48 1.098 910.738663 93.1 10 35
49 1.104 905.574080 92.4 10 35
50 1.128 886.395440 87.5 10 35
51 1.131 884.395587 93.9 10 35
52 1.158 863.869565 91.8 10 35
53 1.164 859.396950 89.3 10 35
54 1.136 880.626169 87.5 10 35
55 1.166 857.511328 91.8 10 35
56 1.189 841.396264 89.8 10 35
57 1.173 852.761928 91.8 10 35
58 1.227 815.170003 94.2 10 35
59 1.226 815.406132 87.5 10 35
60 1.238 807.885656 91.3 10 35
61 1.210 826.757565 86.8 10 35
62 1.248 801.225171 91.2 10 35
63 1.241 805.734323 91.3 10 35
64 1.288 776.559598 89.0 10 35
65 1.296 771.628287 92.9 10 35
66 1.316 759.677832 93.9 10 35
67 1.293 773.108110 90.8 10 35
68 1.287 776.797334 87.0 10 35
69 1.342 744.928782 91.7 10 35
70 1.394 717.433096 88.0 10 35
71 1.332 750.892134 87.5 10 35
72 1.292 774.140130 92.0 10 35
73 1.245 803.196160 82.4 10 35
74 1.386 721.294921 89.1 10 35
75 1.452 688.873649 93.3 10 35
76 1.470 680.095939 89.5 10 35
77 1.442 693.472318 92.2 10 35
78 1.479 676.164253 89.5 10 35
79 1.491 670.869132 91.9 10 35
80 1.524 656.118791 94.0 10 35
81 1.474 678.650105 89.5 8 35
82 1.624 615.826444 91.3 8 35
83 1.574 635.319155 92.7 8 35
84 1.612 620.326939 91.0 8 35
85 1.592 628.238126 92.7 8 35
86 1.658 602.955663 92.1 8 35
87 1.596 626.448099 86.6 8 35
88 1.703 587.105911 93.2 8 35
89 1.598 625.863260 86.3 8 35
90 1.809 552.660775 95.8 10 35
91 1.716 582.876526 93.2 10 35
92 1.783 560.723722 91.7 10 35
93 1.761 567.995272 94.4 10 35
94 1.777 562.802850 86.1 10 35
95 1.874 533.554610 92.9 10 35
96 1.911 523.241909 92.4 10 35
97 1.868 535.429382 91.4 10 35
98 1.901 526.158922 92.4 10 35
99 1.955 511.549740 92.4 10 35
100 1.983 504.195737 92.4 10 35
101 1.993 501.814429 91.5 10 35
102 2.061 485.252528 91.5 10 35
103 1.964 509.126098 86.5 10 35
104 2.116 472.494031 86.5 10 35
105 2.093 477.865189 92.6 10 35
106 2.223 449.822234 92.6 10 35
107 2.118 472.199231 89.0 10 35
108 2.132 469.079773 89.0 10 35
109 2.222 450.029769 86.4 10 35
110 2.256 443.280814 86.4 10 35
111 2.281 438.488376 86.0 10 35
112 2.394 417.628084 86.0 10 35
113 2.546 392.777096 92.3 10 35
114 2.466 405.587221 92.3 10 35
115 2.537 394.110014 92.2 10 35
116 2.553 391.752462 92.2 10 35
117 2.646 377.912209 89.8 10 35
118 2.704 369.879070 89.8 10 35
119 2.652 377.055371 91.7 10 35
120 2.836 352.659240 91.7 10 35
121 2.833 352.973294 93.3 10 35
122 2.744 364.379317 93.3 10 35
123 2.911 343.555287 87.0 10 35
124 3.111 321.448051 87.0 10 35
125 3.207 311.833707 90.0 10 35
126 3.220 310.557877 90.0 10 35
127 2.868 348.652453 84.9 7 35
128 3.472 288.004737 84.9 7 35
129 3.448 290.050482 85.1 7 35
130 3.486 286.857882 85.1 7 35
131 3.589 278.644158 87.5 7 35
132 3.765 265.623233 87.5 7 35
133 3.804 262.877840 91.2 7 35
134 3.994 250.397150 91.2 7 35
135 4.037 247.717520 91.2 7 35
136 4.243 235.660926 85.1 7 35
137 4.165 240.086035 85.1 7 35
138 4.397 227.417023 85.1 7 35
139 5.102 195.998610 86.9 7 35
140 5.007 199.715714 86.9 7 35
141 4.558 219.410535 86.9 7 35
142 5.479 182.525698 88.5 7 34
143 5.442 183.759807 88.5 7 34
144 6.010 166.388856 88.5 7 34
145 6.149 162.637108 86.4 7 35
146 5.470 182.807666 86.4 7 35
147 6.705 149.146230 86.4 7 35
148 6.671 149.894013 86.4 7 35
149 7.305 136.895232 80.0 7 34
150 8.430 118.623153 80.0 7 34
151 7.554 132.380941 80.0 7 34
152 8.378 119.355226 80.0 7 34
153 9.741 102.661154 80.0 7 34
154 10.705 93.416690 81.9 10 34
155 10.149 98.527661 81.9 10 34
156 11.597 86.229827 81.9 10 34
157 12.426 80.473730 81.9 10 34
158 13.111 76.273830 81.9 10 34
159 19.560 51.125105 81.9 10 34
160 12.225 81.800877 81.9 10 34
161 25.191 39.697282 75.8 10 34
162 24.674 40.528793 75.8 10 34

[GMJCZP]

Prefetch 2:

PREFETCH 2

[Script]

LWLibavVideoSource("Sample2.mp4")
AssumeFPS("ntsc_film")
Reverse()
Prefetch(2)


[Runtime info]
Frames processed: 162 (0 - 161)
FPS (min | max | average): 0.286 | 148439 | 1.959
Process memory usage (max): 40 MiB
Thread count: 10
CPU usage (average): 88.4%

Time (elapsed): 00:01:22.710


[Performance data]
Frame Frames/sec Time/frame(ms) CPU(%) Threads Memory(MiB)
1 0.861 1161.010346 88.7 8 32
2 0.867 1153.310189 95.9 8 34
3 816.667 1.224490 95.9 8 34
4 0.864 1156.834527 92.6 8 35
5 0.286 3490.868294 92.4 11 38
6 122.764 8.145730 92.4 11 38
7 714.958 1.398684 92.4 11 38
8 617.760 1.618752 92.4 11 38
9 0.892 1120.871662 91.7 11 38
10 0.296 3376.873175 90.3 11 39
11 736.060 1.358584 90.3 11 39
12 94461.086 0.010586 90.3 11 39
13 141691.641 0.007058 90.3 11 39
14 0.911 1097.751996 92.1 11 39
15 0.305 3275.588608 90.7 11 39
16 183.075 5.462246 90.7 11 39
17 97413.000 0.010266 90.7 11 39
18 119892.922 0.008341 90.7 11 39
19 0.938 1066.195277 94.2 11 39
20 0.312 3206.402249 90.2 11 39
21 143.399 6.973530 90.2 11 39
22 97413.000 0.010266 90.2 11 39
23 141691.641 0.007058 90.2 11 39
24 0.963 1038.887916 93.3 11 39
25 0.320 3121.036215 89.5 11 39
26 338.754 2.951993 89.5 11 39
27 107490.203 0.009303 89.5 11 39
28 100555.359 0.009945 89.5 11 39
29 0.974 1026.398543 92.4 9 39
30 0.981 1018.973359 93.2 9 39
31 0.493 2027.965302 88.1 9 39
32 635.000 1.574803 88.1 9 39
33 124688.633 0.008020 88.1 9 39
34 0.992 1007.803423 93.0 9 39
35 0.343 2919.189075 91.2 8 39
36 97.209 10.287064 91.2 8 39
37 13732.229 0.072821 91.2 8 39
38 111329.148 0.008982 91.2 8 39
39 1.074 931.320740 92.5 8 39
40 0.348 2869.875234 92.1 8 39
41 491.055 2.036433 92.1 8 39
42 103907.195 0.009624 92.1 8 39
43 129884.000 0.007699 92.1 8 39
44 1.061 942.834198 92.5 8 39
45 0.364 2748.311957 92.9 11 39
46 192.766 5.187642 92.9 11 39
47 21952.225 0.045553 92.9 11 39
48 103907.203 0.009624 92.9 11 39
49 1.106 904.053450 93.1 11 39
50 0.375 2663.956327 90.4 11 39
51 772.161 1.295066 90.4 11 39
52 107490.203 0.009303 90.4 11 39
53 141691.641 0.007058 90.4 11 39
54 1.149 870.302840 91.1 11 39
55 1.180 847.475730 90.7 11 39
56 0.399 2505.824395 91.6 11 39
57 81.946 12.203197 91.6 11 39
58 19361.590 0.051649 91.6 11 39
59 100555.359 0.009945 91.6 11 39
60 1.219 820.317202 92.5 11 39
61 1.254 797.715360 86.3 11 39
62 1.266 789.965491 94.1 11 39
63 0.420 2380.945022 88.2 11 40
64 195.584 5.112895 88.2 11 40
65 100555.352 0.009945 88.2 11 40
66 141691.641 0.007058 88.2 11 40
67 1.325 754.595430 92.9 11 39
68 0.443 2258.927492 89.7 11 39
69 90.059 11.103818 89.7 11 39
70 3853.172 0.259526 89.7 11 39
71 82032.000 0.012190 89.7 11 39
72 1.393 717.841176 88.3 11 39
73 0.468 2136.660046 89.8 11 39
74 188.374 5.308583 89.8 11 39
75 19007.414 0.052611 89.8 11 39
76 115452.445 0.008662 89.8 11 39
77 1.482 674.579508 95.3 11 39
78 1.516 659.717849 92.9 11 39
79 1.508 663.160949 91.9 11 39
80 764.398 1.308219 91.9 11 39
81 1.515 659.948187 89.3 11 39
82 0.789 1267.128407 86.6 11 39
83 745.389 1.341582 86.6 11 39
84 115452.445 0.008662 86.6 11 39
85 1.595 626.986087 88.8 11 39
86 0.563 1777.691316 89.0 11 39
87 191.605 5.219080 89.0 11 39
88 3471.287 0.288078 89.0 11 39
89 97413.000 0.010266 89.0 11 39
90 1.725 579.650571 89.2 11 39
91 0.588 1700.497554 88.1 11 39
92 76.892 13.005194 88.1 11 39
93 10976.112 0.091107 88.1 11 39
94 86589.336 0.011549 88.1 11 39
95 1.829 546.765780 80.6 11 39
96 0.637 1568.717683 87.6 11 39
97 835.267 1.197222 87.6 11 39
98 27344.000 0.036571 87.6 11 39
99 119892.922 0.008341 87.6 11 39
100 1.924 519.848494 84.8 11 39
101 0.680 1470.004651 86.2 8 39
102 109.885 9.100428 86.2 8 39
103 69271.469 0.014436 86.2 8 39
104 148438.859 0.006737 86.2 8 39
105 2.095 477.433728 86.2 8 39
106 1.073 931.584788 85.7 8 39
107 316.436 3.160192 85.7 8 39
108 103907.203 0.009624 85.7 8 39
109 2.186 457.430626 85.7 8 39
110 0.784 1276.154071 88.4 8 39
111 75.646 13.219488 88.4 8 39
112 16669.604 0.059989 88.4 8 39
113 5101.827 0.196008 88.4 8 39
114 2.484 402.645518 88.4 8 39
115 2.645 378.039907 86.0 8 39
116 2.525 395.996926 86.0 8 39
117 904.327 1.105794 86.0 8 39
118 2.667 374.976890 85.0 8 39
119 0.907 1101.967568 86.6 11 39
120 74.709 13.385341 86.6 11 39
121 1101.490 0.907861 86.6 11 39
122 403.889 2.475927 86.6 11 39
123 3.240 308.685383 86.6 11 39
124 0.993 1007.434213 85.9 11 39
125 56.075 17.833220 85.9 11 39
126 12723.331 0.078596 85.9 11 39
127 129884.000 0.007699 85.9 11 39
128 3.696 270.590814 85.9 11 39
129 3.558 281.051418 91.9 11 39
130 329.655 3.033476 91.9 11 39
131 1.262 792.377555 89.2 11 39
132 503.345 1.986709 89.2 11 39
133 49479.617 0.020210 89.2 11 39
134 76029.656 0.013153 89.2 11 39
135 4.311 231.941248 89.2 11 39
136 1.431 698.827100 84.7 11 39
137 555.951 1.798720 84.7 11 39
138 103907.203 0.009624 84.7 11 39
139 119892.914 0.008341 84.7 11 39
140 5.395 185.341348 84.7 11 39
141 1.746 572.697864 88.8 11 39
142 534.502 1.870900 88.8 11 39
143 84249.078 0.011870 88.8 11 39
144 129884.000 0.007699 88.8 11 39
145 5.965 167.638050 88.8 11 39
146 6.192 161.508859 88.8 11 39
147 683.600 1.462844 88.8 11 39
148 6.627 150.895544 88.8 11 39
149 2.640 378.835796 79.1 10 39
150 84.395 11.849034 79.1 10 39
151 70845.820 0.014115 79.1 10 39
152 124688.641 0.008020 79.1 10 39
153 9.759 102.472852 79.1 10 39
154 10.885 91.868513 79.1 10 39
155 804.443 1.243096 79.1 10 39
156 10.730 93.195340 79.1 10 39
157 4.468 223.838515 77.9 10 38
158 586.053 1.706330 77.9 10 38
159 241.383 4.142799 77.9 10 38
160 27832.285 0.035929 77.9 10 38
161 24.541 40.748541 77.9 10 38
162 28.543 35.035429 77.9 10 38

[pinterf]

I have done new tests including Reverse(). I have left Performance data because at the beginning of the test with Prefetch 1 it was like stuck:

PREFETCH 1

[Script]

LWLibavVideoSource("Sample2.mp4")
AssumeFPS("ntsc_film")
Reverse()
Prefetch(1)


[Runtime info]
Frames processed: 162 (0 - 161)
FPS (min | max | average): 0.838 | 25.19 | 1.555
Process memory usage (max): 35 MiB
Thread count: 10
CPU usage (average): 89.3%

Time (elapsed): 00:01:44.162


[Performance data]
Frame Frames/sec Time/frame(ms) CPU(%) Threads Memory(MiB)
1 0.868 1152.605727 92.6 7 32
2 0.860 1163.214254 91.3 7 33
3 0.859 1163.807085 93.2 7 34
4 0.871 1147.891894 93.9 7 35
5 0.869 1150.160631 94.6 10 35
6 0.866 1154.832738 91.2 10 35
7 0.854 1170.400746 89.3 10 35
8 0.866 1154.755398 92.6 10 35
9 0.871 1148.168415 91.8 10 35
10 0.901 1109.587501 91.7 10 35
11 0.889 1124.645850 90.3 10 35
12 0.905 1104.950360 91.4 10 35
13 0.900 1111.412557 91.7 10 35
14 0.911 1097.943521 92.9 10 35
15 0.907 1102.785271 91.5 10 35
16 0.922 1084.399055 94.2 10 35
17 0.918 1089.068591 91.4 10 35
18 0.838 1193.176204 82.5 10 35
19 0.868 1152.698459 86.3 10 35
20 0.887 1127.143238 88.4 10 35
21 0.903 1107.059022 86.6 10 35
22 0.891 1121.785248 88.2 10 35
23 0.904 1106.133519 86.6 10 35
24 0.956 1046.068052 94.8 10 35
25 0.968 1033.054167 90.9 10 35
26 0.944 1059.561132 89.7 10 35
27 0.964 1037.391365 92.4 10 35
28 0.958 1043.598816 88.8 10 35
29 0.968 1033.550756 90.2 10 35
30 1.005 994.713861 89.1 10 35
31 0.984 1016.456976 92.3 10 35
32 0.986 1014.562660 86.2 10 35
33 0.992 1008.083794 89.2 10 35
34 1.019 980.900602 92.9 10 35
35 1.011 989.204094 91.3 10 35
36 1.028 972.877384 93.7 10 35
37 1.034 966.910498 93.5 10 35
38 1.019 981.714723 90.3 10 35
39 1.053 949.681402 91.0 10 35
40 1.032 969.054089 91.9 10 35
41 1.093 914.576683 93.2 10 35
42 1.057 946.248242 91.8 10 35
43 1.075 930.512359 91.5 10 35
44 1.082 924.285279 94.2 10 35
45 1.094 913.881815 93.1 10 35
46 1.090 917.400964 89.8 10 35
47 1.086 920.875541 89.0 10 35
48 1.098 910.738663 93.1 10 35
49 1.104 905.574080 92.4 10 35
50 1.128 886.395440 87.5 10 35
51 1.131 884.395587 93.9 10 35
52 1.158 863.869565 91.8 10 35
53 1.164 859.396950 89.3 10 35
54 1.136 880.626169 87.5 10 35
55 1.166 857.511328 91.8 10 35
56 1.189 841.396264 89.8 10 35
57 1.173 852.761928 91.8 10 35
58 1.227 815.170003 94.2 10 35
59 1.226 815.406132 87.5 10 35
60 1.238 807.885656 91.3 10 35
61 1.210 826.757565 86.8 10 35
62 1.248 801.225171 91.2 10 35
63 1.241 805.734323 91.3 10 35
64 1.288 776.559598 89.0 10 35
65 1.296 771.628287 92.9 10 35
66 1.316 759.677832 93.9 10 35
67 1.293 773.108110 90.8 10 35
68 1.287 776.797334 87.0 10 35
69 1.342 744.928782 91.7 10 35
70 1.394 717.433096 88.0 10 35
71 1.332 750.892134 87.5 10 35
72 1.292 774.140130 92.0 10 35
73 1.245 803.196160 82.4 10 35
74 1.386 721.294921 89.1 10 35
75 1.452 688.873649 93.3 10 35
76 1.470 680.095939 89.5 10 35
77 1.442 693.472318 92.2 10 35
78 1.479 676.164253 89.5 10 35
79 1.491 670.869132 91.9 10 35
80 1.524 656.118791 94.0 10 35
81 1.474 678.650105 89.5 8 35
82 1.624 615.826444 91.3 8 35
83 1.574 635.319155 92.7 8 35
84 1.612 620.326939 91.0 8 35
85 1.592 628.238126 92.7 8 35
86 1.658 602.955663 92.1 8 35
87 1.596 626.448099 86.6 8 35
88 1.703 587.105911 93.2 8 35
89 1.598 625.863260 86.3 8 35
90 1.809 552.660775 95.8 10 35
91 1.716 582.876526 93.2 10 35
92 1.783 560.723722 91.7 10 35
93 1.761 567.995272 94.4 10 35
94 1.777 562.802850 86.1 10 35
95 1.874 533.554610 92.9 10 35
96 1.911 523.241909 92.4 10 35
97 1.868 535.429382 91.4 10 35
98 1.901 526.158922 92.4 10 35
99 1.955 511.549740 92.4 10 35
100 1.983 504.195737 92.4 10 35
101 1.993 501.814429 91.5 10 35
102 2.061 485.252528 91.5 10 35
103 1.964 509.126098 86.5 10 35
104 2.116 472.494031 86.5 10 35
105 2.093 477.865189 92.6 10 35
106 2.223 449.822234 92.6 10 35
107 2.118 472.199231 89.0 10 35
108 2.132 469.079773 89.0 10 35
109 2.222 450.029769 86.4 10 35
110 2.256 443.280814 86.4 10 35
111 2.281 438.488376 86.0 10 35
112 2.394 417.628084 86.0 10 35
113 2.546 392.777096 92.3 10 35
114 2.466 405.587221 92.3 10 35
115 2.537 394.110014 92.2 10 35
116 2.553 391.752462 92.2 10 35
117 2.646 377.912209 89.8 10 35
118 2.704 369.879070 89.8 10 35
119 2.652 377.055371 91.7 10 35
120 2.836 352.659240 91.7 10 35
121 2.833 352.973294 93.3 10 35
122 2.744 364.379317 93.3 10 35
123 2.911 343.555287 87.0 10 35
124 3.111 321.448051 87.0 10 35
125 3.207 311.833707 90.0 10 35
126 3.220 310.557877 90.0 10 35
127 2.868 348.652453 84.9 7 35
128 3.472 288.004737 84.9 7 35
129 3.448 290.050482 85.1 7 35
130 3.486 286.857882 85.1 7 35
131 3.589 278.644158 87.5 7 35
132 3.765 265.623233 87.5 7 35
133 3.804 262.877840 91.2 7 35
134 3.994 250.397150 91.2 7 35
135 4.037 247.717520 91.2 7 35
136 4.243 235.660926 85.1 7 35
137 4.165 240.086035 85.1 7 35
138 4.397 227.417023 85.1 7 35
139 5.102 195.998610 86.9 7 35
140 5.007 199.715714 86.9 7 35
141 4.558 219.410535 86.9 7 35
142 5.479 182.525698 88.5 7 34
143 5.442 183.759807 88.5 7 34
144 6.010 166.388856 88.5 7 34
145 6.149 162.637108 86.4 7 35
146 5.470 182.807666 86.4 7 35
147 6.705 149.146230 86.4 7 35
148 6.671 149.894013 86.4 7 35
149 7.305 136.895232 80.0 7 34
150 8.430 118.623153 80.0 7 34
151 7.554 132.380941 80.0 7 34
152 8.378 119.355226 80.0 7 34
153 9.741 102.661154 80.0 7 34
154 10.705 93.416690 81.9 10 34
155 10.149 98.527661 81.9 10 34
156 11.597 86.229827 81.9 10 34
157 12.426 80.473730 81.9 10 34
158 13.111 76.273830 81.9 10 34
159 19.560 51.125105 81.9 10 34
160 12.225 81.800877 81.9 10 34
161 25.191 39.697282 75.8 10 34
162 24.674 40.528793 75.8 10 34

As expected. In order the get the Nth frame it seems that all previous (N-1) frames must be decoded again from the beginning. As it is getting closer to the beginning of the video it gets quicker and quicker. What about testing with a video file where _all_ frames are I frame? (Did not investigate your specific test file how it is encoded)

[GMJCZP]

Here is the Mediainfo text:

General
Complete name : video.mp4
Format : MPEG-4
Format profile : Base Media
Codec ID : iso4 (iso4/hvc1/iso6)
File size : 223 MiB
Duration : 21 min 2 s
Overall bit rate mode : Variable
Overall bit rate : 1 484 kb/s
Encoded date : UTC 2016-07-18 23:03:09
Tagged date : UTC 2016-07-18 23:03:09

VÃ*deo
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Format Range@L3@Main
Codec ID : hvc1
Codec ID/Info : High Efficiency Video Coding
Duration : 21 min 2 s
Bit rate : 1 384 kb/s
Maximum bit rate : 3 199 kb/s
Width : 640 pÃ*xeles
Height : 480 pÃ*xeles
Display aspect ratio : 4:3
Frame rate mode : Constante
Frame rate : 25,000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 12 bits
Bits/(Pixel*Frame) : 0.180
Stream size : 208 MiB (93%)
Writing library : x265 2.0+4-43ca544799c2:[Windows][MSVC 1900][64 bit] 12bit
Encoding settings : wpp / ctu=64 / min-cu-size=8 / max-tu-size=32 / tu-intra-depth=1 / tu-inter-depth=1 / me=1 / subme=2 / merange=57 / no-rect / no-amp / max-merge=2 / temporal-mvp / no-early-skip / rskip / rdpenalty=0 / no-tskip / no-tskip-fast / strong-intra-smoothing / no-lossless / no-cu-lossless / no-constrained-intra / no-fast-intra / open-gop / no-temporal-layers / interlace=0 / keyint=250 / min-keyint=25 / scenecut=40 / rc-lookahead=20 / lookahead-slices=0 / bframes=4 / bframe-bias=0 / b-adapt=2 / ref=3 / limit-refs=3 / no-limit-modes / weightp / no-weightb / aq-mode=1 / qg-size=32 / aq-strength=1.00 / cbqpoffs=0 / crqpoffs=0 / rd=3 / psy-rd=2.00 / rdoq-level=0 / psy-rdoq=0.00 / no-rd-refine / signhide / deblock=0:0 / sao / no-sao-non-deblock / b-pyramid / cutree / no-intra-refresh / rc=crf / crf=21.0 / qcomp=0.60 / qpmin=0 / qpmax=51 / qpstep=4 / ipratio=1.40 / pbratio=1.30
Encoded date : UTC 2016-07-18 23:03:09
Tagged date : UTC 2016-07-18 23:03:14
Codec configuration box : hvcC

[Dogway]

I found something strange. Using the next expression in 16-bit to convert to TV levels yields a byte white level of 234:
"x ymax ymin - range_max / * ymin +"
That would correspond to (n*256):
"x 60160 4096 - 65535 / * 4096 +"

This is fixed if I multiply 235*257 (= 60395) instead to get the correct ymax 16-bit value.
Shouldn't this be fixed internally? I can set scale_inputs to "allf" and it fixes, but I'm not sure that's the original intention of the option (scale expression to 8-bit). Also by doing so I lose the option to use "FloatUV" for 32-bit chroma, since there's no "FloatUVf".

[GMJCZP]

pinterf, due to technical problems I will not be able to continue, at least for now, in Doom9, so I ask you please, continue looking for a solution to the use of Prefetch. Thank you and I hope to return soon, greetings to all.

[zorr]

This page (http://avisynth.nl/index.php/Internal_functions#Functions_for_frame_properties) mentions that frame property value can be a clip reference (PClip). When I try that with

final = propSet(final, "test", function[store_this_clip](clip c) { return store_this_clip} )

I get message "propAdd: Clip frame properties not yet supported". Is that true or am I doing something wrong? I also tried other variations, storing clips into an array etc.

Having clips as frame properties would be very useful, you could for example return debug clips along with the main clip. This would be an elegant solution for returning a mask visualization clip from Delta Restore (https://forum.doom9.org/showthread.php?p=1944267#post1944267).

[pinterf]

pinterf, due to technical problems I will not be able to continue, at least for now, in Doom9, so I ask you please, continue looking for a solution to the use of Prefetch. Thank you and I hope to return soon, greetings to all.
Source filter problem, which has a 10x 100x 1000x time penalty if requested out of order frames. Or do not use Prefetch directly for a serialized filter, or use its internal multithreading switch, if any. MT_SERIALIZED was never meant to guarantee linear access.

[pinterf]

This page (http://avisynth.nl/index.php/Internal_functions#Functions_for_frame_properties) mentions that frame property value can be a clip reference (PClip). When I try that with

final = propSet(final, "test", function[store_this_clip](clip c) { return store_this_clip} )

I get message "propAdd: Clip frame properties not yet supported". Is that true or am I doing something wrong? I also tried other variations, storing clips into an array etc.

Having clips as frame properties would be very useful, you could for example return debug clips along with the main clip. This would be an elegant solution for returning a mask visualization clip from Delta Restore (https://forum.doom9.org/showthread.php?p=1944267#post1944267).
I never tried and debugged how clips behave inside frame properties, so it is disabled at the moment. Though I can check it how must task is enabling it as-is. Worst case: they do not work out of box.

[real.finder]

I found something strange. Using the next expression in 16-bit to convert to TV levels yields a byte white level of 234:
"x ymax ymin - range_max / * ymin +"
That would correspond to (n*256):
"x 60160 4096 - 65535 / * 4096 +"

This is fixed if I multiply 235*257 (= 60395) instead to get the correct ymax 16-bit value.
Shouldn't this be fixed internally? I can set scale_inputs to "allf" and it fixes, but I'm not sure that's the original intention of the option (scale expression to 8-bit). Also by doing so I lose the option to use "FloatUV" for 32-bit chroma, since there's no "FloatUVf".

16bit sources will be limited to 4096 - 60160 luma and 4096 - 61440 chroma as said here https://forum.doom9.org/showthread.php?t=181857

convertbits(16)
ScriptClip("""
Subtitle(String(expr("ymax").AverageLuma()))
""")

show 60160 as it should

[pinterf]

New build, primarily for zorr, propAdd allows Clip type. Not tested at all, I had time only for adding the possibility.
Avisynth 3.7.1 20210618 test5
https://drive.google.com/uc?export=download&id=1epFQMNjU3B7rRY-btSK8oN-xFsvje-G_

[Dogway]

16bit sources will be limited to 4096 - 60160 luma and 4096 - 61440 chroma as said here https://forum.doom9.org/showthread.php?t=181857

convertbits(16)
ScriptClip("""
Subtitle(String(expr("ymax").AverageLuma()))
""")

show 60160 as it should

For conversions you need to full scale stretch if working in PC levels. Read here (https://forum.doom9.org/showthread.php?p=1685118#post1685118).

In any case, I don't care anymore what avisynth+ does, using auto-scaling is very slow, "allf" is about 30% slower than "none" because the constants (ymax, range_max, etc) are computed at runtime. So I'm bulding a look up table which will speed things up quite a bit. Thought pinterf wanted to fix it but he doesn't seem very interested on the topic from lack of feedback.

# 8-bit 10-bit 12-bit 14-bit 16-bit 32-bit
range_min = Select (bits, [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0])
ymin = Select (bits, [ 16, 16], [ 64, 80], [ 256, 272], [ 1024, 1040], [ 4096, 4112], [ 16/255., 16/255.])
cmin = Select (bits, [ 16, 16], [ 64, 80], [ 256, 272], [ 1024, 1040], [ 4096, 4112], [-112/255.,-112/255.])
range_half = Select (bits, [128,128], [ 512, 640], [2048,2176], [ 8192, 8320], [32768,32896], [ 128/255., 128/255.])
yrange = Select (bits, [219,219], [ 876,1095], [3504,3723], [14016,14235], [56064,56283], [ 219/255., 219/255.])
crange = Select (bits, [224,224], [ 896,1120], [3584,3808], [14336,14560], [57344,57568], [ 224/255., 224/255.])
ymax = Select (bits, [235,235], [ 940,1175], [3760,3995], [15040,15275], [60160,60395], [ 235/255., 235/255.])
cmax = Select (bits, [240,240], [ 960,1200], [3840,4080], [15360,15600], [61440,61680], [ 107/255., 107/255.])
range_max = Select (bits, [255,255], [1020,1023], [4080,4095], [16320,16383], [65280,65535], [1.0,1.0])
range_size = Select (bits, [256,256], [1023,1023], [4096,4096], [16384,16384], [65536,65536], [1.0,1.0])

[pinterf]

Thought pinterf wanted to fix it but he doesn't seem very interested on the topic from lack of feedback.

Feedback
I wonder why do you think I'm behind the computer and forums 24/7 and answer _all_ questions immediately at most in a couple of days? Not having time for this hobby (even reading forums) for a couple of _weeks_ is perfectly acceptable I think. Patience, please, I'm not a live L3 support line.

[tormento]

Thought pinterf wanted to fix it but he doesn't seem very interested on the topic from lack of feedback.
FranceBB is the uber color space master :)

[real.finder]

For conversions you need to full scale stretch if working in PC levels. Read here (https://forum.doom9.org/showthread.php?p=1685118#post1685118).

In any case, I don't care anymore what avisynth+ does, using auto-scaling is very slow, "allf" is about 30% slower than "none" because the constants (ymax, range_max, etc) are computed at runtime. So I'm bulding a look up table which will speed things up quite a bit. Thought pinterf wanted to fix it but he doesn't seem very interested on the topic from lack of feedback.

# 8-bit 10-bit 12-bit 14-bit 16-bit 32-bit
range_min = Select (bits, [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0])
ymin = Select (bits, [ 16, 16], [ 64, 80], [ 256, 272], [ 1024, 1040], [ 4096, 4112], [ 16/255., 16/255.])
cmin = Select (bits, [ 16, 16], [ 64, 80], [ 256, 272], [ 1024, 1040], [ 4096, 4112], [-112/255.,-112/255.])
range_half = Select (bits, [128,128], [ 512, 640], [2048,2176], [ 8192, 8320], [32768,32896], [ 128/255., 128/255.])
yrange = Select (bits, [219,219], [ 876,1095], [3504,3723], [14016,14235], [56064,56283], [ 219/255., 219/255.])
crange = Select (bits, [224,224], [ 896,1120], [3584,3808], [14336,14560], [57344,57568], [ 224/255., 224/255.])
ymax = Select (bits, [235,235], [ 940,1175], [3760,3995], [15040,15275], [60160,60395], [ 235/255., 235/255.])
cmax = Select (bits, [240,240], [ 960,1200], [3840,4080], [15360,15600], [61440,61680], [ 107/255., 107/255.])
range_max = Select (bits, [255,255], [1020,1023], [4080,4095], [16320,16383], [65280,65535], [1.0,1.0])
range_size = Select (bits, [256,256], [1023,1023], [4096,4096], [16384,16384], [65536,65536], [1.0,1.0])

how about "x 235 scalef 16 scalef - range_max / * 16 scalef +" ?

maybe ymax problem with full range can be fixed with new "nonef"

edit: btw:-

"allf" is about 30% slower than "none" because the constants (ymax, range_max, etc) are computed at runtime

this is not how it works, when using scale_inputs (auto-scaling) the expression will be in 8bit (unless you use http://avisynth.nl/index.php/Expr#Keywords_for_modifying_base_bit_depth) so ymax will be 235.0 in this case, scale_inputs as it name said it do scale (the clip/s) to 8bit (or http://avisynth.nl/index.php/Expr#Keywords_for_modifying_base_bit_depth) range but with float then the output will be rescale to original bitdepth

[Dogway]

Yes, "{n} scalef" is almost as fast as "none". The problem is we are still using 8-bit constants which is a bit misleading. I thought ymax, ymin... constants were there to replace 8-bit expression syntax and make it cleaner.


Here's the function if you want to play with it, if everything goes according I will add it to ExTools soon, forget about 24-bit for the moment.

function ex_dlut(string "str", int "bits", bool "tv_range") {

str = Default(str, "")
bits = Default(bits, 8)
tv = Default(tv_range, true)

bitd =
\ (bits == 8 ) ? 0
\ : (bits == 10 ) ? 1
\ : (bits == 12 ) ? 2
\ : (bits == 14 ) ? 3
\ : (bits == 16 ) ? 4
\ : (bits == 24 ) ? 5
\ : (bits == 32 ) ? 6
\ : Assert (false, "Unsupported bit depth.")


# 8-bit UINT 10-bit UINT 12-bit UINT 14-bit UINT 16-bit UINT 24-bit UINT 32-bit float
range_min = Select (bitd, [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0., 0.])
ymin = Select (bitd, [ 16, 16], [ 64, 64], [ 256, 257], [ 1024, 1028], [ 4096, 4112], [ 1048576, 1052672], [ 16/255., 16/255.])
cmin = Select (bitd, [ 16, 16], [ 64, 64], [ 256, 257], [ 1024, 1028], [ 4096, 4112], [ 1048576, 1052672], [-112/255.,-112/255.])
range_half = Select (bitd, [128,128], [ 512, 514], [2048,2056], [ 8192, 8224], [32768,32896], [ 8388608, 8421376], [ 128/255., 128/255.])
yrange = Select (bitd, [219,219], [ 876, 879], [3504,3517], [14016,14070], [56064,56283], [14352384,14408448], [ 219/255., 219/255.])
crange = Select (bitd, [224,224], [ 896, 899], [3584,3597], [14336,14391], [57344,57568], [14680064,14737408], [ 224/255., 224/255.])
ymax = Select (bitd, [235,235], [ 940, 943], [3760,3774], [15040,15098], [60160,60395], [15400960,15461120], [ 235/255., 235/255.])
cmax = Select (bitd, [240,240], [ 960, 963], [3840,3854], [15360,15419], [61440,61680], [15728640,15790080], [ 107/255., 107/255.])
range_max = Select (bitd, [255,255], [1020,1023], [4080,4095], [16320,16383], [65280,65535], [16711680,16776960], [ 1., 1.])
range_size = Select (bitd, [256,256], [1024,1024], [4096,4096], [16384,16384], [65536,65536], [16777216,16777216], [ 1., 1.])

tv = tv ? 0 : 1
str = ReplaceStr(str, "ymax ymin -", string(yrange[tv]))
str = ReplaceStr(str, "cmax cmin -", string(crange[tv]))
str = ReplaceStr(str, "range_min", string(range_min[tv]))
str = ReplaceStr(str, "ymin", string(ymin[tv]))
str = ReplaceStr(str, "cmin", string(cmin[tv]))
str = ReplaceStr(str, "range_half", string(range_half[tv]))
str = ReplaceStr(str, "ymax", string(ymax[tv]))
str = ReplaceStr(str, "cmax", string(cmax[tv]))
str = ReplaceStr(str, "range_max", string(range_max[tv]))
str = ReplaceStr(str, "range_size", string(range_size[tv]))
str = ReplaceStr(str, "}", "} "+string(pow(2,bits-8))+" *")

return str }

FranceBB is the uber color space master :)
@tormento: wait (https://www.artstation.com/artwork/KrzrQR)and (https://forums.libretro.com/t/dogways-grading-shader-slang/27148)see (https://github.com/Dogway/Avisynth-Scripts/blob/master/TransformsPack.v1.0.RC17.avsi):rolleyes:

@real.finder: so what do you propose (correct method), to use scalef/scaleb all along the inputs? not only the code is dirtier but also slower*. What was the constants use case for then...

417fps (*about same if using 219 scalef)
Expr("x 235 scalef 16 scalef - range_max / * 16 scalef +",scale_inputs="none")
432fps
Expr(ex_dlut("x ymax ymin - range_max / * ymin +",16,false))

[real.finder]

scalef/scaleb should be as fast as ymax, ymin etc... anyway pinterf should give the right answer

anyway, I will try add new constants for full range ymaxf, yminf etc... and do PR

[Dogway]

That would lead to lots of string if else's or ReplaceStr() to change ie. ymax to ymaxf. Wouldn't be easier an argument for it? I don't see how one would want to mix full scale with stretched constants.

Anyway, thanks for looking into it, this thing was affecting my code progress.

[real.finder]

try https://www.solidfiles.com/v/ZZvV2ezjvvnQn

it has new yfmin, yfmax, cfmin and cfmax keywords

[Dogway]

Thank you. I think that nonef was a good idea except for displacing the option to use FloatUV for 32-bit chroma processing. I'm currently rebasing all my scripts.

[zorr]

New build, primarily for zorr, propAdd allows Clip type. Not tested at all, I had time only for adding the possibility.
Avisynth 3.7.1 20210618 test5

Wow thanks! I'll conduct some testing after my current MVTools test run is finished.

[GMJCZP]

Patience, please, I'm not a live L3 support line.

Thanks for your efforts.
I solved my technical problems, I hope.

[FranceBB]

The new build works fine on XP x86, Ferenc, thanks! :)
About the fact that Visual Studio 2022 won't support v141_xp as target, well... I'm just gonna say that the disastrous day in which Avisynth won't work on XP anymore is gonna be the day in which I'll shut down forever the Windows XP Virtual Machine and I'll fire up the Sucksdown 10 one... :'(

(or if by that time we'll have everything working on Linux, then I'll just stick with Fedora eheheheh)

[zorr]

Test results: I think I was able to store a clip into frame properties, at least there was no error and the clip displays fine after that. This works:

clip = propSet(clip, "test", function[store_this](clip c) { return store_this } )

and I think it can be turned into an array of clips by adding

clip = propSet(clip, "test", function[store_this](clip c) { return store_this }, mode=1 )

However I can't get the clip out. There's no propGetClip() -function. Doing

return ScriptClip(clip, """ return propGetAny(clip, "test") """)

returns Error getting frame property "test": type 'c' not supported

and if I try to read a clip from an array with

return ScriptClip(clip, """ return propGetAsArray(clip, "test")[0] """)

it returns ScriptClip: Function did not return a video clip! (Was the undefined value)

So looks like there needs to be some additional support for reading the clip typed frame properties. :)

Oh, and for some reason the plugins in x64 directory were actually 32bit versions. I just replaced them with my previous x64 versions.

[tormento]

I'll fire up the Sucksdown 10 one... :'(
Or 11 directly. :)
(or if by that time we'll have everything working on Linux, then I'll just stick with Fedora eheheheh)
Picture that I am currently using Windows 10 + WSLg with Debian. They introduced graphic and CUDA support too. In many instances it can get better scores than pure Linux.

[MysteryX]

Levels function isn't working in 16-bit. Output is completely black.

ConvertBits(16)
#ConvertToRGB64(matrix="Rec601")
ConvertToPlanarRGB(matrix="Rec601")
Levels(0, 1/2.2, 255, 0, 255)
ConvertToY(matrix="Rec601")
ConvertToYUV420()

[feisty2]

because "white" in int16 is 65535 not 255:rolleyes:

[pinterf]

Levels function isn't working in 16-bit. Output is completely black.

http://avisynth.nl/index.php/Levels
There is no automatic scaling.

[pinterf]

However I can't get the clip out. There's no propGetClip() -function. Doing

return ScriptClip(clip, """ return propGetAny(clip, "test") """)

returns Error getting frame property "test": type 'c' not supported


Thanks, then I'll arrange a next turn soon.

[pinterf]

Yes, "{n} scalef" is almost as fast as "none". The problem is we are still using 8-bit constants which is a bit misleading. I thought ymax, ymin... constants were there to replace 8-bit expression syntax and make it cleaner.
The zillion options (both commandline arguments and internal helper keywords) of autoscaling or using 8 bit values which are converted automatically were created for help. Because when Avisynth got 10+ bits, expressions supported only 8 bits. To help the transition to translate the scripts to HBD aware (which was and is mainly done by the huge work of real.finder) these various options appeared, first in masktools then later in Expr.

I have to mention (not for you, Dogway, because you experienced a lot with Expr), using 8 bit constants are safe, they are internally treated as 32 bit floats so e.g. 16/255 does not become 0 inside.

[pinterf]

scalef/scaleb should be as fast as ymax, ymin etc... anyway pinterf should give the right answer

anyway, I will try add new constants for full range ymaxf, yminf etc... and do PR

Expressions with operation on constants are optimized. E.g. 2 2 * is precalculated to 4. I have to check but for example the expression '235 scalef' is optimized as well during the parsing time to a single constant.

Other optimizations exists, e.g. x Power of 2 is optimized to x*x (powers -1, 0, 1, 2, 3 and 4 are handled)

[pinterf]

I found something strange. Using the next expression in 16-bit to convert to TV levels yields a byte white level of 234:
"x ymax ymin - range_max / * ymin +"
That would correspond to (n*256):
"x 60160 4096 - 65535 / * 4096 +"

This is fixed if I multiply 235*257 (= 60395) instead to get the correct ymax 16-bit value.
Shouldn't this be fixed internally? I can set scale_inputs to "allf" and it fixes, but I'm not sure that's the original intention of the option (scale expression to 8-bit). Also by doing so I lose the option to use "FloatUV" for 32-bit chroma, since there's no "FloatUVf".
What is your script? This one returns 235.
ColorBarsHD(pixel_type = "YUV444P16")
Expr("65535","65535","65535")
Expr("x ymax ymin - range_max / * ymin +")
ConvertBits(8)
ScriptClip("""
Subtitle(String(expr("x").AverageLuma()))
""")



EDIT:
I got it where you got 234. This happens only when you mix the 8 bit inputs with 16 bit ranges.
(235 - 16) / 255 is not the same conversion factor as (60160 - 4096) / 65535

Calculations:

255.0f * (235.f - 16.f) / 255.f + 16.0f;
Result = 235.0

255.0f * (60160.f - 4096.f) / 65535.f + 16.0f)
Result = 234.1478

65535 * (60160.f - 4096.f) / 65535.f + 4096.0f;
Result: 60160.0

[pinterf]

['Clip' type frame property read]
Next test iteration:
AviSynth+ 3.7.1-test6 20210621 (https://drive.google.com/uc?export=download&id=1xlNwp3XH7RUaRmUMdakcR4x3LdKoTYVT)

[pinterf]

For conversions you need to full scale stretch if working in PC levels. Read here (https://forum.doom9.org/showthread.php?p=1685118#post1685118).

In any case, I don't care anymore what avisynth+ does, using auto-scaling is very slow, "allf" is about 30% slower than "none" because the constants (ymax, range_max, etc) are computed at runtime. So I'm bulding a look up table which will speed things up quite a bit. Thought pinterf wanted to fix it but he doesn't seem very interested on the topic from lack of feedback.

# 8-bit 10-bit 12-bit 14-bit 16-bit 32-bit
range_min = Select (bits, [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0], [ 0, 0])
ymin = Select (bits, [ 16, 16], [ 64, 80], [ 256, 272], [ 1024, 1040], [ 4096, 4112], [ 16/255., 16/255.])
cmin = Select (bits, [ 16, 16], [ 64, 80], [ 256, 272], [ 1024, 1040], [ 4096, 4112], [-112/255.,-112/255.])
range_half = Select (bits, [128,128], [ 512, 640], [2048,2176], [ 8192, 8320], [32768,32896], [ 128/255., 128/255.])
yrange = Select (bits, [219,219], [ 876,1095], [3504,3723], [14016,14235], [56064,56283], [ 219/255., 219/255.])
crange = Select (bits, [224,224], [ 896,1120], [3584,3808], [14336,14560], [57344,57568], [ 224/255., 224/255.])
ymax = Select (bits, [235,235], [ 940,1175], [3760,3995], [15040,15275], [60160,60395], [ 235/255., 235/255.])
cmax = Select (bits, [240,240], [ 960,1200], [3840,4080], [15360,15600], [61440,61680], [ 107/255., 107/255.])
range_max = Select (bits, [255,255], [1020,1023], [4080,4095], [16320,16383], [65280,65535], [1.0,1.0])
range_size = Select (bits, [256,256], [1023,1023], [4096,4096], [16384,16384], [65536,65536], [1.0,1.0])
Let's see only a single example.
How did you get the constant of 80 for 10 bit ymin instead of 64?
Min values must keep the *rule of 8_bit_min * 2^(N-8) where N is the bit depth.
As for limited range maximum values there was a decision of keeping the same 8_bit_max * 2^(N-8) calculation, against the other choice of (8_bit_max + 1) * 2^(N-8) - 1.
Limited range ConvertBits() conversions are simple bit-shifts (upwards) and round-then-bitshift (downwards)

(note: Expr is always rounding and does not truncate before converting back to the integer domain)

EDIT: see calculations in my previous answer (#1116) where I put it where your calculation went astray in my opinion.

[Dogway]

Even if the ex_dlut() function might seem overdone to simply "fix" an issue in Expr() I will keep it to save some operations (ie. "ymax ymin - range_max /") and for reference.

You are referencing an old table which was wrong. You can look my table 4 posts later (https://forum.doom9.org/showthread.php?p=1945324#post1945324). Or simply check the updated function (https://github.com/Dogway/Avisynth-Scripts/blob/c1bbb41ab15128a192b5d7506c9a07c5aa127208/ExTools.avsi#L845) in ExTools 2.0 with float precision. It works on my tests but if you spot something wrong let me know.

Over the development of Transforms Pack, where I need bit perfect conversions I noticed something was off and that led me to this conclusion.

Your sample script is crafted to output the result you look for. A real world scenario is more like the following (no body uses fulls=true to be honest).

ColorBarsHD(pixel_type = "YUV444P8")
Expr("255","255","255")
ConvertBits(16)
Expr("x ymax ymin - range_max / * ymin +")
ConvertBits(8)
ScriptClip("""
Subtitle(String(expr("x").AverageLuma()))
""")


EDIT: To answer your #1116 EDIT. Yes, range_max is scaling in full stretch fashion whereas everything else does in bitshift, disparity that leads to that error.

255.0f * (60160.f - 4096.f) / 65280.f + 16.0f)
Result = 235.0

65280 * (60160.f - 4096.f) / 65280.f + 4096.0f;
Result: 60160.0

[pinterf]

Your sample script is crafted to output the result you look for. A real world scenario is more like the following (no body uses fulls=true to be honest).

Ah, that's the problem then. Wrong use of ConvertBits makes 255 to 255*256 instead of 65535.
Well, if people are not telling the conversion type if it should use full or limited range but still expect perfectly guessed results, I can't help with it. I'm somehow reluctant to support Avisynth Expr with new bad-habit workaround constants.

[Dogway]

ConvertBits(16) # Default bitshift scaling. fulls=false for YUV

Expr("range_max") # Default full stretch scaling.


If you don't see the problem I can't help anymore.

[pinterf]

Side note:

Avisynth supports frame properties, but only provides the internal framework for it. Many external plugins are actively using them, z_xxxxx resizers/format converter for example.

But - due to lack of development resources - there is zero usage of them internally. Color matrix, full-limited, others. ConvertBits would automatically know the proper input format (full scale) and voilá, no more user error.

It is on the roadmap.

[Dogway]

Still need to get into frame properties, see if they impact performance, reliability and do implementation.

I'm somehow reluctant to support Avisynth Expr with new bad-habit workaround constants.

I'm super intrigued in doing a research of how many current HBD aware scripts/plugins work correctly as you expect. I will try to find some time and report if my "bad-habit workaround" works better or not.

[pinterf]

ConvertBits(16) # Default bitshift scaling. fulls=false for YUV

Expr("range_max") # Default full stretch scaling.


If you don't see the problem I can't help anymore.
What problem. It's how it is defined.
Range_max is _not_ representing a scaled 255 maximum, but the maximum value of the actual bit depth.

[zorr]

['Clip' type frame property read]
Next test iteration:
AviSynth+ 3.7.1-test6 20210621 (https://drive.google.com/uc?export=download&id=1xlNwp3XH7RUaRmUMdakcR4x3LdKoTYVT)

Thanks again. I'm running MVTools tests again and looks like it could take a couple of days until they're finished, I'll test after that.

[tormento]

AviSynth+ 3.7.1-test6 20210621
Internal version 3.7.1·3413, such as test5. Is it correct? The two dlls are anyway different in content.

[wonkey_monkey]

Could Expr be updated to treat newlines and tabs (\r, \n, \t, and any other character less than 32 for that matter) inside expressions exactly the same as spaces? It does odd and slightly unpredictable things at the moment - it seems to work okay if a space is put before the newline and if the first thing on the next line is a number, but otherwise it complains about being unable to convert to float.

[StainlessS]

and any other character less than 32 for that matter
Maybe only those considered white space, [EDIT: else, non Parse] ie

Function IsWhite(String s) { s.RT_Ord==32||(s.RT_Ord>=8&&s.RT_Ord<=13) }


EDIT:

32 = SPACE
8 = BACKSPACE
9 = TAB
10 = LINEFEED
11 = VERTICAL TAB
12 = FORMFEED
13 = CARRIAGE RETURN


So eg Chr(7) [BELL, would make a printer go "Ding!"] should maybe produce error abort.
Maybe not good idea to just ignore stuff that defo should not be there.

[gispos]

A small (big?) Wish.
The C interface has problems with Utf-8 and with Delphi Ansi strings.
Utf-8 and Delphi Ansi strings only work with AviSource. 'Eval' or 'Import' does not work.
Frame properties can only be read indirectly.

An update of the C interface would be nice.

[qyot27]

A small (big?) Wish.
The C interface has problems with Utf-8 and with Delphi Ansi strings.
Utf-8 and Delphi Ansi strings only work with AviSource. 'Eval' or 'Import' does not work.
Frame properties can only be read indirectly.

An update of the C interface would be nice.
And do you have the Windows system codepage set to UTF-8?

[kedautinh12]

Yeah, i had

[gispos]

And do you have the Windows system codepage set to UTF-8?
I am using Win7.
Earlier (a long time ago) the Delphi Ansi strings with 'Eval' and 'Import' worked. With the newer Avisynth versions, these strings are sometimes read incorrectly.
If I use e.g. MCTemporalDenoise() there is an error message division by 0.

With AvsPmod I can only open scripts containing Utf-8 characters when I import them with AviSource.

[pinterf]

Internal version 3.7.1·3413, such as test5. Is it correct? The two dlls are anyway different in content.
Yes, same, did not commit the changes until the test succeeds. build number is changed after commits.

[pinterf]

A small (big?) Wish.
The C interface has problems with Utf-8 and with Delphi Ansi strings.
Utf-8 and Delphi Ansi strings only work with AviSource. 'Eval' or 'Import' does not work.
Frame properties can only be read indirectly.

An update of the C interface would be nice.
AnsiString stuff: internally you can use String and not AnsiString.
Then it can be converted to utf8 if needed.
var unistring: String := ...
var Bytes: TBytes;
Bytes := TEncoding.UTF8.GetBytes(unistring);


Import has also an alternative form for utf8 filenames (if windows codepage is not utf8 natively): Import(filename [, ...] [, bool utf8])
then pass this buffer to Eval if Windows system codepage is set to UTF-8.

As for frame properties: there is a fix in these post-3.7 version which does not crash (yes, there was a bug) when using frame property related functions from C.

[pinterf]

Frame properties from C interface
(quick test; I just put all relevant cases meaninglessly into assrender C plugin code)
AVS_VideoFrame* AVSC_CC assrender_get_frame(AVS_FilterInfo* p, int n)
{
udata* ud = (udata*)p->user_data;
ASS_Image* img;

int64_t ts;
int changed;

// AVS_FilterInfo* p from get_frame
AVS_VideoFrame* src = avs_get_frame(p->child, n);

avs_make_writable(p->env, &src);

// src is an AVS_VideoFrame*
AVS_Map* avsmap;
avsmap = avs_get_frame_props_rw(p->env, src);
int error; // needed for error report, for now we'll ignore it
// read existing propery. For test, we set it by to 77 by propSet("TestIntKey", 77) in Avisynth script
int64_t testvalue = avs_prop_get_int(p->env, avsmap, "TestIntKey", 0, &error);

// set new frame properties (by using the just-read integer property)
avs_prop_set_int(p->env, avsmap, "TestIntKey2", testvalue * 2, AVS_PROPAPPENDMODE_REPLACE);
avs_prop_set_float(p->env, avsmap, "TestFloatKey2", (testvalue * 2.0f), AVS_PROPAPPENDMODE_REPLACE);
// store string (in general: any data), by specifying length = -1 we'll notify set_data to have strlen for getting its real size
avs_prop_set_data(p->env, avsmap, "TestStringKey2", "testStringvalue", -1, AVS_PROPAPPENDMODE_REPLACE);

// array test (double). Note: we can have double here, contrary to the fact that Avisynth scripts can handle only floats.
const double test_d_array[] = { 0.5, 1.1 };
avs_prop_set_float_array(p->env, avsmap, "TestFloatArray", test_d_array, 2);

// array test (integer). Note: we can have int64 here, contrary to the fact that Avisynth scripts can handle only 32 bit integers.
const int64_t test_i_array[] = { -1, 0, 1 };
avs_prop_set_int_array(p->env, avsmap, "TestIntArray", test_i_array, 3);

// read back the array size of a property (single properties are an 1-element arrays)
int numElementsOfIntArray = avs_prop_num_elements(p->env, avsmap, "TestIntArray");
avs_prop_set_int(p->env, avsmap, "TestNumElementsOfIntArray", numElementsOfIntArray, AVS_PROPAPPENDMODE_REPLACE);

// Int property array: read back one-by-one, mul by 2, and put into another array
int64_t test_i_array_clone[3];
for (auto i = 0; i < numElementsOfIntArray; i++) {
test_i_array_clone[i] = avs_prop_get_int(p->env, avsmap, "TestIntArray", i, &error) * 2;
}
avs_prop_set_int_array(p->env, avsmap, "TestIntArrayCloneMul2", test_i_array_clone, 3);

// double property array read back as a whole, div by 3, and put into another array
int numElementsOfFloatArray = avs_prop_num_elements(p->env, avsmap, "TestFloatArray");
const double * tmpdarray = avs_prop_get_float_array(p->env, avsmap, "TestFloatArray", &error);
for (auto i = 0; i < numElementsOfFloatArray; i++) {
double tmp_d = tmpdarray[i] / 3.0;
// first element: replace frameprop data, next ones: append new element one by one
avs_prop_set_float(p->env, avsmap, "TestFloatArrayCloneDiv3", tmp_d, i == 0 ? AVS_PROPAPPENDMODE_REPLACE : AVS_PROPAPPENDMODE_APPEND);
}

// delete the key, we defined in Avisynth script
avs_prop_delete_key(p->env, avsmap, "TestIntKey");

// count all keys (frame property count) and put it into another frame property
int numOfKeys = avs_prop_num_keys(p->env, avsmap);
avs_prop_set_int(p->env, avsmap, "TestNumKeysWithoutThisOne", numOfKeys, AVS_PROPAPPENDMODE_REPLACE);

// [... code part left out intentionally]

return src;
}

[pinterf]

Could Expr be updated to treat newlines and tabs (\r, \n, \t, and any other character less than 32 for that matter) inside expressions exactly the same as spaces? It does odd and slightly unpredictable things at the moment - it seems to work okay if a space is put before the newline and if the first thing on the next line is a number, but otherwise it complains about being unable to convert to float.
Probably yes. If it does not collide with existing Avisynth syntax. I can imagine that it can work like e.g. atoi
which discards whitespace (whitespace = detected by isspace (https://www.cplusplus.com/reference/cctype/isspace/)) as well.

Can you explicitely specify a string that has problems?
You can use the 'e' syntax for string literals which can be seen here: http://avisynth.nl/index.php/The_full_AviSynth_grammar
escaped_string = e"Hello \n"
with e prefix right before the quotation mark will store actual control character into the string
Converted literals:
\n to LF-Chr(10)
\r to CR-Chr(13)
\t to TAB-Chr(9)
\0 to NUL-Chr(0)
\a to Chr(1)-alert/beep
\f to FF-Chr(6)
\\ (double \) to Backslash
\" to " (double-quotation mark)

[pinterf]

@Reel.Deel:
An Avisynth wiki request: could you please put a proper internal link into the section http://avisynth.nl/index.php/Script_variables | Variable types | string
to link to http://avisynth.nl/index.php/The_full_AviSynth_grammar | Literals | escaped_string example.
(I was not able to figure out how to do it in a nice way)

[wonkey_monkey]

Can you explicitely specify a string that has problems?


All of the following fail (no trailing spaces):

expr("
x")

expr("100
x +")

expr("x
100 +")


The last one (and only the last one) can be made to work if a space is added after the 'x'.

[pinterf]

All of the following fail (no trailing spaces):

expr("
x")

expr("100
x +")

expr("x
100 +")


The last one (and only the last one) can be made to work if a space is added after the 'x'.
Thanks for the examples, request registered.

[wonkey_monkey]

Something else I find odd about expr2 is that (optSSE2 = false, optAVX2 = true) does not use AVX2. I assume the two options are somewhat mutually exclusive, in that it never uses both at the same time, in which case maybe a single parameter to specify a level of optimisation - 0 for none, 1 for SSE2, 2 for AVX2 - would make more sense.

[pinterf]

Something else I find odd about expr2 is that (optSSE2 = false, optAVX2 = true) does not use AVX2. I assume the two options are somewhat mutually exclusive, in that it never uses both at the same time, in which case maybe a single parameter to specify a level of optimisation - 0 for none, 1 for SSE2, 2 for AVX2 - would make more sense.
optXXXXX-like parameters are primarily debug parameters, they can appear, disappear, change meaning, etc. In this specific case this is how it works, when I was simply put optSSE2 = false then I wanted to test plain C code.

[StainlessS]

escaped_string = e"Hello \n"
with e prefix right before the quotation mark will store actual control character into the string
Converted literals:
\n to LF-Chr(10)
\r to CR-Chr(13)
\t to TAB-Chr(9)
\0 to NUL-Chr(0)
\a to Chr(1)-alert/beep
\f to FF-Chr(6)
\\ (double \) to Backslash
\" to " (double-quotation mark)


P, above looks odd to me, the two in red show,
"\a" converted to Chr(1) [ie SOH, Start Of Header] instead of Chr(7) [ie BEL, aka alert or beep, make a printer go "ding!"]
"\f" converted to Chr(6) [ie ACK, Acknowledge] instead of Chr(12) [ie FF or FORMFEED, feed printer to next page].
[EDIT: Looks like youve used '\a' = CTRL/1, '\f' = CTRL/6, where 'a' is first letter of alphabet and 'f' is 6th.]
[EDIT: Above in RED is wrong on wiki, see 3 posts after this one. Some of below assumes that AVS works as on wiki.]

Is there some reason for this ?

Also, I've personally found another that should maybe be intercepted, ie single '\' at end of line, where I personally copy it ('\') explicitly,
ie end the line with a backslash [but maybe it should be swallowed, ie removed from output line].

[EDIT:
Reason for copy rather than omit, is to show user that maybe erroneous backslash is in the source string, rather than hide it,
I also copy any other non reconised escaped chars verbatim including preceding '\'.
I also process '\v' and '\b' where not mentioned on wiki

while(c=*r) {
if(c=='\\') {
++r;
c=*r;
// abfnrtv
switch (c) {
case '\0' : *p++='\\'; break; // copy single backslash at end of string
case '\\' : *p++=*r++; break; // replace double backslash with single backslash
case 'n' : ++r; *p++='\n'; break; // '\n' = chr(10)
case 'r' : ++r; *p++='\r'; break; // '\r' = chr(13)
case 't' : ++r; *p++='\t'; break; // '\t' = chr(9)
case 'v' : ++r; *p++='\v'; break; // '\v' = chr(11)
case 'f' : ++r; *p++='\f'; break; // '\f' = chr(12)
case 'b' : ++r; *p++='\b'; break; // '\b' = chr(8)
case 'a' : ++r; *p++='\a'; break; // '\a' = chr(7)
default : *p++='\\'; *p++=*r++; break; // anything else we copy backslash and whatever follows
}
} else {
*p++=*r++;
}
}
*p=0; // nul term

END EDIT:]

In my given script function, I also considered BACKSPACE ['\b', Chr(8)] as whitespace, maybe I was wrong on that [was done from memory, and mine dont have parity/ECC :) ].

Also, "\0 to NUL-Chr(0)", will presumably end the string prematurely if there are following chars in string, which maybe should also have warning on Wiki.

EDIT: I personally know nothing about locales and the like, makes me feel sick just to think that, that stuff exists.

EDIT: In RT_String, RT_Subtitle, RT_WriteFile, format string, I convert substring "\a" to BEL Chr(7), so I could not use AVS escaped strings to do the same,
in RT_Subtitle, I use single code '\a' Chr(7) BEL character as a color control code switch, eg substring "\a!" = switch to Hi-lited text, and "\a-" = normal text, + others for different colors. [Single char code '\a' Chr(7) BEL, is a color switch introducer or preamble code, with actual color switched by single char code eg '!' being hilite, and '-' being normal text
(and other color codes any one of these "0123456789ABCDEFGHIJKLMNOPQRSTUV" + a few more)].

EDIT:
A bit O.T.
[EDIT: Looks like youve used '\a' = CTRL/1, '\f' = CTRL/6, where 'a' is first letter of alphabet and 'f' is 6th.]
MI5, Stands for Military Intelligence England {well UK}, 5 = "E" = 5th letter of alphabet.
MI6, Stands for Military Intelligence Foreign {not UK}, 6 = "F" = 6th letter of alphabet. [James Bond works for MI6]
Combat18, Stands for Combat A.H. = Combat Adolf Hitler. {I'm not a member - membership refused for being too radical}
End OT.

[gispos]

AnsiString stuff: internally you can use String and not AnsiString.
Then it can be converted to utf8 if needed.
var unistring: String := ...
var Bytes: TBytes;
Bytes := TEncoding.UTF8.GetBytes(unistring);


Import has also an alternative form for utf8 filenames (if windows codepage is not utf8 natively): Import(filename [, ...] [, bool utf8])
then pass this buffer to Eval if Windows system codepage is set to UTF-8.

As for frame properties: there is a fix in these post-3.7 version which does not crash (yes, there was a bug) when using frame property related functions from C.
Thanks for the answer.
I've already tried various things, with and without a buffer.
As a string with a pointer with @ as PByte as an Array as PChar with and without EncodeUTf8 and none of that worked.
As soon as MCTemporalDenoise() is present in the script, the error message division through 0 appears, I don't know why.
At the moment I don't have any energy to test the whole thing again, someday...

What about the AvsPmod Utf-8 problem, do you have a solution for it?
The script is Utf-8 encoded and is passed with 'Eval', but Avisynth may not accept that.
If I save the script and open the script in a new script with AviSource, there are no problems.

[pinterf]

What about the AvsPmod Utf-8 problem, do you have a solution for it?
The script is Utf-8 encoded and is passed with 'Eval', but Avisynth may not accept that.
What does it mean 'may not accept'?.
If the script text itself is converted to utf8, you have to ensure that it is saved _without_ BOM.
If
- the script is accepted
- win10 codepage is set to utf8
then it will show utf8-encoded strings in Subtitle, open utf8-encoded filenames without specifying utf8=true for methods which accept such parameters.

[zorr]

Thanks again. I'm running MVTools tests again and looks like it could take a couple of days until they're finished, I'll test after that.

Those tests took a bit longer than I expected... but I have now tested the clip type frame property read functionality. IT WORKS! :D

I made myself little helper functions to store and restore clips into/from other clips.

function storeClip(clip c, clip stored, string name) {
return propSetClip(c, name, function[stored](clip c) { return stored } )
}

function restoreClip(clip c, string name) {
return ScriptClip(c, function[name]() { return propGetClip(name) })
}


With these you can do for example

clipA = clipA.storeClip(clipB, "storedB")
and later read the stored clip with

clipB = clipA.restoreClip("storedB")

I wasn't able to make clips with with arrays though. Perhaps I didn't get the syntax right but this

return ScriptClip(clipA, function[]() { return propGetAsArray("storedB")[0] })

returns ScriptClip: Function did not return a video clip! (Was the undefined value) if I store an array of clips and ScriptClip: Function did not return a video clip! (Was a float) if I store an array of floats (this just to demonstrate that I'm able to store something into the array).

I stored two clips using

clipA = propSet(clipA, "storedB", function[clipB](clip c) { return clipB } )
clipA = propSet(clipA, "storedB", function[clipB](clip c) { return clipB }, mode=1 )


Trying to do the same using

clipA = ScriptClip(clipA, function[](clip c) { propSetArray("storedB",function[](clip c) { return [clipB, clipB] } ) })
results in AddProperties: Error in Function: defined at ... This same syntax works for storing floats though:

clipA = ScriptClip(clipA, function[](clip c) { propSetArray("storedB",function[](clip c) { return [1.0, 1.1] } ) })

[StainlessS]

OK, in post #1142 [2 previous : EDIT: 3 previous] the wiki is in error.

escaped_string = e"Hello \n"
with e prefix right before the quotation mark will store actual control character into the string
Converted literals:
\n to LF-Chr(10)
\r to CR-Chr(13)
\t to TAB-Chr(9)
\0 to NUL-Chr(0)
\a to Chr(1)-alert/beep
\f to FF-Chr(6)
\\ (double \) to Backslash
\" to " (double-quotation mark)

'\a' should map to Chr(7), BEL, aka ALERT, BEEP, and '\f' to Chr(12), FF, FORMFEED.


BlankClip(width=320,height=240)

s=e"\a\t\n\f\r" # OK conversions
#s=e"\b\v\" # Not OK conversion, \b an \v not supported/converted : NOTE Final LONE BackSlash

Z=""
For(i=1,strlen(s)) {
c = MidStr(s,i,1)
n = ord(c)
q = qfn(n)
Z = RT_string("%s%d ] %d '%s'\\n",Z, i,n, q) # This appends to existing Z [a bit less string mem usage than Z=Z+String]
}
Subtitle(Z,lsp=0,font="Courier New")

Return Last

Function qfn(int n) {
if(7 <= n <= 13) { return Select(n-7,"\a","\b","\t","\\n","\v","\f","\r") } # '\n' -> '\\n' avoid subtitle prob
else { return chr(n) }
}


s=e"\a\t\n\f\r" # OK conversions
https://i.postimg.cc/9MczcJdh/q-01.jpg (https://postimages.org/)


Also, I've found another that should maybe be intercepted, ie single '\' at end of line, where I personally copy it ('\') explicitly,
ie end the line with a backslash [but maybe it should be swallowed, ie removed from output line].

Uncommenting this line

s=e"\b\v\" # Not OK conversion, \b an \v not supported/converted : NOTE Final LONE BackSlash

s=e"\b\v\"
https://i.postimg.cc/X7rqNZCp/q-01.jpg (https://postimages.org/)

Note, in above image, the final '\' BackSlash in line has actually escaped the closing double quote, and made it part of result string,
which it really should not.
EDIT:
\" to " (double-quotation mark)
It surely cannot mean that the double quote which marks extent of the string can be made part of the string ???
\" to double quote, should only work inside triple quoted strings to escape a single double quote which does not belong
to end triple quotes.

EDIT:
s=e"\b\v\"z" # NOTE, produces syntax error in script (hence triple quote requirement, or parser fix)

But this works
s=e"""\b\v\"z""" # Escaped double quote, must enclose in triple double quotes
https://i.postimg.cc/3wfJHYxg/q-02.jpg (https://postimages.org/)

EDIT: and this, premature end of string [needs mention on wiki for "\0"].
s=e"\t\0something missing"
https://i.postimg.cc/NjBg9bJC/q-03.jpg (https://postimages.org/)

The "\b" and "\v" conversions are I guess not necessary, only to match and complete those available in C,
In RT_Stats, I do convert "\b" and "\v", but only actually use Chr(8) ["\b"] in RT_Subtitle, Chr(11) ["\v"] being as yet the last spare code.

Also, not really sure that you need """\"""", as if it passes script syntax, you probably dont need to escape it (double quote) to begin with.
And "\0", is of dubious use [unless has some purpose for UTF8 or something like that]

[gispos]

What does it mean 'may not accept'?.
If the script text itself is converted to utf8, you have to ensure that it is saved _without_ BOM.
If
- the script is accepted
- win10 codepage is set to utf8
then it will show utf8-encoded strings in Subtitle, open utf8-encoded filenames without specifying utf8=true for methods which accept such parameters.
A video file with Utf-8 characters in the file name is opened with AvsPmod. AvsPmod has no problem with that. It is now written there
LWLibavVideoSource ("Это тестовый Video.mkv")
As I wrote, with 'Eval' I get an error message, I save this script and open it with AviSource there are no problems

[wonkey_monkey]

I'd like to compile Expr as a standalone filter (with a new name) DLL for my own education/amusement. Is there a simple (or even not-so-simple) way to do this? I've used cmake to make a Visual Studio project for the complete 3.7.0 and managed to compile it (the whole of Avisynth into one DLL) without any problems but I don't really know what to do next. Any hints?

[pinterf]

OK, in post #1142 [2 previous : EDIT: 3 previous] the wiki is in error.

Thanks, fixed in wiki (\a and \f byte code)
EDIT and added \b \v and \'

[pinterf]

AviSynth+ 3.7.1-test7 20210630 (https://drive.google.com/uc?export=download&id=1naR4I4tm-EifZIdbkS9M1zhtS9_2wL19)
Changes since test6
- Recognize \' and \b and \v in escaped (e"somethg") string literals (see http://avisynth.nl/index.php/The_full_AviSynth_grammar#Literals)
- Expr: allow TAB, CR and LF characters as whitespace in expression strings
- Clip content support for propGetAsArray and propGetAll


(no propSetArray changes in this version)

[kedautinh12]

Thanks

[StainlessS]

Tanks P, I'll give test7 a whirl :)

[wonkey_monkey]

pinterf,

I've been thinking long and hard about Expr for the last few days, because it's so much quicker than the RPN compiler I've been working on for a while. It does lack a number of features that I have found very useful, though - not that I'm asking you to do anything, Expr is excellent enough as it is and you've already been kind enough to implement a few things at my suggestion, but I just thought it might be worth going over my ideas in public in case of it's any interest to anyone.

As I use RPN for a lot of things, a standalone RPN compiler is something of a necessity for me. But before I go off in my own direction - taking plenty of inspiration from Expr, if not actual code; anything I eventually release will be sure to be open source though - I just wondered about your potential interest in such a project, as in whether it would be something that could be usefully developed in parallel with future iterations of Expr (and/or used to create new programmable features of Avisynth).

On the one hand it seems silly to duplicate so much effort, when Expr already shares so much in common with my aims. But on the other hand, what I'm envisioning would also be quite different:


A standalone RPN compiler, which has a concept of input and output "planes" but only as generic data sources, with width, height, (and possibly additional dimensions), type (byte, short, int, float, double) and pitch. Bit depth conversions and colour channel bias would be left for the calling function to inline as boilerplate RPN ops (more or less what Expr already does, just abstracted one more level)
The ability to do per-clip, per-frame, per-line, and per-pixel calculations (storing variables). Per-clip would probably just be constant folding since there'd be no changeable input. Per-line might be easy to implement directly in Expr - it's just a matter of moving the start of the loop and making sure the stack is empty at the start of the loop. Per-frame is trickier. Per-pixel only applies if the output channels are calculated in an interleaved way, which is natural for packed RGB(A) but not so natural for YUV(A) (you have to jump around between the output planes)
Internal multithreading
Choice of float or double precision (my current compiler uses the x87 stack which offers 80-bit precision - not available with SIMD unfortunately)
Trig functions implemented with SIMD (based on ssemath or some other library), floor, ceil, bitwise operators and so on
Automatic type switching between int and float
Option to return black out-of-bounds pixels instead of repeating edges
Something more akin to true conditional execution - difficult with SIMD but not impossible to emulate - rather than just the basic a/b swap ternary operator
Internal multi-dimensional arrays - build your own LUT with per-clip/per-frame calculations!
Run-time calculable pixel offsets
Support for other planes (Y in U, G in Y, etc) to be referenced (if they're the same size). Allow non-matching input clips as long as their non-matching planes aren't referenced
Temporal offsets to access past/future frames
Inlined Avisynth variables
Support for comments (a simple regex can remove several styles)


The last four I plan to implement first as an Expr wrapper plugin, as a separate/proof-of-concept project.

Hmm... you know, now that I write this out it does seem like a bit of a monster. Still, it would be just as helpful to hear if it's of no real interest to you - no offence will be taken! I'd also welcome your thoughts on the subject, or if anyone else has any feature requests or hints.

PS Thanks for everything you do!

[pinterf]

Meanwhile VapourSynth's Expr got sin and cos (https://github.com/vapoursynth/vapoursynth/pull/693) functions.
This is present. And the future: I recommend you having a look at this exciting Expr implementation which may be the future of Expr.
https://github.com/AkarinVS/vapoursynth-plugin
Not specific to Intel, uses LLVM engine, integer option.

[kedautinh12]

Wow sounds good

[Dogway]

Had a similar post in my mind for a few days but since wonkey_monkey shared his thoughts I will share mine as well.

After fiddling with Expr for some months now I observed that min and max (maybe clip too?) operators are very slow, slower than masktools2, so I might think there's room for optimization on that side.

Obviously pixel addressing further acceleration is desired, but also LUT calculations for 8-bit as pinterf is already working on. On this regard I wondered also an option to define the type of calculation of the expression (float, double, int) to speed up the expression or process with more precision.

Another thought was implementation of absolute pixel addressing, although I presume this wouldn't be very optimized.

One observation I found on the Expr() documentation is the following example:
"x[-1,-1] x[-1,0] x[-1,1] y[0,-10] + + + 4 /"

As I could find this is very unoptimized, Expr() prefers continuous calculations on the stack, so:
"x[-1,-1] 0.25 * x[-1,0] 0.25 * + x[-1,1] 0.25 * + y[0,-10] 0.25 * +"

In this example I also change division by 4 with multiplication by 0.25. I read that in expressions the product is faster than the division but not for all divisions, like for example even integers, is this true? For the time being I'm replacing all divisions with the product of the reciprocal.

Finally I'm starting to use optSingleMode=true for lutxyz operations as I saw an increment in performance, but sometimes if the expression is not complex enough performance is lower. Any word on this is appreciated since the documentation is not very clear.

Anyway thanks for the help, I feel like I opened pandora's box lol.

[real.finder]

Meanwhile VapourSynth's Expr got sin and cos (https://github.com/vapoursynth/vapoursynth/pull/693) functions.
This is present. And the future: I recommend you having a look at this exciting Expr implementation which may be the future of Expr.
https://github.com/AkarinVS/vapoursynth-plugin
Not specific to Intel, uses LLVM engine, integer option.

AkarinVS said in https://github.com/AkarinVS/vapoursynth-plugin/releases/tag/v0.60

You can use this to implement arbitrary convolution kernels (i.e. non-regular shapes), and my benchmark indicated that 3x3 convolution implemented this way is as fast as std.Convolution.

Dogway said that vs Convolution is fast, that mean Pixel addressing will kill vs Convolution? don't know if we can made Expr version for http://www.vapoursynth.com/doc/functions/boxblur.html and have same speed, or boxblur will need backport

[wonkey_monkey]

One observation I found on the Expr() documentation is the following example:
"x[-1,-1] x[-1,0] x[-1,1] y[0,-10] + + + 4 /"

As I could find this is very unoptimized, Expr() prefers continuous calculations on the stack, so:
"x[-1,-1] 0.25 * x[-1,0] 0.25 * + x[-1,1] 0.25 * + y[0,-10] 0.25 * +"

I found the first to be slightly faster than the second, and faster still when I replaced "4 /" with "0.25 *". I haven't dug too deep into Expr code yet, but I don't think having a few items on the stack will cause a slowdown. It may not even evaluate those pixel references until they're required by an operation anyway.

In this example I also change division by 4 with multiplication by 0.25. I read that in expressions the product is faster than the division but not for all divisions, like for example even integers, is this true? For the time being I'm replacing all divisions with the product of the reciprocal.

I would expect multiplication by a reciprocal to be faster in pretty much all cases. Floating point reciprocals are accurate for powers of 2, but may not be perfectly accurate otherwise. 1/3, for example, can't be precisely represented by an IEEE-754 floating point number. But it's unlikely you'll bump into any rounding errors anyway, certainly not with a simple script like this one.

[kedautinh12]

I remember Dogway have ex_boxblur in extools
https://github.com/Dogway/Avisynth-Scripts/blob/2ef63fbb4fa97c8657a3a61461c03d0467133e3d/ExTools.avsi#L392

[real.finder]

I remember Dogway have ex_boxblur in extools
https://github.com/Dogway/Avisynth-Scripts/blob/2ef63fbb4fa97c8657a3a61461c03d0467133e3d/ExTools.avsi#L392

yes, seems I forget it

import vapoursynth as vs
core = vs.get_core(threads=1)
import mvsfunc as mvf
clip=core.lsmas.LWLibavSource(source=r'test.avi')
clip=mvf.GetPlane(clip, 0)
clip=core.std.BoxBlur(clip,hradius =4,vradius =4)
clip.set_output()

about 207 fps and with get_core() (mt) its about 640 fps


LWLibavVideoSource("test.avi")
#RequestLinear(clim=100)
ConvertToY
ex_boxblur(4)
#Prefetch()


about 328 fps and about 854 fps with RequestLinear(clim=100) and Prefetch()

[wonkey_monkey]

I get 308fps from ex_boxblur(4), but 340fps if I modify ex_boxblur so the Expr strings use one trailing multiply instead of multiplying each pixel separately.

[Dogway]

I found the first to be slightly faster than the second.
I think you are right, I might have mixed my benchmarks with the per pixel reciprocal product, will test further. As a side note I found at least for these synthetic benchs that convolutions perform best with Prefetch(physical cores) than with threads.


ex_boxblur(1) is actually very fast, 87% (maybe more with above note) of removegrain(19) but with the benefit of being variable. The problem is the same for all pixel addressing convolutions which are limited to SSSE3, I don't know about VS acceleration on boxblur or Convolution() (benchmarks welcome) but if an internal boxblur in AVS+ can be implemented I think it could improve performance. In any case I don't think this to be a priority, I would rather improve Expr() or implement functions not doable in Expr() like mt_hysteresis().

I also thought on a sandbox function that operates on double float for algebra operations not involving clips, so I can do all the derivations with double float and ultimately feed that to Expr() which -for the moment- operates on single float.


EDIT: some benchs over 16-bit and Prefetch(4). Moved to ExTools thread (https://forum.doom9.org/showthread.php?p=1946603#post1946603)

[wonkey_monkey]

Perhaps we should move this to your thread!

[zorr]

- Clip content support for propGetAsArray and propGetAll

(no propSetArray changes in this version)

Thanks, I tested propGetAsArray and now it works with clips.

[wonkey_monkey]

A long, long time ago I pointed out that BicubicResize, with its default parameters, blurs when (ideally) it shoud not. The principle of least surprise dictates that a resizer should not do anything to the original pixels if it's called as a "null" resizer (e.g. resizing to the exact width and height).

For compatibility reasons this was not changed, however a shortcut which bypassed resizing entirely was, if I recall correctly, removed, so that a "null" resize would still perform the resize. This was to keep it consistent with non-null resizes.

This seems to have been reverted at some point, as this script shows if you step through:

version
shifted = bicubicresize(width, height, src_left=0.00001, src_top = 0.00001) # a very minor pixel shift to avoid the "null-resize" skip code
unshifted = bicubicresize(width, height, src_left = 0, src_top = 0) # this should be nearly identical to the above, but it isn't
interleave(unshifted, shifted)
pointresize(width*4, height*4) # you may need to zoom in to see the difference


The two clips should be practically identical, being only shifted by a fraction of a pixel, but the blur/non-blur versions are distinguishable.

Personally I still think the default b and c parameters should be changed to 0 and 0.5 respectively to remove the blur entirely (backwards compatability be damned! ;) ), but could the "skip" be removed again to remove this discontinuity?

[pinterf]

After fiddling with Expr for some months now I observed that min and max (maybe clip too?) operators are very slow, slower than masktools2, so I might think there's room for optimization on that side.

In Expr _all_ operations are done on solely 32 bit floats. Unlike masktools' lut expressions which operates on 64 bit doubles. But masktools has no internal JIT compiler like Expr, so if the use case is not a LUT (which evaluates the expression N times for filling the lut table once then use this lookup for all frames) then is slow like hell. It's because if the lut table is too large in masktools (imagine a 16 bit xy lookup table which requires 8 GB RAM) then it switches to 'realtime' working mode - in masktools this is not accelerated.

- All inputs are immediately converted to float, 8-16 bit integers. Float pixel type is not converted during load.
- Integer constansts are already stored as floats (no extra time during the evaluation)
- Constants and operators on constants are evaluated during the preprocessing phase so 2 + 2 is not evaluated for each frame because it is stored as 4.
- Expression then is evaluated using 32 bit float instructions
- Output is then stored; either as-is (float pixel format) or after rounding + clamp to valid range + convert back to 8-16 bit integer format

Helper options which do input auto-scaling to 8 (or whatever) bit internal format then back, have an extra overhead during pixel-read and pixel store.

All these auto-scale things (both in masktools and Avisynth Expr) are 'convenience' functions which were requested during the beginning of HBD transition era. They may not provide a fully speed optimized script but helped in translating a lot of 'old 8 bit compatible' scripts for 10+ bits environment.

Note that other auto-scaler keywords (such as 255 scalef) mean no extra burden on execution time because of the preprocessor constant folding (for a 10 bit clip 255 scalef is translated to 255 * (1023.0f/255.0f) that is "255 4.011764705882353 *" in RPN, and finally converted into single constant of 1023 - all this is optimized out in the parsing/preparation phase.

VapourSynth expressions and scripts are solving the high bit depth problem by manually providing scaled constants into the scripts.


On this regard I wondered also an option to define the type of calculation of the expression (float, double, int) to speed up the expression or process with more precision.

Optimizing the expression into assembler code is a handcrafted task in Expr. Now it is using 32 bit float instruction set. How the actual operators are translated into SSE2/AVX2 mnemonics - assembler code - is more or less hardcoded. Using 64 bit doubles (which I have tried once but abandoned due to its extra complexity) requires rewriting everything once more.
And we are still not talking about mixing the types together.

This is why the above mentioned LLVM approach is exciting, their group of developers probably spent years on the background optimization technology.


One observation I found on the Expr() documentation is the following example:
"x[-1,-1] x[-1,0] x[-1,1] y[0,-10] + + + 4 /"

As I could find this is very unoptimized, Expr() prefers continuous calculations on the stack, so:
"x[-1,-1] 0.25 * x[-1,0] 0.25 * + x[-1,1] 0.25 * + y[0,-10] 0.25 * +"

In this example I also change division by 4 with multiplication by 0.25. I read that in expressions the product is faster than the division but not for all divisions, like for example even integers, is this true? For the time being I'm replacing all divisions with the product of the reciprocal.

Float multiplication is quicker on all processor architectures.
_mm_mul_ps (https://software.intel.com/sites/landingpage/IntrinsicsGuide/#expand=5080,3538,4676,348,5538,5595,5080,3573,3556,5316,3077,3928&text=_mm_mul_ps) and _mm_div_ps (https://software.intel.com/sites/landingpage/IntrinsicsGuide/#expand=5080,3538,4676,348,5538,5595,5080,3573,3556,5316,3077,3928,2156&text=_mm_div_ps) (and their 256 bit equivalents) are used.



Finally I'm starting to use optSingleMode=true for lutxyz operations as I saw an increment in performance, but sometimes if the expression is not complex enough performance is lower. Any word on this is appreciated since the documentation is not very clear.

Anyway thanks for the help, I feel like I opened pandora's box lol.
Expr (running a machine code generated by its JIT compiler) is using multimedia registers (128 bit XMM registers for SSE2 or 256 bits YMM registers for AVX2) for the internal operations.
A 128 bit register can hold 4 pixels (4x32 bit float)
A 256 bit register can hold 8 pixels (8x32 bit float)

Originally Expr is handling two sets of such registers in a parallel way, so it loads 2x4 consecutive pixels into two registers (or 2x8 pixels for AVX2) then performs the same operations on both 'lanes'.

Then the results in the two registers are converted back and stored into 2x4 or 2x8 bytes (8 bit pixel type) or 2x8/2x16 bytes (16 bit pixel types)

optSingleMode option is loading/using/storing only a single register intead of the above mentioned dual two-lane method.

I made the optSingleMode option because I saw in the JIT generated code that using two parallel register sets are not always optimal, complex scripts will turn into heavy register-to-memory swapping.

If the expression is complex that is the actual RPN stack is deep, intermediate calculations cannot be fit into available registers. On 32 bit we have only eight registers: XMM0..XMM7. The default dual-lane mode is using them in pairs, so effectively we can use only 4 of them. When the depth of expression/internal stack is over 3 then we are out of available registers. The currently unused registers (holding the result of an earlier calculation) will be swapped into memory in order to allow loading data for the new operations. Then swapped back from memory.
Memory access has cost.

By the fact that optSingleMode needs practically half of the registers than the default working method, it is effectively allowing twice as much playground in the expression evaluation depth before the swap-unused-registers-to-memory event kicks in.

In 64 bit the situation in not that bad, we have extra registers there, it allows more complex expressions w/o penalty compared to 32 bit x86 code.

[feisty2]

pinterf,

I've been thinking long and hard about Expr for the last few days, because it's so much quicker than the RPN compiler I've been working on for a while. It does lack a number of features that I have found very useful, though - not that I'm asking you to do anything, Expr is excellent enough as it is and you've already been kind enough to implement a few things at my suggestion, but I just thought it might be worth going over my ideas in public in case of it's any interest to anyone.

As I use RPN for a lot of things, a standalone RPN compiler is something of a necessity for me. But before I go off in my own direction - taking plenty of inspiration from Expr, if not actual code; anything I eventually release will be sure to be open source though - I just wondered about your potential interest in such a project, as in whether it would be something that could be usefully developed in parallel with future iterations of Expr (and/or used to create new programmable features of Avisynth).

On the one hand it seems silly to duplicate so much effort, when Expr already shares so much in common with my aims. But on the other hand, what I'm envisioning would also be quite different:


A standalone RPN compiler, which has a concept of input and output "planes" but only as generic data sources, with width, height, (and possibly additional dimensions), type (byte, short, int, float, double) and pitch. Bit depth conversions and colour channel bias would be left for the calling function to inline as boilerplate RPN ops (more or less what Expr already does, just abstracted one more level)
The ability to do per-clip, per-frame, per-line, and per-pixel calculations (storing variables). Per-clip would probably just be constant folding since there'd be no changeable input. Per-line might be easy to implement directly in Expr - it's just a matter of moving the start of the loop and making sure the stack is empty at the start of the loop. Per-frame is trickier. Per-pixel only applies if the output channels are calculated in an interleaved way, which is natural for packed RGB(A) but not so natural for YUV(A) (you have to jump around between the output planes)
Internal multithreading
Choice of float or double precision (my current compiler uses the x87 stack which offers 80-bit precision - not available with SIMD unfortunately)
Trig functions implemented with SIMD (based on ssemath or some other library), floor, ceil, bitwise operators and so on
Automatic type switching between int and float
Option to return black out-of-bounds pixels instead of repeating edges
Something more akin to true conditional execution - difficult with SIMD but not impossible to emulate - rather than just the basic a/b swap ternary operator
Internal multi-dimensional arrays - build your own LUT with per-clip/per-frame calculations!
Run-time calculable pixel offsets
Support for other planes (Y in U, G in Y, etc) to be referenced (if they're the same size). Allow non-matching input clips as long as their non-matching planes aren't referenced
Temporal offsets to access past/future frames
Inlined Avisynth variables
Support for comments (a simple regex can remove several styles)


The last four I plan to implement first as an Expr wrapper plugin, as a separate/proof-of-concept project.

Hmm... you know, now that I write this out it does seem like a bit of a monster. Still, it would be just as helpful to hear if it's of no real interest to you - no offence will be taken! I'd also welcome your thoughts on the subject, or if anyone else has any feature requests or hints.

PS Thanks for everything you do!

in that case you should just write a normal C++ plugin, which will be way easier to read, to write and to maintain than ultra complex RPN expressions.

[pinterf]

A long, long time ago I pointed out that BicubicResize, with its default parameters, blurs when (ideally) it shoud not. The principle of least surprise dictates that a resizer should not do anything to the original pixels if it's called as a "null" resizer (e.g. resizing to the exact width and height).

For compatibility reasons this was not changed, however a shortcut which bypassed resizing entirely was, if I recall correctly, removed, so that a "null" resize would still perform the resize. This was to keep it consistent with non-null resizes.

This seems to have been reverted at some point, as this script shows if you step through:

...but could the "skip" be removed again to remove this discontinuity?

The omission is here:
https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/filters/resample.cpp#L709
and
https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/filters/resample.cpp#L740

This is the specific commit (https://github.com/AviSynth/AviSynthPlus/commit/47758e185af0205c38f846a8428b9807f8da8a13).

The behaviour didn't change since then.

It keeps returning the original clip in only one case: when everything is unchanged.

[pinterf]

A video file with Utf-8 characters in the file name is opened with AvsPmod. AvsPmod has no problem with that. It is now written there
LWLibavVideoSource ("Это тестовый Video.mkv")
As I wrote, with 'Eval' I get an error message, I save this script and open it with AviSource there are no problems
Could you please send me the exact files mentioned here?
The file you are opening with avspmod.
The saved file you are opening with AviSource.
(But no need for the Это тестовый Video.mkv, I can create the file with such a name for myself), thanks

[wonkey_monkey]

in that case you should just write a normal C++ plugin, which will be way easier to read, to write and to maintain than ultra complex RPN expressions.

I think you've misunderstood. My bullet points are a list of features I plan on implementing, or have already implemented, in a C++ RPN-compiling plugin. I don't plan on implementing them with RPN. That'd be crazy.

It keeps returning the original clip in only one case: when everything is unchanged.

Well... I'm not sure it should do that either. If you use animate BicubicResize to scroll something, or do a zoom in, there'll be an inconsistency if one frame is uncropped and unresized:

version
animate(0,16, "bicubicresize", width,height,4,4,-16.0,-16.0, width,height,4,4,16.0,16.0)

See frames 7,8,9.

[feisty2]

no, what you're trying to do is insane. your proposed Expr filter is no longer an expression evaluating filter, but rather an exotic DSL.
you can theoretically implement extremely complex filters like mvtools using this so called Expr, so basically you're creating yet another DSL inside a DSL (the avisynth script itself is already a DSL), and that's very harmful, especially considering that the RPN expression is a rather exotic language.

you best bet is to directly extend the avisynth scripting language with frame access, pixel access, ..., whatever feature you planned for your Expr DSL. at least avisynth is a readable DSL.


I think you've misunderstood.

you're the one who misunderstood. I was saying that, if someone wants to write something that requires all of your proposed features (arbitrary spatiotemporal pixel access, etc.) he/she should write a C++ plugin for that, rather than resort to an unreadable RPN-based DSL.

[wonkey_monkey]

no, what you're trying to do is insane.

I've already done it; I'm just looking to improve it.

your proposed Expr filter is no longer an expression evaluating filter, but rather an exotic DSL.

It is what it is. Most of my bullet points are just extensions to what Expr already provides (or Expr provides a subset of my filter's features). Isn't Expr already slightly more than just an expression evaluating filter?

and that's very harmful

What's harmful about it?

you best bet is to directly extend the avisynth scripting language

Doesn't that also apply to Expr itself?

My best bet is definitely this. I've got use cases for it beyond Avisynth, for a start.

(arbitrary spatiotemporal pixel access, etc.) he/she should write a C++ plugin for that, rather than resort to an unreadable RPN-based DSL.

I really don't understand what you're trying to say. I'm writing (written, really, it's just a bit rough around the edges) a C++ plugin that compiles user-provided RPN strings, one feature of which is spatiotemporal pixel access (only arbitrary spatially, not temporally). This is just an extension to Expr's spatial pixel access (an idea which I think may have originated with me in the first place). I'm really not sure what's so offensive about that.

[StainlessS]

I'm really not sure what's so offensive about that.
Aint nuttin' offensive bout it, you know what Feisty's like, just ignore him :)
I'm eager to see what weird stuff you come up with [even if it is a bit niche, and even if I dont understand it].

I'm guessin' that nobody is gonna be forced to use it.

[feisty2]

You change the nature of something when you add enough new things to it. Expr was originally an expression evaluating filter and by its nature, not capable of doing many complex things, that’s why it suited just fine with a somewhat exotic language (RPN).

Now that you’ve changed the nature of Expr, you want DSL capabilities, then you should go for an actual DSL instead of having DSL stuff layered on top of something that no longer serves the actual purpose

[wonkey_monkey]

I'm guessin' that nobody is gonna be forced to use it.

Shhh, you're giving away my grand plan!

[gispos]

Could you please send me the exact files mentioned here?
The file you are opening with avspmod.
The saved file you are opening with AviSource.
(But no need for the Это тестовый Video.mkv, I can create the file with such a name for myself), thanks
You don't need a video from me for this.
1.) Name a video Japanese
2.) Open it with AvsPmod (error)
3.) Save this script
4.) Open the saved script in AvsPmod with AviSource

https://i.postimg.cc/jDr4wSZr/Error.jpg (https://postimg.cc/jDr4wSZr)
https://i.postimg.cc/ZCNL5dPc/Avi-Source.jpg (https://postimg.cc/ZCNL5dPc)

[wonkey_monkey]

This expr behaves unexpectedly (at least to me, I might be missing something):

expr("sxr 0 < 128 192 M^ 255 ?")

"192 M^" should amount to a no-op as far as the stack is concerned, but instead of returning a white clip (which is what happens if you take out "192 M^"), a black clip is returned (it's returning the value of "sxr 0 <", as if the ternary operator and subsequent stack elements have been completely removed).

I think the problem might lie with constant folding and findBranches.

[Dogway]

pinterf, out of curiosity, how is the lookup table filling? If I have 256 values in 8-bit the expression evaluates 256 times for Y plane, so 256 possible values. How is that taking so much RAM space?
Same for 16-bit, for single LUT, it's only 65536 values, I have seen 1D LUTs of length 130000 taking less than 4Mb. Besides you might probably get away with half length and use some kind of interpolation (ie. 3D LUTs can use tetrahedral).
YUV planes are decorrelated so you can use 3x1D LUTs instead of 3D LUTs which are heavier.
Sorry if this is dumb as I have no programming background.

[pinterf]

pinterf, out of curiosity, how is the lookup table filling? If I have 256 values in 8-bit the expression evaluates 256 times for Y plane, so 256 possible values. How is that taking so much RAM space?
Same for 16-bit, for single LUT, it's only 65536 values, I have seen 1D LUTs of length 130000 taking less than 4Mb. Besides you might probably get away with half length and use some kind of interpolation (ie. 3D LUTs can use tetrahedral).
YUV planes are decorrelated so you can use 3x1D LUTs instead of 3D LUTs which are heavier.
Sorry if this is dumb as I have no programming background.
Masktools LUT calculation is allocating 256 (8 bit) or 65536*2 (10-16 bits) bytes for 1D Luts.
This is per instance and per plane.

Latest public version of masktools is able to reuse the similar expression lookups for lut_xyz in order to spare with memory and calculation time.

My workbench version extends this behaviour to mt_lutxy and mt_lut as well. As a side effect, it will no longer allocate a larger buffer size than needed (e.g. 65536 elements for a 10 bit video which would require only 1024 elements).
Maybe I'm gonna build a test version from the actual code.

[pinterf]

Avisynth+ 3.7.1 test build 8 (20210705) (https://drive.google.com/uc?export=download&id=1NK_XRbelWytZYOl7KLDb4JgLGvaYEFjV)
20210705 WIP
------------
- Expr: add 'round', 'floor', 'ceil', 'trunc' operators (nearest integer, round down, round up, round to zero)
Acceleration requires at least SSE4.1 capable processor or else the whole expression is running in C mode.
- Fix: Expr: wrong constant folding optimization when ternary operator and Store-Only (like M^) operator is used together.
For the expression "sxr 0.5 < 128 192 M^ 255 ?" the expected result must be the same as for "sxr 0.5 < 128 255 ?"
Since 192 M^ is no-op regarding the expression parts.
- Add rest of Clip-type frame property setter methods: propSetArray

[pinterf]

This expr behaves unexpectedly (at least to me, I might be missing something):

expr("sxr 0 < 128 192 M^ 255 ?")

"192 M^" should amount to a no-op as far as the stack is concerned, but instead of returning a white clip (which is what happens if you take out "192 M^"), a black clip is returned (it's returning the value of "sxr 0 <", as if the ternary operator and subsequent stack elements have been completely removed).

I think the problem might lie with constant folding and findBranches.
Thanks, good catch, fixed, see test8

[kedautinh12]

Thanks

[wonkey_monkey]

Excellent, thanks pinterf!

[FranceBB]

Well done, thanks! :)

[Dogway]

Thanks for the update!

Is a sort operator possible? I found myself the last days fighting with implementing some median algos (removegrain modes 2-4). I know I can't compete in speed but for completion in ExTools this is probably the last function I'm adding.

Tired to do permutations to no avail (https://pastebin.com/2U9XYJZz) I searched some maths and found Sorting Networks, this is also proving hard to achieve with current Expr in-string tools.

Pseudo-code for 8 inputs (swap here means variable swap, 0 as nop sub)
A C < A C swap 0 ?
B D < B D swap 0 ?
E G < E G swap 0 ?
F H < F H swap 0 ?
A E < A E swap 0 ?
B F < B F swap 0 ?
C G < C G swap 0 ?
D H < D H swap 0 ?
A B < A B swap 0 ?
C D < C D swap 0 ?
E F < E F swap 0 ?
G H < G H swap 0 ?
C E < C E swap 0 ?
D F < D F swap 0 ?
B E < B E swap 0 ?
D G < D G swap 0 ?
B C < B C swap 0 ?
D E < D E swap 0 ?
F G < F G swap 0 ?

I thought on assigning new variables but Expr falls short with only A to Z. Maybe possible add AA, AB...?

I also found myself occasionally wanting to do the next:
"x y * sqrt A^ A[-1,0] B^ A[0,0] C^ ...."

Not asking, just thinking out loud, feel free to implement what's easy/useful.

[pinterf]

Thanks for the update!

Is a sort operator possible? I found myself the last days fighting with implementing some median algos (removegrain modes 2-4). I know I can't compete in speed but for completion in ExTools this is probably the last function I'm adding.

Tired to do permutations to no avail (https://pastebin.com/2U9XYJZz) I searched some maths and found Sorting Networks, this is also proving hard to achieve with current Expr in-string tools.

Pseudo-code for 8 inputs (swap here means variable swap, 0 as nop sub)
A C < A C swap 0 ?
B D < B D swap 0 ?
E G < E G swap 0 ?
F H < F H swap 0 ?
A E < A E swap 0 ?
B F < B F swap 0 ?
C G < C G swap 0 ?
D H < D H swap 0 ?
A B < A B swap 0 ?
C D < C D swap 0 ?
E F < E F swap 0 ?
G H < G H swap 0 ?
C E < C E swap 0 ?
D F < D F swap 0 ?
B E < B E swap 0 ?
D G < D G swap 0 ?
B C < B C swap 0 ?
D E < D E swap 0 ?
F G < F G swap 0 ?

I thought on assigning new variables but Expr falls short with only A to Z. Maybe possible add AA, AB...?

I also found myself occasionally wanting to do the next:
"x y * sqrt A^ A[-1,0] B^ A[0,0] C^ ...."

Not asking, just thinking out loud, feel free to implement what's easy/useful.

Getting median is a bit easier for limited number of entries e.g. 3 or 5, and needs no explicite sorting, like is done here.
For example here:
Median of 3
https://github.com/pinterf/MedianBlur2/blob/master/MedianBlur2/medianblur_sse2.cpp#L81
Median of 5
https://github.com/pinterf/MedianBlur2/blob/master/MedianBlur2/medianblur_sse2.cpp#L155

For these magnitudes (3, 5) they are not even complex ones and can be written specifically.

For larger radius we probably need a generic implementation with sorting but then Expr would starting to become an enormously complex filter (embedded MedianBlur :) ) rather than an expression evaluator.

Generic sorting algorithms are available for avx2 and avx512, (search for "avx2 avx512 simd sorting"); I can say that there are really nice solutions, but they are beyond my mental limits :).

[pinterf]

I thought on assigning new variables but Expr falls short with only A to Z. Maybe possible add AA, AB...?

At the moment I have exactly 26 memory slots for them (A..Z), assigning letters to slots means one-to-one relation at the moment.

[Myrsloik]

Thanks for the update!

Is a sort operator possible? ...

Why are you not using real compilers at this point? WHY?

I mean even modern javascript engines will probably run better with that number of variables and have cleaner syntax.

Obviously the solution is to implement a secondary stack where you can push and pop overflow values. As a solution to the number of addressable input clips use stackvertical to get around it.

WE DON'T NEED NO STINKING COMPILERS!

[wonkey_monkey]

Thanks for the update!

Is a sort operator possible?

Pseudo-code for 8 inputs (swap here means variable swap, 0 as nop sub)
A C < A C swap 0 ?


The thing to remember about the ternary operator ("?") is that everything prior to that is already evaluated, so you've already swapped the variables and put 0 on the stack. In fact, what you're evaluating there is the value of C, because the stack currently contains:

(A C <) - 0 or 1
C
A
0


You could do the following:

A C < A C ? A C < C A ?

This will put either A C or C A on the stack depending on the comparison.

This isn't great because of the two comparisons. You could always store the first result into a variable:

A C < R@ A C ? R C A ?

Or you could be a bit cleverer and do:

A C < A C ? dup A C + -

Then again it's probably most efficient just to do

A C min A C max

Or you might be able to do some swaps and dups to avoid loading the variables twice (I'm not sure how efficiently Expr recalls them).

Thank you for listening to my TED Talk.

[Dogway]

wonkey_monkey: nice TED hehe. I was going with the new var assignment starting from Z, and then thought what a sudoku I would end playing so asked for AA kind of vars. I will go with the most efficient method, ternaries are costly so min max probably. I'm not sure it might work but will test. Hopefully I can build a nice median and repair library as I did with ex_edge().

Myrsloik: It's just too much for me... and my 2013 SSD. I'm art oriented so my knowledge and disk space goes to DCC (3D, compo, video) and audio software mostly (no games). I fear that getting into nitty gritty programming can derail me too much, or simply make either my brain or SSD explode. I only do basic scripting in AVS, AHK and Python, some regex in there too and very basic GLSL. Will look forward how Expr evolves.

[pinterf]

New version
Avisynth+ 3.7.1 test build 9 (20210706) (https://drive.google.com/uc?export=download&id=1lbiMMPsSFTKoKhUKdNl0NdcKiv0bgLKY)
- Expr: allow arbitrary variable names (instead of A..Z), up to 256 variables can be used. Do not use existing keywords.
Variable names must start with '_' or alpha, continued with '_' or alphanumeric characters.

[kedautinh12]

Thanks

[FranceBB]

I just replaced the files on my installation and there's already a new version of AVS+.
You never take a break, Ferenc, thanks!! :D

[feisty2]

Myrsloik: It's just too much for me... and my 2013 SSD. I'm art oriented so my knowledge and disk space goes to DCC (3D, compo, video) and audio software mostly (no games). I fear that getting into nitty gritty programming can derail me too much, or simply make either my brain or SSD explode. I only do basic scripting in AVS, AHK and Python, some regex in there too and very basic GLSL. Will look forward how Expr evolves.

it is a common misconception to assume that writing a proper C++ plugin requires more work than dirty hacks like feature creep Expr.

to describe a 3x3 gauss blur filter, you have in Expr: "x[-1, -1] x[-1, 0] 2 * + x[-1, 1] + x[0, -1] 2 * + x 4 * + x[0, 1] 2 * + x[1, -1] + x[1, 0] 2 * + x[1, 1] + 16 /"

meanwhile to describe the same thing in C++: https://github.com/IFeelBloated/vapoursynth-plusplus/blob/master/Examples/GaussBlur.hxx#L22

you tell me which one is easier to write and to understand.

a temporal median filter (https://github.com/IFeelBloated/vapoursynth-plusplus/blob/master/Examples/TemporalMedian.hxx#L25) that works for any temporal radius is also no more than a few lines in C++. but in this case, you can't even describe it (by ab)using Expr.

it's year 2021 and things have changed, drastically. writing a C++ plugin is scripting (given that it's a simple filter like 3x3 conv or temporal median), and C++ is a scripting language, if you will it to.

[StainlessS]

As Feisty says, some things are actually easier in C/CPP, and Doggy would make for a good coder. [So would Real.Finder].
The only real difficult bit about learning C is figuring out where to put all them damn semi-colons, after that its no more
difficult than AVS script.

[feisty2]

davidh*****, I wasn't replying to you. I was replying to Dogway continuing on Myrsloik's point. I suggest that you do not cut in on a conversion not centered around you, and stop acting like you own the place or something. It's a public forum, you do not have the right to stop me from talking just because you don't agree with my opinions.

[feisty2]

now back to your points

Oh, feisty, please give it a rest with the moaning. No-one's ever going to force you to use Expr. Clearly some of us find it very useful, even those like me who can write fully-fledged C++ plugins when we need to. I just don't understand why you're so offended that we have a different tool to use when the occasion rises.

I am a frequent user of Expr and it would be ridiculous for you to think that I am "offended" by Expr. However, I use Expr for the right purpose, namely expression evaluation.
what I am concerned about is that, if this feature creep Expr gets widely adopted, people would start abusing Expr for things that should be written in a proper programming language (Dogway is already asking for the sort functionality). the consequence of this is that it creates a shit mountain of unreadable and unmaintainable code, because: a) RPN itself is a rather "exotic" language, hard to decipher by human. b) Expr, even with all that feature creep, is still nowhere as expressive as a real programming language, so it is very likely that sometimes there's no direct way to describe what you want using Expr. and people would then "invent" weird tricks to do what they want indirectly, and those "tricks" can be extremely confusing to the reader of the code.

[feisty2]

Expr makes customised filtering incredibly quick and easy. I have scripts where I have to do dozens of Expr-friendly operations. It's ridiculous to write dozens of C++ filters, or even one multi-choice filter, when a single line of Expr per use will do.
...
And then when you're testing, you need to close VirtualDub or AVSpmod every time you recompile.

a C++ filter takes arguments, you simply make the WIP part, something like a not yet determined convolution kernel, a parameter of the filter and you're free to fill in all kinds of different arguments in your script.


And it's disingenous to imply that it universally doesn't require more work to write a C++ plugin (and Expr is not a dirty hack).

At the very best, with a .cpp template already saved somewhere, you've still got to create your project, add your source code, link avsynth.lib, make sure avisynth.h is included, fiddle about with optimisation switches, make sure you've got the right architecture targetted, that you're not accidentally doing a Debug build, add a build event to copy the DLL to your plugins folder... the list goes on.

that's your overcomplicated workflow. for me it's just one command line: g++ -shared -std=c++2b -lstdc++ -static -O3 -flto -march=native -finline-limit=1000000000000000000000000000 -funroll-all-loops -funsafe-loop-optimizations -o Filter.dll EntryPoint.cxx vapoursynth.lib

[feisty2]

Since you ask, the Expr one, definitely on the first count - it's far less code and far easier to debug

is this supposed to be a joke? RPN is easy to debug? RPN expressions are incredibly "nonlocal", in ways that there's no parentheses to identify the local sections of a complex formula. you'd have to imagine a stack in your head, and emulate the evaluation process with your imaginary stack to determine the operands of an operator. good luck with modifying a local term of a multi-line long expression.


x[-1, -1] x[-1, 0] 2 * + x[-1, 1] + x[0, -1] 2 * + x 4 * + x[0, 1] 2 * + x[1, -1] + x[1, 0] 2 * + x[1, 1] + 16 /

is hardly "far less code" than

auto GaussKernel = [](auto Center) {
auto WeightedSum = Center[-1][-1] + Center[-1][0] * 2 + Center[-1][1] +
Center[0][-1] * 2 + Center[0][0] * 4 + Center[0][1] * 2 +
Center[1][-1] + Center[1][0] * 2 + Center[1][1];
return WeightedSum / 16;
};

[feisty2]

First you complain about feature creep, now you're complaining that it doesn't have enough features. Make your mind up! Expr is never going to do everything and that's not why it exists.


it is clear that you want Expr to do everything, don't lie.
pinterf,

I've been thinking long and hard about Expr for the last few days, because it's so much quicker than the RPN compiler I've been working on for a while. It does lack a number of features that I have found very useful, though - not that I'm asking you to do anything, Expr is excellent enough as it is and you've already been kind enough to implement a few things at my suggestion, but I just thought it might be worth going over my ideas in public in case of it's any interest to anyone.

As I use RPN for a lot of things, a standalone RPN compiler is something of a necessity for me. But before I go off in my own direction - taking plenty of inspiration from Expr, if not actual code; anything I eventually release will be sure to be open source though - I just wondered about your potential interest in such a project, as in whether it would be something that could be usefully developed in parallel with future iterations of Expr (and/or used to create new programmable features of Avisynth).

On the one hand it seems silly to duplicate so much effort, when Expr already shares so much in common with my aims. But on the other hand, what I'm envisioning would also be quite different:


A standalone RPN compiler, which has a concept of input and output "planes" but only as generic data sources, with width, height, (and possibly additional dimensions), type (byte, short, int, float, double) and pitch. Bit depth conversions and colour channel bias would be left for the calling function to inline as boilerplate RPN ops (more or less what Expr already does, just abstracted one more level)
The ability to do per-clip, per-frame, per-line, and per-pixel calculations (storing variables). Per-clip would probably just be constant folding since there'd be no changeable input. Per-line might be easy to implement directly in Expr - it's just a matter of moving the start of the loop and making sure the stack is empty at the start of the loop. Per-frame is trickier. Per-pixel only applies if the output channels are calculated in an interleaved way, which is natural for packed RGB(A) but not so natural for YUV(A) (you have to jump around between the output planes)
Internal multithreading
Choice of float or double precision (my current compiler uses the x87 stack which offers 80-bit precision - not available with SIMD unfortunately)
Trig functions implemented with SIMD (based on ssemath or some other library), floor, ceil, bitwise operators and so on
Automatic type switching between int and float
Option to return black out-of-bounds pixels instead of repeating edges
Something more akin to true conditional execution - difficult with SIMD but not impossible to emulate - rather than just the basic a/b swap ternary operator
Internal multi-dimensional arrays - build your own LUT with per-clip/per-frame calculations!
Run-time calculable pixel offsets
Support for other planes (Y in U, G in Y, etc) to be referenced (if they're the same size). Allow non-matching input clips as long as their non-matching planes aren't referenced
Temporal offsets to access past/future frames
Inlined Avisynth variables
Support for comments (a simple regex can remove several styles)


The last four I plan to implement first as an Expr wrapper plugin, as a separate/proof-of-concept project.

Hmm... you know, now that I write this out it does seem like a bit of a monster. Still, it would be just as helpful to hear if it's of no real interest to you - no offence will be taken! I'd also welcome your thoughts on the subject, or if anyone else has any feature requests or hints.

PS Thanks for everything you do!

I don't see anything wrong with my argument which suggests that you attempt to make Expr capable of everything (thus the feature creep) and your attempt failed.

[feisty2]

you go ahead delete your posts if you want, I won't, I am not a big fan of self-censorship.

edit: for davidh, the definition of self-censorship (https://www.merriam-webster.com/dictionary/self-censorship): the act or action of refraining from expressing something (such as a thought, point of view, or belief) that others could deem objectionable. the perfect word for what you just did.

[wonkey_monkey]

Not what the word means but okay...

Self-censoring is quite distinct from censorship. That's why it has "self-" in front of it.

I deleted my posts not for their objectionality but because they were off-topic.

[pinterf]

O.K., by popular demand I consider integrating brainfuck language (https://en.wikipedia.org/wiki/Brainfuck) elements into Expr :)

[ChaosKing]

Finally something I can understand easily!

[Myrsloik]

O.K., by popular demand I consider integrating brainfuck language (https://en.wikipedia.org/wiki/Brainfuck) elements into Expr :)

We'll soon announce a commercial implementation of Expr that uses whitespace as the programming language. It's the only way to ensure nobody can read secret corporate scripts and is one of the most requested features in VapourSynth. The backend will be obviously be based on freepascal.

[wonkey_monkey]

pinterf, do you know if there's any documentation for jitasm? I see references to a "GettingStarted wiki" but I don't know where to find that or if it still exists. I'm slowly grasping it through reading exprfilter.cpp but I could use the context of an actual guide.

[GMJCZP]

pinterf, do you know if there's any documentation for jitasm? I see references to a "GettingStarted wiki" but I don't know where to find that or if it still exists. I'm slowly grasping it through reading exprfilter.cpp but I could use the context of an actual guide.

Will this do?
Here (https://asmjit.com/doc/index.html)

[Dogway]

Thanks for the update pinterf. I got the Sorting Network to work, it is not THAT slow though. Given that the top speed one can get with a median in Expr is "undot" (min and max clamp), sorting is only 18% below in performance. It's fine for implementing exotic kernels and use in production. The only problem I see is that unlike ex_boxblur() median Expr can't match removegrain speed, and the only difference is the operator used (min and max), since pixel fetching is shown to give the same performance. I do think there might be room for improvement but can't actually tell as I don't understand CPP.

# 100% removegrain(1,-1)
# 83% ex_median(mode="undot",UV=1)
# 65% ex_median(mode="undot2",UV=1) # 301fps
# 1.2% MedianBlur(1,0,0)

I wanted to implement your median of 3 and 5 kernels, but didn't understand the code. What is the window size, 3x3? So I actually tested it and it matched removegrain(4,-1) (for radius 1). I think there's value for the MedianBlur plugin because it can use bigger kernels and also temporal, but the performance of 6fps struck me a bit.

# mode name prone to change (suggestions welcome :P )
mode == "undot2" ? "x[-1,1] A^ x[0,1] B^ x[1,1] C^ x[-1,0] D^ x[1,0] E^ x[-1,-1] F^ x[0,-1] G^ x[1,-1] H^ " \
+"A C min AA^ A C max CC^ " \
+"B D min BB^ B D max DD^ " \
+"E G min EE^ E G max GG^ " \
+"F H min FF^ F H max HH^ " \
+"AA EE min A^ AA EE max E^ " \
+"BB FF min B^ BB FF max F^ " \
+"CC GG min C^ CC GG max G^ " \
+"DD HH min D^ DD HH max H^ " \
+"A B min AA^ A B max BB^ " \
+"C D min CC^ C D max DD^ " \
+"E F min EE^ E F max FF^ " \
+"G H min GG^ G H max HH^ " \
+"CC EE min C^ CC EE max E^ " \
+"DD FF min D^ DD FF max F^ " \
+"BB E min B^ BB E max EE^ " \
+"D GG min DD^ D GG max G^ " \
+"B C min BB^ B C max CC^ " \
+"DD EE min D^ DD EE max E^ " \
+"F G min FF^ F G max GG^ " \
+"x[0,0] BB GG clip" : \



@wonkey_monkey: I tested with the ternary option as I thought one ternary might be faster than min+max but I don't think your code example works.
"A C < Q@ A AA^ C CC^ ? Z^ Q C CC^ A AA^ ? Z^ "
I think Expr ternaries don't look for a true or false (in case Q@ catches that at all) but expect a comparison operator.

PD: I had a look at the MSVC compiler size, 40Gb, ouch!

[wonkey_monkey]

Will this do?
Here (https://asmjit.com/doc/index.html)

That's asmjit rather than jitasm, but actually yes, it will make for informative reading, so thanks!

[wonkey_monkey]

@wonkey_monkey: I tested with the ternary option as I thought one ternary might be faster than min+max but I don't think your code example works.
"A C < Q@ A AA^ C CC^ ? Z^ Q C CC^ A AA^ ? Z^ "
I think Expr ternaries don't look for a true or false (in case Q@ catches that at all) but expect a comparison operator.


A quick "fingers" test - lift a finger for each variable that gets added to the stack, lower a finger for each popstore or regular operator, lower two fingers for a "?" - suggests you've only got one item on the stack when you reach the "?"

Remember that everything before the "?" gets executed - all "?" does is select which of the two topmost stack items to remove. Maybe you meant AA@ and CC@, so as not to remove A and C from stack, but in any case both will then be overwritten by the next section anyway, as will the result you would have stored in Z.

So your code says:

Compare A and C (stack contains result)
Store result in Q (stack contains result)
Load A, but immediately store and pop into AA (stack contains result)
Load C, but immediate store and pop into CC (stack contains result)
? - needs three items on the stack so it fails


Also shouldn't the last line be:


x[0,0] FF GG clip"

?

[feisty2]

PD: I had a look at the MSVC compiler size, 40Gb, ouch!


GCC 11 (mingw-w64 target) is less than 100MB, and msvc is far less than 40GB if you only need C++ components.

[Dogway]

@wonkey_monkey: true, true, typo (^ for @). I was also overwriting the second ternary.

This thing is tricky, this should do it but it doesn't, so I will leave this here and think in the evening:

+"A C < Q@ A AA@ C CC@ ? Z^ Q C CC@ A AA@ ? Z^ "
+"B D < Q@ B BB@ D DD@ ? Z^ Q D DD@ B BB@ ? Z^ "
+"E G < Q@ E EE@ G GG@ ? Z^ Q G GG@ E EE@ ? Z^ "
+"F H < Q@ F FF@ H HH@ ? Z^ Q H HH@ F FF@ ? Z^ "
+"AA EE < Q@ AA AAA@ EE EEE@ ? Z^ Q EE EEE@ AA AAA@ ? Z^ "
+"BB FF < Q@ BB BBB@ FF FFF@ ? Z^ Q FF FFF@ BB BBB@ ? Z^ "
+"CC GG < Q@ CC CCC@ GG GGG@ ? Z^ Q GG GGG@ CC CCC@ ? Z^ "
+"DD HH < Q@ DD DDD@ HH HHH@ ? Z^ Q HH HHH@ DD DDD@ ? Z^ "
+"AAA BBB < Q@ AAA A@ BBB B@ ? Z^ Q BBB B@ AAA A@ ? Z^ "
+"CCC DDD < Q@ CCC C@ DDD D@ ? Z^ Q DDD D@ CCC C@ ? Z^ "
+"EEE FFF < Q@ EEE E@ FFF F@ ? Z^ Q FFF F@ EEE E@ ? Z^ "
+"GGG HHH < Q@ GGG G@ HHH H@ ? Z^ Q HHH H@ GGG G@ ? Z^ "
+"C E < Q@ C CC@ E EE@ ? Z^ Q E EE@ C CC@ ? Z^ "
+"D F < Q@ D DD@ F FF@ ? Z^ Q F FF@ D DD@ ? Z^ "
+"B EE < Q@ B BB@ EE EEE@ ? Z^ Q EE EEE@ B BB@ ? Z^ "
+"DD G < Q@ DD DDD@ G GG@ ? Z^ Q G GG@ DD DDD@ ? Z^ "
+"BB CC < Q@ BB BBB@ CC CCC@ ? Z^ Q CC CCC@ BB BBB@ ? Z^ "
+"DDD EEE < Q@ DDD D@ EEE E@ ? Z^ Q EEE E@ DDD D@ ? Z^ "
+"FF GG < Q@ FF FFF@ GG GGG@ ? Z^ Q GG GGG@ FF FFF@ ? Z^ "


The last line should be the second from the minimum and second from the maximum, since order is A B C D E F G H, I chose B G. I compared it against removegrain(2) anyway.

[wonkey_monkey]

This thing is tricky, this should do it but it doesn't, so I will leave this here and think in the evening:

I'm afraid you're still misunderstanding how "?" works. Could you start a new thread so we can discuss it there, or I can PM you?

[Dogway]

I have used ternaries for years now, this is rather a stack issue, posted here (https://forum.doom9.org/showthread.php?p=1947059#post1947059) a simplified code.

[qyot27]

It took several months, but I finally got around to working on a macOS installer package. 3.7.0 is available on the normal Releases page, in two forms:
High Sierra and Mojave builds (10.13 & 10.14)
Catalina and higher builds (10.15+)

And also filesonly tarballs if you'd rather skip the installer (although they do have shell scripts to help users unfamiliar with the process).

[wonkey_monkey]

Is there any way to load the contents of a text file as a string using built-in Avisynth functions? I can't seem to find one but maybe there is a trick.

[StainlessS]

Is there any way to load the contents of a text file as a string
Not that I ever found. [even trying to abuse Import() failed for me].

Only thing I'm aware of,

RT_ReadTxtFromFile(String ,Int "Lines"=0,Int "Start"=0)
Non-clip function.
String Filename, Name of text file to load into a string.
Lines=0=unlimited. Set to number of leading lines in text file to load, eg 1 = load only the first line of text file.
The return string is n/l ie Chr(10) separated, and carriage returns are removed from the returned string.
If source file was missing newline on very last line, it will append a newline so that all lines are similarly formatted.
v1.03, Added Start arg default=0=very first line (relative 0). Would have been nice to have start and lines in reverse
order but implemented as above to not break scripts.
Throws an error if your requested Start is >= to the number of lines in the file, or zero len file.
To fetch the last line of a text file, use eg Start = RT_FileQueryLines(Filename) - 1 (Start is zero relative).
You could eg get the last line of a d2v file which might look like this:- "FINISHED 100.00% VIDEO"

[pinterf]

Is there any way to load the contents of a text file as a string using built-in Avisynth functions? I can't seem to find one but maybe there is a trick.
Abusing ConditionalReader?
http://avisynth.nl/index.php/ConditionalReader

[pinterf]

It took several months, but I finally got around to working on a macOS installer package. 3.7.0 is available on the normal Releases page, in two forms:
High Sierra and Mojave builds (10.13 & 10.14)
Catalina and higher builds (10.15+)

And also filesonly tarballs if you'd rather skip the installer (although they do have shell scripts to help users unfamiliar with the process).
Congratulations!

[LigH]

:eek: AviSynth+ on Mac? Selur might enjoy that for Hybrid...

[DTL]

As users tried to use output of ColorBars() for resamplers testing ( https://forum.doom9.org/showthread.php?p=1947958#post1947958 ) it is recommended to add 'mode' parameter to ColorBars() internal function with at least 3 values:
1. 'pixel-art' (as today ColorBar() ouput)
2. 'film-look' ( I think it is default for real prof color bar generator)
3. 'video-look'

Possible processing is suggested in https://forum.doom9.org/showthread.php?p=1947974#post1947974 .

And may be at least put a note to Avisynth wiki about non-compatibility of current ColorBar() output for testing of 'film/video' motion pictures data processing software for quality and results of levels transition processing.

Linked wiki .pdf https://www.arib.or.jp/english/html/overview/doc/6-STD-B28v1_0-E1.pdf also notes about correct levels transitions preparation:

Annex A5: A.5 Transient
Ringing may occur when the stripe level of this color bar is suddenly changed, then this may possibly
cause operational inconvenience. Therefore, it is necessary to carry out design while limiting
bandwidths for leading edge and falling edge.
The number of samples to be used for the transient shall be 6 to 9, in the case of 1920
horizontal samples, although it may depend upon the scale of hardware, process performance
and the so-called “make up”.

Also the older ITU recs describes some valid forms of levels transition in details:
https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.801-1-199510-W!!PDF-E.pdf
Rec. ITU-R BT.801-1 (for SDTV)
Test signals for digitally encoded colour television signals conforming
with Recommendations ITU-R BT.601
and ITU-R BT.656
These digital waveforms are made up of pulses in uniform ranges, ramps between two uniform
ranges, and transitions between two uniform ranges, shaped by a filter whose impulse response R(t)
is defined as a function of time t as follows:
– for –3T < t < 3T, R(t) = 0.42 + 0.50 cos(π t/ 3T) + 0.08 cos(2π t/ 3T)
– otherwise R(t) = 0
(R(t): Blackman window).
The value of T is 74 ns for digital waveforms A1, A2, A3 and A4

And Rec. ITU-R BT.1729 https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.1729-0-200504-I!!PDF-E.pdf (for HDTV)
Rise and fall times of bar transitions
The 10% to 90% rise time and 90% to 10% fall time of the colour bars should be identical to each
other and should be consistent with the frequency response specifications in Recommendations
ITU-R BT.601, ITU-R BT.1358, ITU-R BT.1847, ITU-R BT.1543 or ITU-R BT.709, as
appropriate to the format2. The shape of the leading and trailing edges should be similar to a raised
cosine function.

[Dogway]

Just wanted to report that for some time I have been having issues with variables that start with the same letter:

mode == "edgeW" ? " y[0,0] I@ y[0,1] B@ - abs I y[0,-1] G@ - abs max BB^ " \
+"I y[1,1] C@ - abs I y[-1,-1] F@ - abs max CC^ I y[-1,0] D@ - abs I y[1,0] E@ - abs max DD^ " \
+"I y[-1,1] A@ - abs I y[1,-1] H@ - abs max BB min CC min DD min " \
+"DI@ DD == x D E dup1 dup1 min I min swap2 max I max clip DI BB == x B G dup1 dup1 min I min swap2 max I max clip " \
+"DI CC == x C F dup1 dup1 min I min swap2 max I max clip x A H dup1 dup1 min I min swap2 max I max clip ? ? ?" : \

If I change the DI variable to S, everything is fine again.

[wonkey_monkey]

Another little note, not on Expr per se but on Format and String, which are useful in preparing Expr expressions. They only expand floats to six decimal digits after the point, which could cause some unnecessary inaccuracy. I think you need around 9 or 10 significant digits to guarantee hitting the true closest floating point number. Could consideration be given to a higher-precision flag or variations of those functions?

Or even allow %a format: https://stackoverflow.com/questions/4826842/the-format-specifier-a-for-printf-in-c/10128764

[cretindesalpes]

I’m just installing AVS+ for the first time and suddenly I feel like a total newbie… So I have a question I cannot find the answer on the official website nor on the wiki:

What are the host applications compatible with 64-bit AVS+ ? For:
– Editing and previewing
– Piping
– Encoding

Also, maybe the installer should be a bit more clear that you can also keep intact your existing 32-bit AviSynth 2.5 just by unchecking the 32-bit dll. I needed to keep my 32-bit AVS 2.6 because I found incompatibilities with my existing plug-in set and the 32-bit AVS+.

[Sharc]

Also, maybe the installer should be a bit more clear that you can also keep intact your existing 32-bit AviSynth 2.5 just by unchecking the 32-bit dll. I needed to keep my 32-bit AVS 2.6 because I found incompatibilities with my existing plug-in set and the 32-bit AVS+.
You may find Groucho's Universal Avisynth Installer useful for easy switching between versions:
https://forum.doom9.org/showthread.php?t=172124

[Boulder]

What are the host applications compatible with 64-bit AVS+ ? For:
– Editing and previewing
– Piping
– Encoding


Editing and previewing : VirtualDub2 (https://sourceforge.net/p/vdfiltermod/wiki/Home/)
Piping : Avs2yuv64 (https://github.com/MasterNobody/avs2yuv/releases)
Encoding : f.ex. x264, x265 64-bit builds

[real.finder]

Editing and previewing : VirtualDub2 (https://sourceforge.net/p/vdfiltermod/wiki/Home/)


and/or AvsPmod https://forum.doom9.org/showthread.php?t=175823

maybe the installer should be a bit more clear that you can also keep intact your existing 32-bit AviSynth 2.5 just by unchecking the 32-bit dll. I needed to keep my 32-bit AVS 2.6 because I found incompatibilities with my existing plug-in set and the 32-bit AVS+.

the installer can keep the old 32 avs if you uncheck the avs+ 32, I always did this, my 32 avs always avs 2.6 MT by SEt, but I often use 64 so it's last avs+

[kedautinh12]

Megui
https://sourceforge.net/projects/megui/

[LigH]

StaxRip (github (https://github.com/staxrip/staxrip) / VideoHelp (https://www.videohelp.com/software/StaxRip))

[Dogway]

Use avpsmod by gispos. Works fine with latest AVS+ 3.7.1.
For encoding I use avs2pipemod64 as always and x264_r2935_x64(Ligh).
To debug I find avsmeter and AVSInfoTool great tools too. I also had issues on the past and this tells you what is set on the register.

[cretindesalpes]

Thank you all very much for your replies!

[FranceBB]

Wow! @Cretindesalpes, you're back! :D
It's nice to see the creator of Dither Tools getting Avisynth+ for the very first time.
You won't ever feel the need to go back to the normal Avisynth, I can tell you that, you'll love AVS+ ;)

[kedautinh12]

Wow! @Cretindesalpes, you're back! :D
It's nice to see the creator of Dither Tools getting Avisynth+ for the very first time.
You won't ever feel the need to go back to the normal Avisynth, I can tell you that, you'll love AVS+ ;)

And he will port fmtconv to avs
https://forum.doom9.org/showthread.php?p=1948178#post1948178

[FranceBB]

And he will port fmtconv to avs
https://forum.doom9.org/showthread.php?p=1948178#post1948178

Nice! :D

[qyot27]

I’m just installing AVS+ for the first time and suddenly I feel like a total neebie… So I have a question I cannot find the answer on the official website nor on the wiki:

What are the host applications compatible with 64-bit AVS+ ? For:
– Editing and previewing
– Piping
– Encoding

Also, maybe the installer should be a bit more clear that you can also keep intact your existing 32-bit AviSynth 2.5 just by unchecking the 32-bit dll. I needed to keep my 32-bit AVS 2.6 because I found incompatibilities with my existing plug-in set and the 32-bit AVS+.
FFmpeg and mpv as well, so long as FFmpeg was configured with --enable-avisynth (which itself checks for the presence of AviSynth+'s headers in the system includes).

On all the other OSes* and CPU arches** AviSynth+ can now be used on natively, FFmpeg (and thus anything built against it, like mpv or VLC or kdenlive) and DJATOM's fork of avs2yuv are the direct points of contact.

*Linux, macOS, and BSD (since 3.5.0), Haiku (3.7.0)
**ARM (3.6.0), PowerPC (3.7.0), RISC-V and SPARC (git)

[StainlessS]

– Editing and previewing
Gispos did lots of work on AvsPMod:- https://forum.doom9.org/showthread.php?t=175823
And the old dead VirtualDub is reborn, VirtualDub2:- https://forum.doom9.org/showthread.php?t=172021
[VD2 not seem an update since about beginnning of pandemic, but its great and has AVS script editor, similar to old VDubMod].

[GAP]

Can you have AVISynth and AVISynth + on the same computer? It seems like you can only have one or other but some programs such as AVSMod can only use AVISynth vanilla.

[real.finder]

Can you have AVISynth and AVISynth + on the same computer? It seems like you can only have one or other but some programs such as AVSMod can only use AVISynth vanilla.

you mean avspmod? it work with avs+, seems you use outdated one

anyway, you can have AVISynth and AVISynth+ on the same computer by temporarily switch the dll of AVISynth or AVISynth+ with MPP, avspmod can do it too, and some programs/tools can use the AVISynth.dll that in same folder with it

[GAP]

So when I install AVISynth + and it asks me to override AVISynth files, I just select that option?

[LigH]

Yes, the core DLL(s) need to be in the system folder to be registered systemwide, and additional files get in their carefully planned locations. Switching is easy, using the Universal Avisynth Installer (https://forum.doom9.org/showthread.php?t=172124) by Groucho2004. But when you intend to just be and stay up-to-date, just run the official installer once and agree.

[cretindesalpes]

I have another question about the plane (or channel) identification, from a user point of view. In RGB, the channels are numbered in the B-G-R-(A) order. If I want to let a user specify a plane index (as a number) in a filter parameter, should I follow the internal Avisynth convention? Or the more common R-G-B-(A) order? What other plug-ins are doing regarding this topic?

I see that some old Avisynth plug-ins have been ported to support planar RGB in addition to YUV. These plug-ins often have “y”, “u” and “v” parameters. Are these parameters implicitly used for RGB planes too? If yes, what is the mapping convention?

[real.finder]

I have another question about the plane (or channel) identification, from a user point of view. In RGB, the channels are numbered in the B-G-R-(A) order. If I want to let a user specify a plane index (as a number) in a filter parameter, should I follow the internal Avisynth convention? Or the more common R-G-B-(A) order? What other plug-ins are doing regarding this topic?

I see that some old Avisynth plug-ins have been ported to support planar RGB in addition to YUV. These plug-ins often have “y”, “u” and “v” parameters. Are these parameters implicitly used for RGB planes too? If yes, what is the mapping convention?

in masktools
https://i.postimg.cc/j251vC8k/Untitled.png (https://postimages.org/)

others just ignore “y”, “u” and “v” parameters for RGB, but in any case nothing can stop you from use arrays for planes just like vs plugins does but this mean it will not work in old avs+ and avs 2.6 just like vsLGhost (vsLGhost still use “y”, “u” and “v” parameters though but use arrays for mode, shift, and intensity)

[Reel.Deel]

I have another question about the plane (or channel) identification, from a user point of view. In RGB, the channels are numbered in the B-G-R-(A) order. If I want to let a user specify a plane index (as a number) in a filter parameter, should I follow the internal Avisynth convention? Or the more common R-G-B-(A) order? What other plug-ins are doing regarding this topic?

I see that some old Avisynth plug-ins have been ported to support planar RGB in addition to YUV. These plug-ins often have “y”, “u” and “v” parameters. Are these parameters implicitly used for RGB planes too? If yes, what is the mapping convention?

From a users point of view I think it makes more sense using RGB(A).

MaskTool2 follows the YUVA-->RGBA mapping, also Expr maps the expressions to YUVA/RGBA. As real.finder said, most of the plugins that have been ported from VS always process RGB with no way to override it. Don't know if this would work but it would be nice if said plugins had a "planes" parameter at the very end that accepts an array just like VS, and when defined overrides the Y-U-V parameters for any colorspace.

[cretindesalpes]

Thanks for your answers. I’m kinda reluctant to use the array solution for compatibility reasons, so I think I’ll use a string instead. "all" processes all planes, "" nothing, and any combination containing '0', '1', '2', '3', 'y', 'u', 'v', 'r', 'g', 'b', 'a' selects the right planes, with the 'r', 'g', 'b' values being equivalent to 'y', 'u', 'v'.

[cretindesalpes]

I couldn’t find any clear specification about the floating point ranges in Avs+. It would be great if the Avisynth+ color format page (http://avisynth.nl/index.php/Avisynthplus_color_formats) could indicate this information. In a few locations it is implied that the f. p. range is “full-scale”, so I understand that black is 0.f and 100 % white is 1.f. Also, I’m not sure if the chroma center is 0.f or 0.5f (found some tests for a FLOAT_CHROMA_IS_HALF_CENTERED macro in the source code).

However:

– I’m not sure it is a documentation issue or a bug, but it seems that ColorBarsHD generates levels that are not “full-range” in floating point (pixel_type="YUV444PS"): pure black is 16/255 (not 0) and 100 % white is 235/255 (not 1). Is this the expected behaviour? The scale factor is defined here (https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/filters/source.cpp#L851) in the source code.

– In the ConvertBits (http://avisynth.nl/index.php/ConvertBits) documentation, it is specified that “Conversion from and to float is always full-scale”. However the floating point scale seems to depends on the fulls argument, so the aforementioned ColorBarsHD is correctly converted to PC.709 (full scale) by setting fulls=false, fulld=true. Again, is this a documentation imprecision or a bug?

– In the Levels (http://avisynth.nl/index.php/Levels) documentation, there is a table giving the equivalence between values for different bitdepth, and the f. p. values have a denominator of 256. This is a slightly different convention as the 255 used in ColorBarsHD.

So, is Avs+ floating point data subject to the TV/PC range madness too? And if yes, what are the exact conversion conventions?

[DTL]

"ColorBarsHD generates levels that are not “full-range” in floating point (pixel_type="YUV444PS"): pure black is 16/255 (not 0) and 100 % white is 235/255 (not 1). Is this the expected behaviour?"

It may be a general idea for better-precision coding of moving pictures digital levels:
The base range of nominal black and white is 16 and 235. And any additional bits only increases precision without moving the range. Like 10bit only allow to encode 16.25, and float 32bit allow to encode 16.12345 (about 7 decimal digits of precision).
With integer formats it was useful for converting from >8 bits to 8 bits with just skipping lower bits.
So may be with addition of single-pres float we just have only better precision of coding natural 16..235 range. Also it makes easy the conversion between different precisions with just convert float to int and back without additional scaling and shifting and it increases processing speed.

[jpsdr]

I've never seen in avisynth filters whith lookhead (that doesn't mean they don't exist). I know in VDub when you are processing frame n, how to access previous and next frames (n-1,n-2,...,n+1,n+2,...), provided by the API (without having to do it yourself with your own internal buffers) (you must have a recent VDub version using the last API), but i don't know how to do this in avisynth, even if it's possible (i mean directly with API, not doing it yourself with your own buffer).

[cretindesalpes]

DTL:

I can’t see any clear advantage for the user in transferring the full/limited range standards to floating point data, instead of keeping a consistent 0 = nominal black, 1 = nominal white. However the center of the chroma signals (0 or 0.5) is a bit more debatable.

The efficiency argument is mostly insignificant: bitdepth conversions should be only a tiny part of the whole processing calculations, especially if you go the floating point way (complex calculations requiring great accuracy or extended ranges, use of hardware shaders, etc.) There is no “fast bit shift” for floating point conversions, you always have to multiply. Setting a fixed range may introduce an extra addition, which is very light compared to actual processing functions like non-linear operations, multi-dimension table look-up, etc. And given that some common operations expect the black to be 0, other processing devices may have to rescale data back and forth if in/out black is not 0.

The only advantage is for the designers and coders of bitdepth conversion programs. It simplifies their operations, they don’t have to think about it when converting between float and int.

We have both full and limited range in the integer world. Digital consumer medias and broadcasts have generally limited signals, whereas most video cameras output full-range signals. Limited range for bitdepths > 8 bits is specified for a long time by international standards, and they make sense. Anyway we have to live with both ranges, and this is a PITA for low-level video processing users as well as plug-in developers, because we have to care about it for most operations.

At least with floating point data we have the opportunity to use a unique range, because the f. p. format holds plenty of headroom making the “limited” range meaningless. This is better for both users and plug-in developers. Vapoursynth made this choice and this is much easier for everybody with no drawback on the performance side. It’s like in the audio world: in floating point, 0 dBFS is almost universally set to ±1.0 in a processing graph. Nobody would use another reference, even if some storage formats may use a different convention for historical reasons.

However, my point was not about changing the choice made by the Avisynth+ team regarding floating point ranges. I just pointed out some discrepancies and wanted to get a clear description about the chosen data format convention.

[DTL]

If you set nominal 'video' to 0.0 and 1.0 it mean you still need to process and keep now negative -0.xx undershoots and over 1.0 overshoots.

I think the most prof video was with having footroom and headroom 'limited' coding and the only new and possibly degrading thing is PQ-full HDR that may be not have footroom and headroom and it will increase processing distortions but giving a bit more bits for HDR levels encoding.

The only exact physically defined display level is 'black' and 'white' is completely picked by system designer as it want. And for processing we still need to have footroom below black and over white (with both max super-white legal level + processing overshoots over max system super-white) so we need as much as possible headroom over nominal system white level. So I think no any reason to put black and white to 'nice' numbers like 0 and 1.

Working with about -0.1..1.x range may cause more precision lost because of walking around zero (very small values of positive and negative floats): There is a warning about using close to denornals in float32: https://stackoverflow.com/questions/9044555/for-float-and-double-why-it-is-asymmetric-for-the-negative-and-positive-numbers
"The range on the bottom end is further extended by allowing a floating value to be denormal. The smallest possible non-zero float has a 1 in the least significant bit of the mantissa.
You never actually want to get close to denormals, they lose significant digits in a hurry. They really only help to avert division-by-zero problems, at a price."
Also about denornals around zero in floats: https://en.wikipedia.org/wiki/Denormal_number

"We have both full and limited range in the integer world."

There are 2 digital picture worlds - of static and moving pictures. For static pictures typically full range is used. For moving pictures - limited. Unfortunatelly most of Avisynth's design from old times do not cleary shows it and still have lots of defects for processing moving pictures data.

The word 'limited' is not very nice - it may be better for the great olds to use word 'nonresticted' range. It better displays the requirements for coding method to have at least some footroom and headroom for coded values above and below nominals.

"most video cameras output full-range signals. "

Normal video camera for moving pictures must feed RGB in limited range also. See for example HDTV https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.709-6-201506-I!!PDF-E.pdf - 4 Digital representation
4.1 Coded signal R, G, B or Y, CB, CR
4.6 Quantization levels - both RGB and Y black levels are of 16 for 8bit and 64 (16.00) for 10bit limited range.

[real.finder]

I couldn’t find any clear specification about the floating point ranges in Avs+. It would be great if the Avisynth+ color format page (http://avisynth.nl/index.php/Avisynthplus_color_formats) could indicate this information. In a few locations it is implied that the f. p. range is “full-scale”, so I understand that black is 0.f and 100 % white is 1.f. Also, I’m not sure if the chroma center is 0.f or 0.5f (found some tests for a FLOAT_CHROMA_IS_HALF_CENTERED macro in the source code).


floating point ranges in Avs+ is same as vs now, 0..1.0 for luma and -0.5..0.5 (zero centered) for chroma (and of course both can have out of range Values)


However:

– I’m not sure it is a documentation issue or a bug, but it seems that ColorBarsHD generates levels that are not “full-range” in floating point (pixel_type="YUV444PS"): pure black is 16/255 (not 0) and 100 % white is 235/255 (not 1). Is this the expected behaviour? The scale factor is defined here (https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/filters/source.cpp#L851) in the source code.

– In the ConvertBits (http://avisynth.nl/index.php/ConvertBits) documentation, it is specified that “Conversion from and to float is always full-scale”. However the floating point scale seems to depends on the fulls argument, so the aforementioned ColorBarsHD is correctly converted to PC.709 (full scale) by setting fulls=false, fulld=true. Again, is this a documentation imprecision or a bug?

– In the Levels (http://avisynth.nl/index.php/Levels) documentation, there is a table giving the equivalence between values for different bitdepth, and the f. p. values have a denominator of 256. This is a slightly different convention as the 255 used in ColorBarsHD.

So, is Avs+ floating point data subject to the TV/PC range madness too? And if yes, what are the exact conversion conventions?

start read from https://forum.doom9.org/showthread.php?p=1926780#post1926780

[cretindesalpes]

real.finder:

Ah, great, thank you for the clarification. So if I understand correctly, 0–1 and -0.5–+0.5 is the target standard. Float was added earlier to the various Avs+ filters without a well-defined spec so their behaviours may vary and they would require a conforming update.

DTL:

Working with about -0.1..1.x range may cause more precision lost because of walking around zero (very small values of positive and negative floats): There is a warning about using close to denornals in float32
This is the other way round: with f.p., working close to 0 helps with the precision. The exponent part takes care of the range reduction. This is important when working with linear light, because of the (approximate) power curve of the eye sensitivity. So the numerical noise is related to the magnitude of the value. When representing a pixel value b + x (where b is a non-null constant assigned to nominal black), the precision is limited by the mantissa and very small values of x cannot be represented. It becomes more or less the equivalent of a 24-bit fixed-point coding. Anyway even in this case, the mantissa precision is more than enough for usual video processing.

Denomal numbers are not an issue here (funnily I wrote one of the articles referenced by the Wikipedia page you linked), the involved magnitudes are way too small for optical signals. Usually they caused problems because they were handled very slowly (esp. on Intel hw) by special micro-code in the FPU and could possibly trigger an exception. But the goal of the denormals was to increase the dynamic range of the f.p. numbers, so from a numeric point of view they are a good thing. Nowadays FPU is replaced by vector engines like SSE, AVX or NEON and denormals are ignored or muted by default with the appropriate flags (DAZ/FTZ).

Normal video camera for moving pictures must feed RGB in limited range also. See for example HDTV https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.709-6-201506-I!!PDF-E.pdf
Video camera (consumer grade, I don’t know for high-end professional cams) outputs are generally files containing AVC or HEVC streams in Y’Cb’Cr’ colorspace, so full range is perfectly legal.

[DTL]

Video camera (consumer grade, I don’t know for high-end professional cams) outputs are generally files containing AVC or HEVC streams in Y’Cb’Cr’ colorspace, so full range is perfectly legal.

Good consumer video camera also writes YUV 4:2:0 in 'limited' range. The only example of RGB-full may be some USB-connected web-cams with RGB24 'full' range output mode.

[LigH]

SONY camcorders may use "x.v.Colour", AVCHD with full range YUV.

[FranceBB]

real.finder:

Video camera (consumer grade, I don’t know for high-end professional cams)

SDI Stream in Limited TV Range with some possible overshoots in the luma if there are bright objects, like for instance if the cameraman is recording a person sitting on a chair and there's a bright lamp behind.
But yeah, it's always limited TV Range, unless you're recording in a logarithmic curve to go to HDR as in that case it's full range, however even if the flag in the file would be "full range", due to the nature of the log the waveform would appear as if it was "compressed" in the middle of a waveform monitor, so it shouldn't peak at the extreme boundaries of the full range values.

Anyway, we're getting out of track, you already have the answer with the values you're looking for, so good for it, mine was just to satisfy your curiosity. ;)

[kedautinh12]

SDI Stream in Limited TV Range with some possible overshoots in the luma if there are bright objects, like for instance if the cameraman is recording a person sitting on a chair and there's a bright lamp behind.
But yeah, it's always limited TV Range, unless you're recording in a logarithmic curve to go to HDR as in that case it's full range, however even if the flag in the file would be "full range", due to the nature of the log the waveform would appear as if it was "compressed" in the middle of a waveform monitor, so it shouldn't peak at the extreme boundaries of the full range values.

Anyway, we're getting out of track, you already have the answer with the values you're looking for, so good for it, mine was just to satisfy your curiosity. ;)

Sr for don't relate comment, are you test last build of L-SMASH Works
https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/commit/e5bb1ecb71f0edb5b7632b1013faaed77277e2b4#r54560114

[guest]

This may not be the right place to ask this....I have been searching for info on MDegrain scripts, that I know used to be available, now I can't find diddly squat !

Has SMDegrain (NotSMDegrain) completely replaced it, and wiped it of the face of the earth ??

I have some scripts that were kinda optimised for HD content, as I find just "straight" SMDegrain is VERY slow with 4K filtering / encoding.

Any info would be welcome.

[FranceBB]

Sr for don't relate comment, are you test last build of L-SMASH Works
https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/commit/e5bb1ecb71f0edb5b7632b1013faaed77277e2b4#r54560114

Ah, crap, I saw the notification the other day, but I was at the park (https://photos.app.goo.gl/Z9ZR8EvZ2Cs3wMM3A) with my phone; not exactly the best place to test it ehehehehehe

I'm now in front of my laptop with an RDP session to the server at work, so I can totally test it. I'll let you know on GitHub and thanks for reminding me.

[real.finder]

in this (https://github.com/AviSynth/AviSynthPlus/tree/master/distrib/Readme) read me there are
"array" or "val_array": array of any type.
When unnamed, then this kind of parameter must be the very last one.
Unnamed free-typed parametes cannot be followed by any other parameter.
Translates to ".*" in a plugin parameter definition rule.
if I save this as Average.avsi
Function Average(array a)
{
LoadPlugin("C:\path\Average.dll")
Average(a)
}

then load it like this

ClearAutoloadDirs()
ColorBars(width=640, height=480, pixel_type="yv12")
import("Average.avsi")
Average(Blur(1),0.5,last,0.5)

it should work but I get
https://i.postimg.cc/4ddwGBQK/Untitled.png (https://postimages.org/)

seems I missed something or something is already missing in avs side (way to convert array to arguments maybe?)

[wonkey_monkey]

It's probably picking up the implict last as the first argument. Try

Blur(1).Average(0.5,last,0.5)

or

ClearAutoloadDirs()
a = ColorBars(width=640, height=480, pixel_type="yv12")
import("Average.avsi")
Average(a.Blur(1),0.5,a,0.5)

[real.finder]

It's probably picking up the implict last as the first argument. Try

Blur(1).Average(0.5,last,0.5)

or

ClearAutoloadDirs()
a = ColorBars(width=640, height=480, pixel_type="yv12")
import("Average.avsi")
Average(a.Blur(1),0.5,a,0.5)

both not work

[wonkey_monkey]

Ah, then probably Average just doesn't handle arrays as input. It's getting only one parameter, the array, which is not the right number as far as it is concerned. You need to separate out the elements and send them as individual parameters (if you can do such a thing).

[real.finder]

yes, that why I said "way to convert array to arguments"

[cretindesalpes]

Another thing: is the Avisynth+ ABI still restricted to MSVC? Or could Clang/MSYS2 or GCC compile plug-ins now?

[FranceBB]

Another thing: is the Avisynth+ ABI still restricted to MSVC? Or could Clang/MSYS2 or GCC compile plug-ins now?

If I remember correctly MSVC yes, Clang yes, GCC no, in the sense that if you compile a plugin with GCC, then you need an Avisynth build compiled with GCC as well, otherwise you're not gonna be able to run such plugin, so you can't mix GCC builds of plugins with MSVC builds of Avisynth and since almost every build of everything is done with MSVC, then no one compiles with GCC.
Stephen (also called "qyot" for the Doom9 friends) did some successful experiments a while ago in which he compiled Avisynth with GCC and tried to use GCC Compiled plugins with it and it worked, however no one uses it for obvious reasons...
At least this is true on Windows, but it's different for Linux and Mac OSX (yep, as hard to believe as it is, Avisynth is cross platform now eheheheh)

[qyot27]

Another thing: is the Avisynth+ ABI still restricted to MSVC? Or could Clang/MSYS2 or GCC compile plug-ins now?
Clang is probably okay if you're using clang-cl. Other than that, there be dragons. The only way that's guaranteed would be as a C plugin (in which case the GCC-centric symbol logic is the assumed default), not a C++ plugin.

And currently something has broken the ability for FFmpeg to use GCC builds, although the GCC builds actually compile. It definitely is something on our side, since archived builds (r2831, from 2019) work with the same newer (May 2021) FFmpeg binary.

[Myrsloik]

Another thing: is the Avisynth+ ABI still restricted to MSVC? Or could Clang/MSYS2 or GCC compile plug-ins now?

It's still the MS C++ ABI only. The only two compilers that support it are MSVC and clang-cl (not other variants of clang, they use the GCC ABI). It's trivial to try out clang-cl, simply select clang in the visual studio installer and then you can choose it as the platform toolset. Binaries are usually noticeable faster.

Do however note that clang-cl doesn't understand the instruction set selection (as done through property pages) very well which can cause problems with intrinsics being rejected.

[cretindesalpes]

I see. So I’ll stick with MSVC for the Avisynth+ version of fmtconv for the moment.

[real.finder]

I see. So I’ll stick with MSVC for the Avisynth+ version of fmtconv for the moment.

if MSVC is not good for you, you can always use C API of avs/avs+ https://github.com/Asd-g/AviSynth-VMAF/commit/5c1a5e08fc6c8979dd3fec2a87c9cac81f082269

AviSynth+ can even autoload C plugins unlike the old AviSynth, also you don't need to use loadCplugin in avs+, loadplugin work with it too

[vcmohan]

Sorry for asking a simple question. How does one read a double value for an input parameter by the plugin? args[n].AsFloat() I vaguely remembered actually outputs a double but when I tried args[n].AsFloat(0.001) I got a warning of truncating double to float. The input string has f for the value.

[cretindesalpes]

You can use AsDblDef(), but anyway the stored type within the AVSValue is always a float.

[StainlessS]

You can append default value with trailing 'f', args[n].AsFloat(0.001f)
[not sure if Avs+/visual studio, gives warning now for default value as double without trailing f]
EDIT: AsDblDef() probably better way, never used it.
With VS2008 and earlier, I sometimes had to use something like eg

d = (double)args[n].AsFloat(0.001f)
f = (float)args[n].AsFloat(0.001f)

[StvG]

Sorry for asking a simple question. How does one read a double value for an input parameter by the plugin? args[n].AsFloat() I vaguely remembered actually outputs a double but when I tried args[n].AsFloat(0.001) I got a warning of truncating double to float. The input string has f for the value.

The original post. (https://forum.doom9.org/showthread.php?p=1937801&highlight=double#post1937801)

There is an AsFloatf version of AsFloat that should be used to avoid warnings. This function exists for this very reason.
EDIT: maybe I explained the opposite: to avoid return value warning. You can use AsDblDef, but since there is no double in AVSValue there is no point of doing that.

Side note: internally there is no double type in Avisynth. Double and 64 bit integer types are impossible to implement. Reason: AVSValue defined in avisynth.h has its maximum value size dependent of the size of pointer - pointer type is 4 bytes on a 32 bit system - so only a 32 bit float can be held there, no 64 bit double. 64 bit systems would easily support them - of course this needs an interface update - I had plans for that but there are always more important stuffs in the development queue. With the extinction of 32 bit Avisynth versions nobody will care if interface would change.

[DTL]

Some idea to Avisynth core clip properties:

Add property to indicate full-band color 4:4:4 clip and half-color band (half vertical, half-horizontal or half-both, may be bitmask ?).

For example fresh non-distorted in spectrum 4:4:4 clip upconverted from 4:2:0 source is half-banded in color in both vertical and horizontal directions. But the next in chain filters can not get this information currently. Also this property need to be user-controllable so if user damage half-banding it may set property to full-band.

It may be useful hint for processing like color subsampling processing and other.

Also use it automatically in Convert4:4:4To4:2:0 and from 4:4:4To4:2:2 operations with defaulting chromaresample to sinc for half-band marked clips and may be current default bicubic (gauss prefferable with fixed p-param adjusted after tuning) for full-band marked 4:4:4 sources.
Also it is very great to add to Contert() the abiliti to pass kernel chromaresampler params (for example p for gauss) because currently used defaults (?) may be far from good for chromaresample.

[FranceBB]

Also it is very great to add to Contert() the ability to pass kernel chromaresampler params (for example p for gauss) because currently used defaults (?) may be far from good for chromaresample.

^This^
As result of this discussion: Link (https://forum.doom9.org/showthread.php?t=160038&page=13)

In Converttoyv12, YUY2, yv16, yv24, YUV420, YUV422, YUV444 etc only the built in resizing kernels are supported and with no additional parameters, so that is actually limiting.
I know that just only around 5% of the Doom9 population is actually even bothering to specify the resizing kernel when changing chroma, but anyway I'm in favor of a full args support in the default built-in conversion, to be fair.

[vcmohan]

You can append default value with trailing 'f', args[n].AsFloat(0.001f)
[not sure if Avs+/visual studio, gives warning now for default value as double without trailing f]
On MS VC 2019 community version tried:-
float d = args[].AsFloat(0.5f)
warning conversion double to float
float d = args[].AsFloat(0.5)
argument truncation from double to float
initializing from double to float.
So it always expects a trailing f in default value, but outputs a double. A bit confusing and is not right.
double d = args[].AsDblDef(0.5) works ok

[cretindesalpes]

You also can use AsFloatf() which takes a float and returns a float.

[StainlessS]

Dont think AsDblDef() or AsFloatf() work in avs 2.58, so I dont use either.

[DTL]

An idea - it looks avisynth support dropping out only error messages. But it may be good for hinting and warning of user to have some method to output warning messages too. In complex processing it may be method to put info about not very best current processing path or where developer not sure and/or ask to pay special attention (where quality may be degraded or where parameter in most cases need adjustment etc).

[Dogway]

Is there a way to know how string floats get converted to single floats in Expr? I was reading IEEE 754 but nothing on it hinted the following findings:
Merge(a,b,0.25)

equals

Expr(a,b,"x 0.749999 * y 0.249999 * + ")
or
Merge(a,b,0.5)

equals

Expr(a,b,"x y + 0.500001 * ")


For example I tested in this (https://www.h-schmidt.net/FloatConverter/IEEE754.html)online converter, and 0.25 turns out as 0.25 and 0.5 as 0.5.

[cretindesalpes]

Expr only works with float (no double). Integer multiples of powers of 2 can be represented exactly (like 0.75 = 3 * 2^-2). Other values are rounded, and the exact rounding process depends on the implementation of the standard library used to compile the program. Generally you can expect about 6 digits of precision.

[Dogway]

Yes I know, but as I was suggested (https://forum.doom9.org/showthread.php?p=1946601#post1946601) you can help the rounding by declaring more float values or like in the example I posted adding or substracting epsilon

I would like to know where I can read more about this, precisely related to the Expr rounding method, and if this also affects summation and not only multiplications.

[RRD]

On https://avs-plus.net/, at the bottom of the page, "You can find us on our Doom9 thread (http://forum.doom9.org/showthread.php?t=168856)" links to the old thread (http://forum.doom9.org/showthread.php?t=168856) instead of the new one (http://forum.doom9.org/showthread.php?t=181351).

[Dogway]

Since mid-grey in YUV (128) is not centered within the range (16-235), is it ok that range conversions don't keep mid-grey? Converting from TV range 128 to PC range leads to 130, and the opposite to 126. Shouldn't we compensate for that?

[wonkey_monkey]

"Mid-grey" is different in TV range than in PC range, just as black and white are. No need for compensation, I think.

Also bearing gamma in mind it's not even really "mid".

[Dogway]

I thought mid-grey for PC levels was also 128, at least for sRGB. I wasn't talking about scene referred middle grey.
Here's a little experiment, ColorYUV "correctly" converts mid grey to PC range 130, but then when converting that to RGB it keeps 130 when RGB mid-grey is known (https://en.wikipedia.org/wiki/Middle_gray#Table_of_middle_grays)to be 128 for sRGB.
BlankClip(width=256,height=256,color=$828282,pixel_type="YV12")
ColorYUV(levels="TV->PC",matrix="Rec709")
ConvertToPlanarRGB(matrix="PC.709")

[DTL]

Can Avisynth+ support sequential function calling in scripting with freeing memory after each return ? I trying to make full-frame 1080p (and need 2160p) rasterizer of text for shifting with Animate(). Text is rendered with Subtitle and 100x SSAA with small blocks, but number of blocks is about 12 and system died with swapping at 4 GB memory PC. I think it is because all calls to function with Subtitle() and Overlay() are performed in parallel for each frame ? May exist solution for serializing ?

I can not even fill 1080p frame with text - I trying to render blocks of 132 lines into raw files for later stacking.

Current script:

LoadPlugin("plugins_JPSDR.dll")

function Ast1(clip c, int iFontSize, string strText, string strFont, float fHPos, int iVPos)
{
iHSubPos=fHPos*10 - (10*Int(fHPos))
temp=BlankClip(c, width=8000, height=(iFontSize*15)+40, color=$101010)
temp=Subtitle(temp, strText, x=iHSubPos, font=strFont, size=iFontSize*10, align=4, halo_color=$FF000000, text_color=$00e0e0e0)
temp=UserDefined2ResizeMT(temp, temp.width/10, temp.height/10, b=130,c=23)
return Overlay(c, temp, Int(fHPos), iVPos)
}

function Ast2(clip c, int iFontSize, string strText, string strFont, float fHPos, int iVPos)
{
iHSubPos=fHPos*10 - (10*Int(fHPos))
temp=BlankClip(c, width=8000, height=(iFontSize*15)+60, color=$101010)
temp=Subtitle(temp, strText, x=iHSubPos, font=strFont, size=iFontSize*10, align=4, halo_color=$FF000000, text_color=$00e0e0e0)
temp=UserDefined2ResizeMT(temp, temp.width/10, temp.height/10, b=95,c=-10)
return Overlay(c, temp, Int(fHPos), iVPos)
}

function MyFullFrameTextH(clip c, string strText, float fHPos, int iVShift)
{
c=Ast1(c, 10, strText, "Arial", fHPos + 10, iVShift + 20)
c=Ast1(c, 10, strText, "Times New Roman", fHPos + 70, iVShift + 20)
c=Ast1(c, 10, strText, "Wolfgang Amadeus Mozart", fHPos + 130, iVShift + 20)

c=Ast1(c, 20, strText, "Arial", fHPos + 180, iVShift + 20)
c=Ast1(c, 20, strText, "Times New Roman", fHPos + 300, iVShift + 20)
c=Ast1(c, 20, strText, "Wolfgang Amadeus Mozart", fHPos + 410, iVShift + 20)

c=Ast1(c, 35, strText, "Arial", fHPos + 490, iVShift + 20)
c=Ast1(c, 35, strText, "Times New Roman", fHPos + 690, iVShift + 20)
c=Ast1(c, 35, strText, "Wolfgang Amadeus Mozart", fHPos + 880, iVShift + 20)

c=Ast1(c, 50, strText, "Arial", fHPos + 1000, iVShift + 20)
c=Ast1(c, 50, strText, "Times New Roman", fHPos + 1280, iVShift + 20)
c=Ast1(c, 50, strText, "Wolfgang Amadeus Mozart", fHPos + 1550, iVShift + 20)

c=Ast2(c, 10, strText, "Arial", fHPos + 10, iVShift + 80)
c=Ast2(c, 10, strText, "Times New Roman", fHPos + 70, iVShift + 80)
c=Ast2(c, 10, strText, "Wolfgang Amadeus Mozart", fHPos + 130, iVShift + 80)

c=Ast2(c, 20, strText, "Arial", fHPos + 180, iVShift + 80)
c=Ast2(c, 20, strText, "Times New Roman", fHPos + 300, iVShift + 80)
c=Ast2(c, 20, strText, "Wolfgang Amadeus Mozart", fHPos + 410, iVShift + 80)

c=Ast2(c, 35, strText, "Arial", fHPos + 490, iVShift + 80)
c=Ast2(c, 35, strText, "Times New Roman", fHPos + 690, iVShift + 80)
c=Ast2(c, 35, strText, "Wolfgang Amadeus Mozart", fHPos + 880, iVShift + 80)

c=Ast2(c, 50, strText, "Arial", fHPos + 1000, iVShift + 80)
c=Ast2(c, 50, strText, "Times New Roman", fHPos + 1280, iVShift + 80)
c=Ast2(c, 50, strText, "Wolfgang Amadeus Mozart", fHPos + 1550, iVShift + 80)

return c

}

function MyFullFrameTextH_upper(clip c, string strText, float fHPos, int iVShift)
{
c=Ast1(c, 10, strText, "Arial", fHPos + 10, iVShift + 20)
c=Ast1(c, 10, strText, "Times New Roman", fHPos + 70, iVShift + 20)
c=Ast1(c, 10, strText, "Wolfgang Amadeus Mozart", fHPos + 130, iVShift + 20)

c=Ast1(c, 20, strText, "Arial", fHPos + 180, iVShift + 20)
c=Ast1(c, 20, strText, "Times New Roman", fHPos + 300, iVShift + 20)
c=Ast1(c, 20, strText, "Wolfgang Amadeus Mozart", fHPos + 410, iVShift + 20)

c=Ast1(c, 35, strText, "Arial", fHPos + 490, iVShift + 20)
c=Ast1(c, 35, strText, "Times New Roman", fHPos + 690, iVShift + 20)
c=Ast1(c, 35, strText, "Wolfgang Amadeus Mozart", fHPos + 880, iVShift + 20)

c=Ast1(c, 50, strText, "Arial", fHPos + 1000, iVShift + 20)
c=Ast1(c, 50, strText, "Times New Roman", fHPos + 1280, iVShift + 20)
c=Ast1(c, 50, strText, "Wolfgang Amadeus Mozart", fHPos + 1550, iVShift + 20)

return c

}

function MyFullFrameTextH_lower(clip c, string strText, float fHPos, int iVShift)
{
c=Ast2(c, 10, strText, "Arial", fHPos + 10, iVShift + 80)
c=Ast2(c, 10, strText, "Times New Roman", fHPos + 70, iVShift + 80)
c=Ast2(c, 10, strText, "Wolfgang Amadeus Mozart", fHPos + 130, iVShift + 80)

c=Ast2(c, 20, strText, "Arial", fHPos + 180, iVShift + 80)
c=Ast2(c, 20, strText, "Times New Roman", fHPos + 300, iVShift + 80)
c=Ast2(c, 20, strText, "Wolfgang Amadeus Mozart", fHPos + 410, iVShift + 80)

c=Ast2(c, 35, strText, "Arial", fHPos + 490, iVShift + 80)
c=Ast2(c, 35, strText, "Times New Roman", fHPos + 690, iVShift + 80)
c=Ast2(c, 35, strText, "Wolfgang Amadeus Mozart", fHPos + 880, iVShift + 80)

c=Ast2(c, 50, strText, "Arial", fHPos + 1000, iVShift + 80)
c=Ast2(c, 50, strText, "Times New Roman", fHPos + 1280, iVShift + 80)
c=Ast2(c, 50, strText, "Wolfgang Amadeus Mozart", fHPos + 1550, iVShift + 80)

return c

}


function MyAnimFF(clip c, float fHPos, int VPos)
{
temp=MyFullFrameTextH_upper(c, " TXT Sample ", fHPos, VPos)
return MyFullFrameTextH_lower(temp, " TXT Sample ", fHPos, VPos)
# MyFullFrameTextH(c, " TXT Sample ", fHPos, VPos)
}

full=BlankClip(501,1920,135,"Y16",fps=25, color=$101010)

Animate(full, 1,500, "MyAnimFF", 50, 0, 1, 0)

ConvertToRGB24()



Attempt to make cut function MyFullFrameTextH(clip c, string strText, float fHPos, int iVShift) into 2 pieces did not helps any - the amount of used memory is the same. The size of intermediate frames for AA operation is max about 8000x810 and it is about 12 MByte in size in Y16 format.

[poisondeathray]

I thought mid-grey for PC levels was also 128, at least for sRGB. I wasn't talking about scene referred middle grey.
Here's a little experiment, ColorYUV "correctly" converts mid grey to PC range 130, but then when converting that to RGB it keeps 130 when RGB mid-grey is known (https://en.wikipedia.org/wiki/Middle_gray#Table_of_middle_grays)to be 128 for sRGB.
BlankClip(width=256,height=256,color=$828282,pixel_type="YV12")
ColorYUV(levels="TV->PC",matrix="Rec709")
ConvertToPlanarRGB(matrix="PC.709")



But if you use that assumption , and defining "middle grey" in sRGB as 128,128,128, shouldn't you should start with sRGB ?

RGB 128,128,128 => Y 126 in limited range, Y 128 in full range

So how are you defining "middle grey" in Y ?



BlankClip(width=256,height=256, color=$808080, pixel_type="RGB24")
ConvertToYV12(matrix="Rec709")
ColorYUV(levels="TV->PC",matrix="Rec709")
ConvertToPlanarRGB(matrix="PC.709")


RGB 128,128,128

[Dogway]

Well, YUV can also be sRGB (aka PC.709), so it wasn't a fault on my part, but reading on YCbCr (https://en.wikipedia.org/wiki/YCbCr#ITU-R_BT.601_conversion)I concluded that Y middle grey is actually 125.5.

Coming from analog (full range) RGB where 0.5 is asummed to be middle grey and converting to YUV by 0.5*(235-16)+16 = 125.5

This actually correctly transforms to 127.5 in PC range.

[Dogway]

Can this be fixed? Expr() gets the variables mixed in the first example.

Expr("x[0,0] M^ x[1,0] N^ x[2,0] O^
M N + WA@ O + WB@ N - WB WA + + 4 /","")


Expr("x[0,0] M^ x[1,0] N^ x[2,0] O^
M N + W@ O + X@ N - X W + + 4 /","")

[pinterf]

Can this be fixed? Expr() gets the variables mixed in the first example.

Expr("x[0,0] M^ x[1,0] N^ x[2,0] O^
M N + WA@ O + WB@ N - WB WA + + 4 /","")


Expr("x[0,0] M^ x[1,0] N^ x[2,0] O^
M N + W@ O + X@ N - X W + + 4 /","")

Sure. WA and WB? Anyway that must be an obvious bug, thanks for the report
EDIT: there was a bug with handling multi-letter parameter names ended with @. Fix is coming soon

[pinterf]

Fixed in
Avisynth+ 3.7.1 test build 11 (20210911) (https://drive.google.com/uc?export=download&id=1L0P-QNyWNiylKd6KTE9kB52S5eS-fRER)

[Reel.Deel]

Pinterf,

I found a bug in Overlay when the bitdepth is 16 and opacity is less than 1. Happens in RGB and YUV but the problem goes away at any other lower bitdepths.

base = Blankclip(width=270, height=69, color=$FFFFFF, pixel_type="YUV444P16")
over = Blankclip(width=270, height=19, color=$FF0000, pixel_type="YUV444P16")
mask = ExtractY(over).mt_lutspa(mode="relative", expr="x range_max *")

Overlay(base, over, mask=mask, x=0, y=3, mode="blend", opacity=.75)

[pinterf]

Pinterf,
I found a bug in Overlay when the bitdepth is 16 and opacity is less than 1. Happens in RGB and YUV but the problem goes away at any other lower bitdepths.

What processor type do you have?
edit: issue emerges from SSE4.1 and up

[Reel.Deel]

What processor type do you have?

Intel i7 4930k. Let me update to the latest test version avs+. I have 3.7 installed right now.

Edit:

https://i.ibb.co/nMrSpRX/cpu.png (https://imgbb.com/)

Edit 2:

Still the same behavior with latest test version and setting SetMaxCPU("sse3") or less fixes the problem.

[pinterf]

Fixed.
Avisynth+ 3.7.1 test build 12 (20210911) (https://drive.google.com/uc?export=download&id=1tZKYyQEaru1K_QyD_Q4CFiW8arze6aWl)

[FranceBB]

Wow, so many builds, so little time to upgrade ehehehehe
Thanks for the new version, Ferenc, testing now. :)

[Dogway]

Sure. WA and WB? Anyway that must be an obvious bug, thanks for the report
EDIT: there was a bug with handling multi-letter parameter names ended with @. Fix is coming soon

Thank you! It was hard to work around it, I will test new version and if it works it allows for great optimizations in Expr()

[Reel.Deel]

Fixed.
Avisynth+ 3.7.1 test build 12 (20210911) (https://drive.google.com/uc?export=download&id=1tZKYyQEaru1K_QyD_Q4CFiW8arze6aWl)

Just tested the new build and the shift is still there. It is not as drastic as before but with opacity < 1 it shifts the overlay image by 6 pixels to the left. Now the shift is consistent, before it would change depending on the opacity value.

[pinterf]

Just tested the new build and the shift is still there. It is not as drastic as before but with opacity < 1 it shifts the overlay image by 6 pixels to the left. Now the shift is consistent, before it would change depending on the opacity value.
Thanks. I'm curious. Since I accidentally switched off my remote developer PC instead of the actual :) one over TeamViewer we'll have to wait until Monday.

[Reel.Deel]

Thanks. I'm curious. Since I accidentally switched off my remote developer PC instead of the actual :) one over TeamViewer we'll have to wait until Monday.

Excellent way to get off of working on a Sunday :D

----

I think I found another discrepancy. Using the color presets from here (http://avisynth.nl/index.php/Color_presets) I cannot get the same results in Blankclip; it returns a different color.

Blankclip(width=132, height=152, color=$9400D3, pixel_type="rgb24") # color_darkviolet

I know it worked in the past but have not needed a colored blankclip in a while.

[DTL]

it's simply gamma aware resampling, many alternatives.
also resampling under linear light introduces way more ringing and I don't actually consider it "HQ"

The amount of ringing depends both on source and used at resampling process 'conditioning' filter. One of the reason to have content for final distribution be 'conditioned' against ringing in linear domain - the use of simple and non consumer-adjustable filter at displaying (ideal LPF - sinc).
Also at the new dark age of very different HDR-TFs it allow to prepare content once and use any HDR (SDR) TF for compression and expanding of bitdepth without preparation special version 'conditioned' for each TF. Also may be made formats convertors more simple and cheap without putting more evil in the spectrum. Like Any_input_TF->Linear->Any_output_TF without re-conditioning.
It is good to consider moving pictures data in TF-domain (like old gamma of newer more advanced HDR TFs) as compressed (it is really sort of bitdepth compression not very great for SDR but about 3:1 for HDR) and not attempt to make processing in this compressed form. As well as typically user can not perform MPEG-domain compressed data resize because it will damage data and make non-decodable. Same is with TF-compressed data - any resize will make data damaged and lower the quality of decoded form.

It was reply to old enough post but I hope in the progress of Avisynth development (and in the dark age of different HDR TFs + SDR) the more attention will be put to the moving pictures data processing and the possible spectrum-changing operations will be performed in linear domain.

[qyot27]

I think I found another discrepancy. Using the color presets from here (http://avisynth.nl/index.php/Color_presets) I cannot get the same results in Blankclip; it returns a different color.

Blankclip(width=132, height=152, color=$9400D3, pixel_type="rgb24") # color_darkviolet

I know it worked in the past but have not needed a colored blankclip in a while.
I'm pretty sure I know what happened, since 3.7.0 is fine, and so is both pixel_type="rgbp" by itself as well as followed by ConvertToRGB24(), or using pixel_type="rgb32" and then RemoveAlphaPlane().

In the meantime, you can work around it by shifting the last value inward: $94D300

[qyot27]

Okay, it *should* be fixed on git now.

[Reel.Deel]

Thanks qyot27. I'll wait for the next test release and report back if it's still troublesome.

[pinterf]

Just tested the new build and the shift is still there. It is not as drastic as before but with opacity < 1 it shifts the overlay image by 6 pixels to the left. Now the shift is consistent, before it would change depending on the opacity value.
Are you sure, there is a shift? It seemed to be a mod8 issue: the rightmost pixels in the line (1-7 pixels, depending on the width) were not processed; 10-16 bits and 32 bit formats are affected.

[Reel.Deel]

Are you sure, there is a shift? It seemed to be a mod8 issue: the rightmost pixels in the line (1-7 pixels, depending on the width) were not processed; 10-16 bits and 32 bit formats are affected.

Yes, just double checked.

https://i.ibb.co/61MQbNT/overlaybug.png (https://imgbb.com/)

The shift here is 6 pixels. Image is an animated png, hope you can see it correctly.

[pinterf]

Yes, just double checked.

https://i.ibb.co/61MQbNT/overlaybug.png (https://imgbb.com/)

The shift here is 6 pixels. Image is an animated png, hope you can see it correctly.
Width is 270 but only 264 pixels were processed (264 modulo 8 = 0). The rightmost 6 pixels were omitted.

[Reel.Deel]

Width is 270 but only 264 pixels were processed (264 modulo 8 = 0). The rightmost 6 pixels were omitted.

Ahh I see what you're saying now. Did not look close enough, sorry. No more shift, just pixels being omitted. That's not intended behavior, is it?


Edit:
Fixed on git. I'm gonna build another test later on today.

Thanks pinterf. I'll test again whenever it gets released. Just FYI, I've not been stockpiling these bugs and just waiting for you to return :D. I discovered them recently when I was testing my OverlayPlus script.

[pinterf]

Ahh I see what you're saying now. Did not look close enough, sorry. No more shift, just pixels being omitted. That's not intended behavior, is it?
Fixed on git. I'm gonna build another test later on today.

[Dogway]

Can chroma motion search be improved by feeding normalized chroma planes or is mvtools already doing it internally?

Also is the string line carriage working correctly on Linux or other systems (to Windows)? I suspect this (https://forum.doom9.org/showthread.php?p=1952076#post1952076) might be a bug related to it.

[pinterf]

Can chroma motion search be improved by feeding normalized chroma planes or is mvtools already doing it internally?

Planes are processed as they are received, no normalization occurs. From the docs: adjust the luma:chroma ratio

scaleCSAD

Fine tune chroma part weight in SAD calculation (since 2.7.18.22)
Possible values for luma:chroma SAD ratio
-2: 4:0.5
-1: 4:1
0: 4:2 (default, same as the native ratio for YV12)
1: 4:4
2: 4:8



Also is the string line carriage working correctly on Linux or other systems (to Windows)? I suspect this (https://forum.doom9.org/showthread.php?p=1952076#post1952076) might be a bug related to it.
Expr treats space, CR, LF and TAB as allowed separators within an expression string. (in 3.7.0 only space was allowed).

I need more info to reproduce it.

[Dogway]

Ah thank you. Guess I can do something like this to chroma:
ExtractU().expr(" x range_half - range_max cmax cmin - / * abs 6 *")

Will check if it improves matters.

I will investigate for the other issue.

[Dogway]

Why is not possible to Eval arrays? is there a way to do this?

function ArrayAdd( int_array a, int_array b) {

as = ArraySize(a)-1
bs = ArraySize(b)-1
na = ""
for (i = 0, as, 1) {
cm = i != as ? "," : ""
na = na + String(Eval(Format("a[{i}]"))) + cm
}
Eval("["+na+"]") }

Also I'm required to set argument to 'int_array' since val_array will not parse ints (didn't test other types)

[pinterf]

Missing "return"?

[real.finder]

Dogway, keep in mind that script array only "'*': zero or more" while plugins can be "'+': one or more" and "'*': zero or more" start read from https://github.com/AviSynth/AviSynthPlus/issues/226#issuecomment-910076325 and I think you made this for bm3d, and bm3d is + array one

there are suggestion to add + array to script function https://github.com/AviSynth/AviSynthPlus/issues/226#issuecomment-910186715 but I think pinterf still wait for better name https://github.com/AviSynth/AviSynthPlus/issues/226#issuecomment-910230404

[Dogway]

Nevermind, just made it:
function ArrayAdd( int_array a, int_array b) {

as = ArraySize(a)-1
bs = ArraySize(b)-1
na = ""
for (i = 0, as+bs+1, 1) {

o = i - as - 1
cm = i != as+bs+1 ? "," : ""
na = na + ( i > as ? String(Eval(Format("b[{o}]"))) : \
String(Eval(Format("a[{i}]")))) + cm
}

Eval("["+na+"]") }

Still doesn't work with val_array though



EDIT: apparently the example here (https://github.com/AviSynth/AviSynthPlus/blob/47b6141ac27fc5c33cb4186695cca88b6961b3c8/distrib/Readme/readme.txt#L338) was working at some point, but now it doesn't.

Function Summa(array "x", array "y", int "N")
{
sum = 0.0
FOR(i=0,N-1) {
sum = sum + x[i] * y[i]
}
return sum
}

[pinterf]

This syntax is working for me

Function Summa(float_array "x", float_array "y", int "N")
{
sum = 0.0
FOR(i=0,N-1) {
sum = sum + x[i] * y[i]
}
return sum
}

Function Summa2(val_array "x", val_array "y", int "N")
{
sum = 0.0
FOR(i=0,N-1) {
sum = sum + x[i] * y[i]
}
return sum
}

ColorBarsHD()
subTitle(String(Summa([1,2],[3,5],2)) + " " + String(Summa2(x=[1,2],y=[3,5],n=2)))

[pinterf]

And this syntax is valid a well:
(untyped arrays can appear only as named parameters in this example)
function ArrayAdd( int_array a, int_array b) {

as = ArraySize(a)-1
bs = ArraySize(b)-1
na = ""
for (i = 0, as+bs+1, 1) {

o = i - as - 1
cm = i != as+bs+1 ? "," : ""
na = na + ( i > as ? String(Eval(Format("b[{o}]"))) : \
String(Eval(Format("a[{i}]")))) + cm
}

return Eval("["+na+"]")
}

function ArrayAddNamed( val_array "a", val_array "b") {

as = ArraySize(a)-1
bs = ArraySize(b)-1
na = ""
for (i = 0, as+bs+1, 1) {

o = i - as - 1
cm = i != as+bs+1 ? "," : ""
na = na + ( i > as ? String(Eval(Format("b[{o}]"))) : \
String(Eval(Format("a[{i}]")))) + cm
}

return Eval("["+na+"]")
}

ColorBarsHD()
a1 = ArrayAdd([1,2],[4,5,6])
a2 = ArrayAddNamed(a=[11,12],b=[14,15,16])
s = "Unnamed, typed arrays:"
for(i=0,ArraySize(a1)-1) {
s = s + " " + String(a1[i])
}

s = s + " Named params, val_array: "

for(i=0,ArraySize(a2)-1) {
s = s + " " + String(a2[i])
}

SubTitle(s)

[Dogway]

oh, you have to declare the argument, ok, strange but that did the trick thank you.

[pinterf]

Avisynth+ 3.7.1 test build 14 (20210914) (https://drive.google.com/uc?export=download&id=1f1tfB40Ny0rZcmBApSiulFl6ndfpta4z)
Since last test build:
20210914 WIP
------------
- New: script functions now supports avisynth function array signature '+' (one or more) with _nz type suffix.
Previously only '*' style (zero or more) was supported by the original naming.
E.g.: val_array -> .* val_array_nz -> .+, int_array -> i* int_array_nz -> i+
Others: bool_array_nz, float_array_nz, string_array_nz, clip_array_nz, func_array_nz.
- Fix: Overlay "blend" 10+ bit clips and "opacity"<1 would leave rightmost non-mod8 (10-16 bit format) or non-mod4 (32 bit format) pixels unprocessed.
- Fix BlankClip rgb24 color part swap (regression in 3.7.1 test builds)

[DTL]

Based on note in https://tech.ebu.ch/docs/techreview/trev_304-rec601_bbc.pdf
More satisfactory was the consideration of chrominance filtering which recognized the need for
sharp-cut filters at all conversions except the last one. This allowed the bandwidth needed for
chroma-key and other processing to be retained through the system; however, the inclusion of a
slow roll-off in the composite coder for analogue broadcasts or in the picture monitor for direct
component signals avoided the dreadful chrominance ringing that would otherwise occur.

It looks current ConvertToRGB() and all other chroma-upsampling Convert() like from 4:2:0 and 4:2:2 to 4:4:4 need to be supplemented with additional option like 'in-between' conversion or 'last/display' conversion. Or may be at least Convert() filter documentation need to be supplemented with this feature of digital sampled moving pictures systems design (from rec.601 and may be up to latest HDR).
I sometime use ConvertToRGB24() before checking output result in VirtualDub and it creates some distorted output because for 'last' conversion the UV planes need to be filtered with some low-pass slow-roll-off filter to fix posible colour-difference source of ringing.
Or special function need to be used for 'monitor conversion to RGB' like (pseudo-functions)

u_plane=ExtractU()
v_plane=ExtractV()
u_plane=Blur(u_plane,some_param) or GeneralConvolution(u_plane,some_param) or special ColourDifferenceLastMonitorProc(u_plane)
v_plane=Blur(v_plane,some_param) or GeneralConvolution(v_plane,some_param) or special ColourDifferenceLastMonitorProc(v_plane)
yuv_back=CombinePlanes(last,u_plane,v_plane)
ConvertToRGB(yuv_back)

[LigH]

Do you mean something like the "coring" parameter of some AviSynth functions (disabling the clamping to TV scale in YUV modes)?

[DTL]

Do you mean something like the "coring" parameter of some AviSynth functions (disabling the clamping to TV scale in YUV modes)?

I made example of working conversion functions to show the difference in the post https://forum.doom9.org/showthread.php?p=1952273#post1952273 .

Also the post https://forum.doom9.org/showthread.php?p=1952213#post1952213 have link to updated hand-drawing schematic of different sub-sampled to 4:4:4 decoding - for final/control displaying and for intermediate processing.

Because the exact 'anti-ringing' filter in display/final transform looks like not standard-defined it also possible to have 'colour-sharpness' adjustment at 422(420)ToRGB transform to make adjustments between 'film-looking' with smooth transients and 'video-look' with peaked/overshooted.

In that example I use UserDefined2ResizeMT() with src_left !=0 to enable filter-processing (convolution with kernel) without actual resizing of UV channels.

[DTL]

Oh - again some (minor) complain for Avisynth resampler:

LoadPlugin("fmtcavs.dll")

ColorBarsHD(9600,1000, pixel_type="YV24")
ConvertBits(16) # YUV 444 16
Crop(0,0,9600,500)
AddBorders(300,300,300,300)

GaussResize(width/10,height/10, p=10)

SincResize(last.width*4, last.height*4, taps=16)
#fmtc_resample(w=last.width*4, h=last.height*4, kernel="sinc", taps=16)

ConvertBits(8)

Levels(0,1,20,0,255).Crop(1000,0,width-1000, height).ConvertToRGB24(matrix="PC.709")


It produces additional (and colored) ringing-like lines (though it constant not-faded amplitude and different frequency) from the very edges of the frame.
Image: https://i3.imageban.ru/out/2021/09/17/0097b84914ff6256cd2288d58ca4e007.png

When trying to process in 'linear light' and convert to and from modern highly-non-linear TFs like HDR the bug significally increases in amplitude.

The kernel's response between SincResize and fmtc(sinc) checked and looks like equal. So I assume it is another issue with edge-workarounds in the avisynth's resampler.

The fmtc(sinc) frame edges is clear (there is some minor ringing from the useful data transitions but it correct).

SincLin2ResizeMT in jpsdr's plugin also affected - same resampler core.

LanczosResize(taps=16) output clear.

[pinterf]

It produces additional (and colored) ringing-like lines (though it constant not-faded amplitude and different frequency) from the very edges of the frame.
Image: https://i3.imageban.ru/out/2021/09/17/0097b84914ff6256cd2288d58ca4e007.png

The kernel's response between SincResize and fmtc(sinc) checked and looks like equal. So I assume it is another issue with edge-workarounds in the avisynth's resampler.


Those lines are independent from the bit depth. Unfortunately I don't really even know what to look for.

[DTL]

The top and bottom buggy lines exist only if top and bottom 'borders' (in the AddBorders()) < 320 (320 few lines, 300 and less full number). If it is 'edge computational bug of pure sinc' it should disappear with moving to SincLin2 weighting of the edge of sinc kernel - but it only shifts buggy lines a bit. Also it absent with fmtc resampler without sinc kernel modification.

With 'borders' 300 it starts (around colour patches) from taps=11, no with taps=12, taps=13 yes, 15 yes - more height, taps=18 - yes, moves to the edges of frame.

With SincResizeMT with possibly extended taps value (to 100..150) it looks dissappear after taps >35..40 (when the edges of kernel fades below some max abs value). Also Lanczos weighting have much more suppression of the wider parts of sinc kernel edges so it do not appear.

So it looks interconnected with 'edges of sinc features' but in fmtc it possibly have workaround (for edges of frame processing) for any taps value and without kernel additional weighting. Though fmtc(sinc) do have 'standard sinc edge issue' in any position inside frame - as shown in https://forum.doom9.org/showthread.php?p=1950633#post1950633 .

It looks the buggy lines depends on distance between useful data and buffer border. If useful data is closer about taps*2 from buffer edge - there is a chance of bug (may be also in some range of taps param like 11..30). So as temporal workaround it may be recommended to pad useful buffer with borders >2*taps size. And crop after processing.

Addition: I test at different CPU and Win10 and the script above (borders '300' and taps 16) produces much smaller number of bright buggy lines.
Image: https://i4.imageban.ru/out/2021/09/18/07c0cf04d5f8fa3ed7f4026c2093b402.png
It looks depends on memory or CPU SIMD. Though at my old home CPU (about Core-2 6400) I tried SetMaxCPU('none') and it changes nothing.

I make temporal workaround function for auto-padding and cropping:

Function SafeSincResize(clip c, int width, int height, int taps)
{
xratio = width/c.width
yratio = height/c.height
tt=2*taps
c=AddBorders(c,tt,tt,tt,tt)
c=SincResize(c, c.width*xratio, c.height*yratio, taps=taps)
return Crop(c,tt*xratio,tt*yratio,width,height)
}

Same should work with SincLin2ResizeMT(). Though the source must have some black borders already to not introduce additional stepping with AddBorders(color=black).

[qyot27]

Okay, so I'm going to need those of you that are really invested in the particulars of the color representation in AviSynth+ to test this for me, as I don't have nearly the amount of samples needed for this:
FFmpeg (https://www.mediafire.com/file/ig4oqq4i9lfs8cu/ffmpeg_r103791%252B10.7z/file)
mpv (https://www.mediafire.com/file/hp1n6zhvqevieop/mpv_r49201.7z/file)

This build of FFmpeg has a patch (https://github.com/qyot27/FFmpeg/commit/2771580eae058a642384ef02187620afb7787663) that reads some of the frame properties from AviSynth+ and populates them in what I hope is the correct fashion. The mpv build is provided because it's more comfortable than ffplay, and because unlike ffplay, it doesn't require additional filtering steps to work on the additional information (read: mpv knows the script is outputting HDR and can tonemap it automatically for SDR displays if that's how you have it set up).

Will require one of the test builds pinterf provided from after April, since there had been a bug in 3.7.0 (and probably the earlier versions that were also inside interface version 8) preventing the C interface from accessing frame properties; if you use 3.7.0, it will crash*. You'll also need a source filter that sets frame properties (I used the test build of ffms3000); propSetInt does work to force the properties to a given value, but that's artificial and I want more real-world examples just in case there's a problem with how the equivalencies between the frame props and FFmpeg's color enums were set up.

*this will be resolved in the final version of the patch, since it's going to require bumping to interface version 9 just to make the detection in FFmpeg work correctly (but bumping to version 9 should probably go hand in hand with more new API functions and a way of future-proofing against bugs like this).

What frame properties is it reading?
_FieldBased (the one that doesn't have to do with color; people interested in interlaced video, take note of this one)
_ChromaLocation
_Primaries
_Transfer
_Matrix
_ColorRange

I used the LG New York HDR UHD 4K Demo (https://4kmedia.org/lg-new-york-hdr-uhd-4k-demo/) for verifying that mpv with the patch can pick up and use the information instead of seeing the script as BT.709. But there's obviously a lot more entries under all those different frame properties, and I have no clue if the way I set this up actually can handle all of them the way they're ostensibly supposed to (admittedly, some of that is also up to the source filters setting the correct value and the other filters in the script not screwing it up somehow, but making sure the script->libavformat chain works as intended is the goal here). I also didn't have any interlaced content on hand to really test the field detection on.

[pinterf]

The top and bottom buggy lines exist only if top and bottom 'borders' (in the AddBorders()) < 320 (320 few lines, 300 and less full number). If it is 'edge computational bug of pure sinc' it should disappear with moving to SincLin2 weighting of the edge of sinc kernel - but it only shifts buggy lines a bit. Also it absent with fmtc resampler without sinc kernel modification.

With 'borders' 300 it starts (around colour patches) from taps=11, no with taps=12, taps=13 yes, 15 yes - more height, taps=18 - yes, moves to the edges of frame.
...

In last days I spent quite a few hours on the topic and feeling now what the problem is. The relevant part is basically untouched since 2002. It is a new part for me so I need more knowledge gathering on my side.
btw this is what they implemented: http://avisynth.nl/index.php/Resampling

[pinterf]

Meanwhile.
Avisynth+ 3.7.1 test build 17 (20210924) (https://drive.google.com/uc?export=download&id=1XUtEI5qL0NPy85v3gDYHHh75w828G7dq)
20210924 WIP
------------
- Expr: allow x.framePropName syntax (Akarin's idea)
Where x is the usual clip identifier letter, and after the . is the name of the frame property.
Nonexistent or non-number frame properties return with 0.0 value
Example (increasing brightness until frame nunmber 255)
ColorbarsHD()
ScriptClip("""propset("medi", current_frame)""")
expr("x.medi","","")

- More checks on array parameters in user defined functions.
Array-typed parameters with "name" have the value "Undefined" when they are not passed.
Note: but the value is defined and is a zero-sized array if the parameter is unnamed, like in other Avisynth functions.
Special thanks to real.finder for not allowing me to be too lazy :)

[DTL]

btw this is what they implemented: http://avisynth.nl/index.php/Resampling

It is described 'common theory' of resampling. Also it is even not very good calls different kernels only as 'different resamplers' - Avisynth have only one resampler engine and it accepts different kernels (named *Resize()) at creation of 'resampling program' for the resampling engine.

Also it have a direct 'bugs' like
"The resampling kernel, sinc(x), is symmetric. That is, sinc(x) = sinc(-x). This means that the samples s(n*T) and s(-n*T) will contribute equally to s(0)."
The main and very important property of sinc(x) that it =1 at sinc(0) and =0 at the all other integer n*pi. That practically means that _samples_ of the data are _independent_ . So no sample of 1D sinc processing do not contribute anything to s(0) (and also to any other sample). But _all_ samples (in theory) contribute to any interpolated (new calculated) samples in-between original input samples. Because in-between sinc(x) !=0.
And all this nice properties of sinc works only for 1D tansform and not work as nice for attempt of 2D processing with rectangular sampling grid because sinc (distance_to_diagonal(angled)_samples*pi) !=0 (and the jinc() is too).

Also it have almost nothing about Gibbs phenomenon and working against ringing. Only mention about sometime ringing occur.

Also that 'common theory' applied to infinite size of buffer (theoretical) and may be not lists the issues with real buffers of non-infinite size at all. Processing these real buffers require special workarounds for resample engines to create less distorted results (at the edges and at some distance from the edge (typical ~=size of kernel, filter support, etc). Part of these workarounds looks like implemented in the 'create resampling program' and part may be in the resampler engine.

"untouched since 2002. "

Now with large dynamic range of HDR and much more curved HDR transfer functions even small bugs at the LSBs of 16bit become highly visible if trying to process HDR in linear and convert back to HDR transfer. With old 8bit and SDR transfers most of small amplitude bugs was about invisible.

[FranceBB]

mpv (https://www.mediafire.com/file/hp1n6zhvqevieop/mpv_r49201.7z/file)

Remember the DNX120 green field issue (https://forum.doom9.org/showthread.php?t=182885)?
Well, I'm glad you compiled a new build of MPV 'cause with your build I could check again and it is actually fixed: https://i.imgur.com/0PGbiB0.png

Cheers ;)




What frame properties is it reading?
_FieldBased (the one that doesn't have to do with color; people interested in interlaced video, take note of this one)
_ChromaLocation
_Primaries
_Transfer
_Matrix
_ColorRange



This is a game changer!! Thank you so much for improving metadata passthrough from Avisynth! I really really really appreciate it!
Testing now but if it works it's gonna make my life so much easier, thank you!! :D

[Dogway]

pinterf, is it possible to add atan2 to expr. Currently I'm using the convoluted method of ternaries which is very slow, this will come useful for cylindrical color spaces:

atan2
"x 0 > y x / atan A@ x 0 < y 0 >= & A pi + x 0 < y 0 < & A pi - x 0 == y 0 > pi 0.5 * y 0 < pi -0.5 * 0 0 ? ? ? ? ? ?"

[Reel.Deel]

pinterf, is it possible to add atan2 to expr.

+1 on that request. I'm currently using Gavino's method:


function atan2(string y, string x) {
# Returns an expression for atan2(y, x) (in range 0-360), corresponding to
# (x == 0 ? (y > 0 ? 90 : (y < 0 ? 270 : 0)) : atan(y/x)*180/pi + (x < 0 ? 180 : (y < 0 ? 360 : 0)))
# Form of expression chosen to evaluate atan() once only.
return "("+x+" == 0 ? ("+y+" > 0 ? 90 : ("+y+" < 0 ? 270 : 0)) : " +
\ "atan(("+y+")/("+x+"))*180/pi + " +
\ "("+x+" < 0 ? 180 : ("+y+" < 0 ? 360 : 0)))"
}

[pinterf]

pinterf, is it possible to add atan2 to expr. Currently I'm using the convoluted method of ternaries which is very slow, this will come useful for cylindrical color spaces:

atan2
"x 0 > y x / atan A@ x 0 < y 0 >= & A pi + x 0 < y 0 < & A pi - x 0 == y 0 > pi 0.5 * y 0 < pi -0.5 * 0 0 ? ? ? ? ? ?"
Probably I could make it work but note that 'atan' will keep being slow because expressions containing trigonometric functions have zero SIMD acceleration, they are executed as plain C interpreted code. Akarin implemented sin and cos acceleration lately for VapourSynth which I have not yet imported into Avisynth.

[tormento]

Probably I could make it work but note that 'atan' will keep being slow because expressions containing trigonometric functions have zero SIMD acceleration, they are executed as plain C interpreted code. Akarin implemented sin and cos acceleration lately for VapourSynth which I have not yet imported into Avisynth.
Can't you use Taylor series? Usually with 5/6 terms you have very good approximation.

[pinterf]

Yes of course this is the way how other functions are implemented. And one should specify the allowed input range and the max error on valid inputs. As it was documented on sin and cos implementations.

[wonkey_monkey]

pinterf, is it possible to add atan2 to expr. Currently I'm using the convoluted method of ternaries which is very slow, this will come useful for cylindrical color spaces:

atan2
"x 0 > y x / atan A@ x 0 < y 0 >= & A pi + x 0 < y 0 < & A pi - x 0 == y 0 > pi 0.5 * y 0 < pi -0.5 * 0 0 ? ? ? ? ? ?"

Is that a work in progress? It seems to have a few bugs and oddities.

[tormento]

As it was documented on sin and cos implementations.
And being tan the ratio of them…

[Dogway]

@wonkey_monkey: It's supposed to be final (for 32-bit). I took the atan2() from Wikipedia example.
# Opposing RGB. Like HSV but with opposing warm and cool axis: Luma-RED/GREEN-YELLOW/BLUE
# https://graphics.stanford.edu/~boulos/papers/orgb_sig.pdf
function RGB_to_oRGB (clip RGB, bool fulls) {

bi = BitsPerComponent(RGB)
fs = Default (fulls, false)

# R'G'B' to L'C'C' to
LCC = [ 0.298967, 0.586421, 0.114612, \
0.500000, 0.500000, -1.000000, \
0.866000, -0.866000, 0.000000]

LCC = MatrixClip(RGB,MatrixTranspose(LCC),"YUV")
R = ExtractR(RGB)
L = ExtractY(LCC)
C1 = ExtractU(LCC)
C2 = ExtractV(LCC)

G = Expr(C2, C1, R, ex_dlut(Format("x 0 > y x / atan A@ x 0 < y 0 >= & A pi + x 0 < y 0 < & A pi - x 0 == y 0 > pi 0.5 * y 0 < pi -0.5 * 0 0 ? ? ? ? ? ?
O@ pi 0.333333 * < O 1.5 * O pi 0.333333 * >= pi O >= & pi 0.5 * O 0.75 * + 0.785398163 - 0 ? ? O - z * y * "), bi, fs), optSingleMode=false)
B = Expr(C1, C2, R, ex_dlut(Format("x 0 > y x / atan A@ x 0 < y 0 >= & A pi + x 0 < y 0 < & A pi - x 0 == y 0 > pi 0.5 * y 0 < pi -0.5 * 0 0 ? ? ? ? ? ?
O@ pi 0.333333 * < O 1.5 * O pi 0.333333 * >= pi O >= & pi 0.5 * O 0.75 * + 0.785398163 - 0 ? ? O - z * x * "), bi, fs), optSingleMode=false)
CombinePlanes(L, G, B, planes="YUV") }


function oRGB_to_RGB (clip oRGB, bool fulls) {

bi = BitsPerComponent(oRGB)
fs = Default (fulls, false)

R = ExtractY(oRGB)
G = ExtractU(oRGB)
B = ExtractV(oRGB)

# L'C'C' to R'G'B'
RGB = [ 1.0, 0.11461199820041656, 0.7433336973190308, \
1.0, 0.11461199820041656, -0.4114006757736206, \
1.0, -0.8853879570960999, 0.16596652567386627]

# Evaluating for atan2(C2,C1) since Expr lacks the operator
G = Expr(B, G, R, ex_dlut(Format("x 0 > y x / atan A@ x 0 < y 0 >= & A pi + x 0 < y 0 < & A pi - x 0 == y 0 > pi 0.5 * y 0 < pi -0.5 * 0 0 ? ? ? ? ? ?
O@ pi 0.5 * < O 0.666666 * O pi 0.5 * >= pi O >= & pi 0.333333 * O 1.333333 * + 2.094395102 - 0 ? ? O + y * z /"), bi, fs), optSingleMode=false)
B = Expr(G, B, R, ex_dlut(Format("x 0 > y x / atan A@ x 0 < y 0 >= & A pi + x 0 < y 0 < & A pi - x 0 == y 0 > pi 0.5 * y 0 < pi -0.5 * 0 0 ? ? ? ? ? ?
O@ pi 0.5 * < O 0.666666 * O pi 0.5 * >= pi O >= & pi 0.333333 * O 1.333333 * + 2.094395102 - 0 ? ? O + x * z /"), bi, fs), optSingleMode=false)

MatrixClip(CombinePlanes(R, G, B, planes="RGB"),MatrixTranspose(RGB),"RGB") }


HSV doesn't use atan2() (but other cylindrical models do)
function RGB_to_HSV (clip R, clip G, clip B, bool fulls) {

bi = BitsPerComponent(R)
fs = Default (fulls, false)

V = Expr(R, G, B, "x y max z max", optSingleMode=false) # Lightness Hexcone
S = Expr(R, G, B, V, "a 0 == 0 a x y min z min - a / ?", optSingleMode=true) # Hexagonal Chroma
H = Expr(R, G, B, V, "a x y min z min - N@
0 == 0
x a == y z - N / % 6
y a == z x - N / 2 +
z a == x y - N / 4 + 0 ? ? ? ? 60 360 / *", optSingleMode=true)

CombinePlanes(H, S, V, planes="RGB") }


function RGB_to_HSV2 (clip R, clip G, clip B, bool fulls) {

bi = BitsPerComponent(R)
fs = Default (fulls, false)


V = Expr(R, G, B, "x y max z max", optSingleMode=false)
S = Expr(R, G, B, V, "a x y min z min - a /", optSingleMode=true)
H = Expr(R, G, B, V, S, "a x - a x y min z min M@ - / R^
a y - a M - / G^
a z - a M - / B^
b 0 == 0
x a == Rm@ y M == & 5 B +
Rm y M != & 1 G -
y a == Gm@ z M == & 1 R +
Gm z M != & 3 B -
Rm 3 G + 5 R - ? ? ? ? ? ? 60 360 / *", optSingleMode=true)

CombinePlanes(H, S, V, planes="RGB") }


function HSV_to_RGB (clip H, clip S, clip V, bool fulls) {

bi = BitsPerComponent(H)
fs = Default (fulls, false)

fix = "1 +"
Hu = " x "+fix+" 360 60 / * "
Ch = " y z * "
X = " 1 H % 2 1 - abs - C * "

m = " z C - " # addition to the end

R = Expr(H, S, V, Hu+" H@ 0 == 0 H 1 < "+Ch+" C@ H 2 < "+X+" X@ H 4 < 0 H 5 < X C ? ? ? ? ? "+m+" +", optSingleMode=false)
G = Expr(H, S, V, Hu+" H@ 0 == 0 "+Ch+" C^ H 1 < "+X+" X@ H 3 < C H 4 < X H 6 < 0 0 ? ? ? ? ? "+m+" +", optSingleMode=false)
B = Expr(H, S, V, Hu+" H@ 2 < 0 "+Ch+" C^ H 3 < "+X+" X@ H 5 < C H 6 < X 0 ? ? ? ? "+m+" +", optSingleMode=false)

CombinePlanes(R, G, B, planes="RGB") }


function HSV_to_RGB2 (clip H, clip S, clip V, bool fulls) {

bi = BitsPerComponent(H)
fs = Default (fulls, false)

# pr = Expr(H, "x 60 / floor")
# se = Expr(H, pr, "x y -")
# a = Expr(S, V, "1 x - y *")
# b = Expr(S, V, se, "1 x z * - y *")
# c = Expr(S, V, se, "1 x 1 z - * - y *")

pr = " x 60 360 / * floor "
se = " x P - "
a = " 1 y - z * "
b = " 1 y SE * - z * "
c = " 1 y 1 "+se+" SE@ - * - z * "

R = Expr(H, S, V, pr+" P@ 0 == P 5 == or z P 4 == "+c+" P 1 == "+b+a+" ? ? ? ", optSingleMode=false)
G = Expr(H, S, V, pr+" P@ 0 == "+c+" P 1 == P 2 == or z P 3 == "+b+a+" ? ? ? ", optSingleMode=false)
B = Expr(H, S, V, pr+" P@ 0 == P 1 == or "+a+" P 2 == "+c+" P 5 == "+b+" z ? ? ? ", optSingleMode=false)

CombinePlanes(R, B, G, planes="RGB") }

EDIT: continued here (https://forum.doom9.org/showthread.php?p=1953120#post1953120).

[real.finder]

since avs+ has array since 3.6, isn't better to add vs ShufflePlanes? or at least update CombinePlanes to make it support arrays same as ShufflePlanes

[pinterf]

+1 on that request. I'm currently using Gavino's method:
standard atan2 returns the range of -Pi..+Pi, I'm gonna keep that convention.
EDIT: new build with atan2
Avisynth+ 3.7.1 test build 18 (20210927) (https://drive.google.com/uc?export=download&id=13-lNFkFHkRg4-mwE2uCI16UbpyrE_REp)

[pinterf]

Avisynth+ 3.7.1 test build 19 (20210928) (https://drive.google.com/uc?export=download&id=1PjSi1wK1ChTpLvvSv42xdufCQe5fTqml)
Expr sin and cos SIMD implementation ported from VapourSynth. Thanks to Akarin for adding the feature.
(The internals of Expr source code has been changed significantly since I have ported Expr many years ago. Now only the basic sin and cos logic and instruction order was ported.)

height =240
BlankClip(width=360, height=height * 2, pixel_type = "YV24")
sin_cos = "cos"
c1=expr(Format("sx 180 / pi * {sin_cos} {height} * {height} + sy < 255 0 ?"),"","", optSSE2=true, optAvx2=true ).SubTitle("AVX2")
c2=expr(Format("sx 180 / pi * {sin_cos} {height} * {height} + sy < 255 0 ?"),"","", optSSE2=true, optAvx2=false ).SubTitle("SSE2")
c3=expr(Format("sx 180 / pi * {sin_cos} {height} * {height} + sy < 255 0 ?"),"","", optSSE2=false, optAvx2=false ).SubTitle("C")
c11=expr(Format("sx 180 / pi * {sin_cos} {height} * {height} + sy < 255 0 ?"),"","", optSSE2=true, optAvx2=true, optsingleMode=true).SubTitle("AVX2 SingleMode")
c12=expr(Format("sx 180 / pi * {sin_cos} {height} * {height} + sy < 255 0 ?"),"","", optSSE2=true, optAvx2=false, optsingleMode=true).SubTitle("SSE2 SingleMode")
Interleave(c1,c2,c3,c11,c12)

You can expect speed ratios something like this (one clip, Y8 and w/o SubTitle)
5160 fps (AVX2)
2282 fps (SSE2)
220 fps (C)
3711 fps (AVX2 single mode)
1703 fps (SSE2 single mode)

[FranceBB]

Wow new version again. I can't keep up with those eheheheh
Downloaded now, thanks ferenc, as always! :)

[Dogway]

Thanks a lot! but what do I do now with my Taylor series?! lol

[tormento]

Thanks a lot! but what do I do now with my Taylor series?! lol
Perhaps they are faster! Never lose hope! :D

[Reel.Deel]

@pinterf

Thank you for the updates and adding atan2 to Expr.

I was looking in the "readme_history", is there any plans to add topleft chroma placement to the convert functions?

On the side note, I just discovered the Text filter and noticed some differences to FreeSub, I think they way the Text filter handles alpha value in text_color and halo_color may be incorrect or inconsistent. I can prepare a script showing the problem if you'd like.

[FranceBB]

s there any plans to add topleft chroma placement to the convert functions?

+1

That would be useful, given the widespread use of 4:2:0 Type 2 nowadays. It would actually allow me to send the avs straight to x265 rather than going through FFmpeg to convert first...

[Reel.Deel]

+1

That would be useful, given the widespread use of 4:2:0 Type 2 nowadays. It would actually allow me to send the avs straight to x265 rather than going through FFmpeg to convert first...

There's already 3 plugins that support topleft (type 2) chroma placement :p so you can already do that. But seeing how the convert functions already support "Rec2020" and "PC.2020" matrix I think it would be logical to also support the corresponding chroma placement.

[pinterf]

Isn't top left the same as DV?

[Reel.Deel]

Isn't top left the same as DV?

I'm not sure, all I know is that chromaloc type 1 is known as mpeg1, center, or jpeg and chromaloc type 0 is known as mpeg2 or left.

Here's page 430/31 from Recommendation ITU-T H.265:

https://i.ibb.co/0CTXy4Q/page430.png (https://ibb.co/y6tRFvV)

https://i.ibb.co/TKmRLfX/page431.png (https://ibb.co/0BhZt7N)

[FranceBB]

There's already 3 plugins that support topleft (type 2) chroma placement :p so you can already do that. But seeing how the convert functions already support "Rec2020" and "PC.2020" matrix I think it would be logical to also support the corresponding chroma placement.

Ah... Would you be kind enough to copy-paste the function call with the parameters for that so I can grab it and toss FFMpeg? xD

[Reel.Deel]

Ah... Would you be kind enough to copy-paste the function call with the parameters for that so I can grab it and toss FFMpeg? xD

Cross that out about there being 3 plugins, I thought ResampleMT and fmtconv mentioned chromaloc type 2 but they don't. avsresize can change chroma placement, for example mpeg2 to topleft: chromaloc_op="mpeg2=>top_left"

But if pinter is right that DV is top left that means AviSynth, Dither, and fmtconv can do it also.

[pinterf]

https://en.wikipedia.org/wiki/Chroma_subsampling
see 4:2:0 section:
"In 4:2:0 DV ... also called top-left."

[StvG]

https://en.wikipedia.org/wiki/Chroma_subsampling
see 4:2:0 section:
"In 4:2:0 DV ... also called top-left."

4:2:0 DV implies interlaced content? ( here for example (https://www.mir.com/DMG/chroma.html) )

Currently fmtconv doesn't have cplace top-left (progressive) ( https://forum.doom9.org/showthread.php?p=1950533#post1950533 )

[Dogway]

Is this illegal call?
Expr("f32 x 0.5 +","",scale_inputs="intf")

I want to scale integer inputs to float so I skip a bunch of manual conversions.

[Reel.Deel]

https://en.wikipedia.org/wiki/Chroma_subsampling
see 4:2:0 section:
"In 4:2:0 DV ... also called top-left."

Im not sure about that. I did some test and avs' and fmtc_resample's output when cplace = dv do not match top_left from avsresize. Even avs' and fmtc_resample's dv placement differs.

Here's the script I used to check that:

Blankclip(width=288, height=48, pixel_type="RGBP8", color=$FF0000) #red
AddBorders(0,0,0,48,$00FF00) # green
AddBorders(0,0,0,48,$0000FF) # blue
AddBorders(0,0,0,48,$00FFFF) # cyan
AddBorders(0,0,0,48,$FF00FF) # magenta
AddBorders(0,0,0,48,$FFFF00) # yellow
StackHorizontal(last, last.Turnleft())

z_ConvertFormat(pixel_type="YUV444P8", colorspace_op="rgb:709:709:full=>709:709:709:limited")

avs = ConvertToYUV420(ChromaOutPlacement="dv", chromaresample="spline36")
avsr = z_ConvertFormat(pixel_type="YUV420P8", chromaloc_op = "center=>top_left", resample_filter_uv="spline36")
fmtc = fmtc_resample(css="420", cplaced="dv", kernel="spline36")
avst = Convert444to420Test(cplace="topleft")

diff(avsr, avst)


# Courtesy of pinterf :)
Function Diff(clip src1, clip src2)
{
return Subtract(src1.ConvertBits(8),src2.ConvertBits(8)).Levels(120, 1, 255-120, 0, 255, coring=false)
}

# little test fuction to see what the actual filters are doing
function Convert444to420Test(clip input, string "cplace")
{
default(cplace, "mpeg2")
Assert(cplace == "mpeg1" || cplace == "mpeg2" || cplace == "topleft", "Only mpeg1, mpeg2, topleft allowed")
Assert(Is444(input), "Only YUV444 allowed")

src_left = (cplace == "mpeg1") ? 0.0 : -0.50
src_top = (cplace == "topleft") ? -0.5 : 0.0
u = ExtractU(input).z_Spline36Resize(input.width/2, input.height/2, src_left, src_top)
v = ExtractV(input).z_Spline36Resize(input.width/2, input.height/2, src_left, src_top)


CombinePlanes(input, U, V, planes="YUV", source_planes="YYY", pixel_type="YUV420P8")
}

I even made a test function to see the offsets that are used when the chroma is scaled. To go from YUV444->YUV420:

topleft = src_left=-0.50, src_top=-0.50
mpeg2 = src_left=-0.50, src_top=0.0
mpeg1 = src_left=0.00, src_top=0.0


By the way, it would be nice if CombinePlanes accepted "YUV"/"RGB" as pixel_type and the choose the format based on bitdepth and chroma sampling.

[pinterf]

Shift for U and V shift is not the same for DV. I'm gonna make a test build.

[pinterf]

New build. ChromaInPlacement and ChromaOutPlacement allows "top_left" (and "jpeg" or "center" like "mpeg1"; "left" like "mpeg2")
Avisynth+ 3.7.1 test build 20 (20210930) (https://drive.google.com/uc?export=download&id=1b-sTu_IsnxalIp5QNI1ugW1DYcGUhl05)
Pls. test for interlaced as well.

[pinterf]

I do hope that I'm gonna put frameprop handling into avisynth before winter comes.

[FranceBB]

New build. ChromaInPlacement and ChromaOutPlacement allows "top_left" (and "jpeg" or "center" like "mpeg1"; "left" like "mpeg2")
Avisynth+ 3.7.1 test build 20 (20210930) (https://drive.google.com/uc?export=download&id=1b-sTu_IsnxalIp5QNI1ugW1DYcGUhl05)
Pls. test for interlaced as well.

Wow thanks! :D

I do hope that I'm gonna put frameprop handling into avisynth before winter comes.

https://i.imgur.com/PHFeAs9.png

TL;DR for those who are not familiar with the show, in Game of Thrones there's a scene in which they say "Winter is coming" but this meme swaps it with "Frame Properties are coming" as they will come before Winter...

[pinterf]

Thanks for the explanation :) I'm not watching TV/series/films other than a few actual trail running or MTB videos on Youtube. Or go to a cinema twice a year.

[FranceBB]

Thanks for the explanation :) I'm not watching TV/series/films

No worries. It doesn't matter if you're not familiar with tv series, you're very much familiar with ASM x86 and that's much much better xD

[Reel.Deel]

New build. ChromaInPlacement and ChromaOutPlacement allows "top_left" (and "jpeg" or "center" like "mpeg1"; "left" like "mpeg2")
Avisynth+ 3.7.1 test build 20 (20210930) (https://drive.google.com/uc?export=download&id=1b-sTu_IsnxalIp5QNI1ugW1DYcGUhl05)
Pls. test for interlaced as well.

Thanks pinterf! So far so good, I tested YUV420 "top_left" to and from "mpeg1"/"mpeg2" YUV420/YUV422/YUV444 (when applicable) and the input matches that of avsresize. There are some very minor differences only visible with the diff function and undetectable without it. I've yet to test interlaced.

[FranceBB]

Ok, so, now we have:

____________________________________________________________
#From normal 4:2:0 to 4:2:0 Type 2

z_ConvertFormat(chromaloc_op="mpeg2=>top_left")

____________________________________________________________
#From normal 4:2:0 8bit planar to 4:2:0 Type 2 8bit planar (Internal)

Converttoyv12(matrix="Rec2020", ChromaInPlacement="MPEG2", ChromaOutPlacement="top_left")

____________________________________________________________
#From normal 4:2:0 to 4:2:0 Type 2 (Internal)

ConverttoYUV420(matrix="Rec2020", ChromaInPlacement="MPEG2", ChromaOutPlacement="top_left")


So far so good, Ferenc! :D

https://i.imgur.com/KsBFusX.png

[cretindesalpes]

fmtc_resample also has top_left chroma location now on the Github repository (https://github.com/EleonoreMizo/fmtconv/commit/458cebf08e8bbf0839e70c381135d8014abcff03); it will be included in the next release.

[DTL]

"ConverttoYUV420(matrix="Rec2020", ChromaInPlacement="MPEG2", ChromaOutPlacement="top_left")"

Tried to make tests for possible degradation on sharp colour transients between sub-sampled and full-band formats (4:2:2 and 4:4:4) -
results https://i2.imageban.ru/out/2021/10/02/20913dde5e930ae7554c6cb63221c1a8.png
It looks 'default=bicubic' cause continuous colour sharpness degradation. The most theoretically-perfect 'sinc' do not work best with 'too sharp to ring' transients (and looks like return ringing) + inherits kernel-edge issue that cause 'shadow' at about 'taps' distance from transient.
The most perfect in 8 generations of 4:2:2 -> 4:4:4 ->4:2:2 currently look 'lanczos4' (of the currently 3 tested kernels - default, sinc, lanczos4). May be make it default for typical use ?

Also the ConvertToYUV422() do not have ChromaOutPlacement param - it mean it always use 'left' i.e. odd-Y samples co-siting in output ?

Addition: With new samples-defining to clip method (post https://forum.doom9.org/showthread.php?p=1953693#post1953693 ) it looks 'lanczos4' also not very good. So the best multi-generation method of sub-sampled format to 4:4:4 and back still need to be discovered.

[Reel.Deel]

Also the ConvertToYUV422() do not have ChromaOutPlacement param - it mean it always use 'left' i.e. odd-Y samples co-siting in output ?


Isn't YUV 4:2:2 always left aligned (mpeg2 chroma placement)?

[FranceBB]

Isn't YUV 4:2:2 always left aligned (mpeg2 chroma placement)?

It is afaik.
Every time I've received a 4:2:2 HDR10 masterfile from companies it was always with the standard MPEG-2 chroma placement, so I think Type 2 is only for 4:2:0, but I might be wrong.

[StainlessS]

See Here:- https://forum.doom9.org/showthread.php?p=1953742#post1953742



EDIT:
Several of above funcs [eg UPlaneMax "c[threshold]fi"] indicate a non optional final arg i of type int,
strictly speaking, I think all parameters following any optional parameter, MUST be also optional (I think they should maybe also be named optionals),
this is true in many if not all languages with optional arguments.
Some similar functions in AVS+ had similarly erroneous non optionals [now fixed I think].

EDIT: Actually, the same AVS+ builtin functions also have non optionals following an optional one, ie [below wrong in both Grunt and AVS+]

UPlaneMax "c[threshold]fi"
UPlaneMin "c[threshold]fi"
UPlaneMinMaxDifference "c[threshold]fi"
VPlaneMax "c[threshold]fi"
VPlaneMin "c[threshold]fi"
VPlaneMinMaxDifference "c[threshold]fi"
YPlaneMax "c[threshold]fi"
YPlaneMin "c[threshold]fi"
YPlaneMinMaxDifference "c[threshold]fi"


EDIT:

/* BAD trailing compulsory arg i, following an optional arg. [BAD in both AVS+ and Grunt]

UPlaneMin "c[threshold]fi"
UPlaneMinMaxDifference "c[threshold]fi"
VPlaneMax "c[threshold]fi"
VPlaneMin "c[threshold]fi"
VPlaneMinMaxDifference "c[threshold]fi"
YPlaneMax "c[threshold]fi"
YPlaneMin "c[threshold]fi"
YPlaneMinMaxDifference "c[threshold]fi"ing compulsory arg
*/

#YPlaneMin "c[threshold]fi"
Blankclip(pixel_type="YV12")
SSS="""
# y=YPlaneMin(0.4,1) # OK, Threshold = 0.4, Offset = 1 ie current_frame+1
y=YPlaneMin(threshold=0.4,1) # Script Error: The named argument "threshold" was passed more than once to YPlaneMin
# y=YPlaneMin(Last,threshold=0.4,1) # Script Error: The named argument "threshold" was passed more than once to YPlaneMin # EDIT: ADDED
RT_Subtitle("%d] Y=%d",current_frame,Y)
return last
"""
ScriptClip(SSS)


EDIT:
(I think they should maybe also be named optionals)
think above error message "# Script Error: The named argument "threshold" was passed more than once to YPlaneMin"
is down to requiring a named optional after using an optional name, but parameter 'i' dont have a name so cannot use 'i' arg together with Threshold=x.

EDIT: I'll [EDIT: try] knock up a parser to check all builtin parameter lists for un-named optionals or compulsary parameters after an optional parameter.

EDIT: From here:- http://avisynth.nl/index.php/User_defined_script_functions
Facts about user defined script functions

Functions can take up to sixty arguments and the return value can be of any type supported by the scripting language (clip, int, float, bool, string, AVS+array).

Although not recommended practice, an argument type may be omitted, and will default to val, the generic type.

If the function expects a video clip as its first argument, and that argument is not supplied, then the clip in the special Last variable will be used.

Functions support named arguments. Simply enclose an argument's name inside double quotes to make it a named argument. Note that after doing so the following apply:
All subsequent arguments in the argument list must be named also.
A named argument is an optional argument, that is, it need not be supplied by the caller.
One presumes that the rules for script functions also apply to builtin/plugin,
so after 1st named argument, all following others must also be named,
and when calling a function [script or plugin], after providing an optional name, so all following args must also be called with names.
[there seem to be a few rule breakers]

[DTL]

It looks like documentation of ConvertBits() is outdated: https://forum.doom9.org/showthread.php?p=1953899#post1953899

http://avisynth.nl/index.php/ConvertBits
says
bool fulls = (auto)
Use the default value unless you know what you are doing.
If true (RGB default), scale by multiplication: 0-255 → 0-65535;
if false (YUV default), scale by bit-shifting.
Use case: override greyscale conversion to fullscale instead of bit-shifts.
Conversion from and to float is always full-scale.
Alpha plane is always treated as full scale.
bool fulld = fulls
Use the default value unless you know what you are doing.
At the moment, must match fulls.

Nowdays (with last pinterf build) fulld may be not equal to fulls and fulld for float input may be false.
So correct conversion from float32 output of fmtc plugin to 8bit limited require ConvertBits(8,fulls=true,fulld=false) that works.

[StainlessS]

Thanks for pointing that out DTL. :)

[FranceBB]

Well, not that anyone cares or uses YUY2 anymore, but...

ConverttoYUY2(matrix="Rec2020")

and

ConverttoYUY2(matrix="PC.2020")

don't work, however since those have been added for Converttoyv12, Converttoyv16, Converttoyv24, ConverttoYV411, ConverttoY8, ConverttoRGB32, ConverttoRGB24, ConverttoRGB, ConverttoYUV420, ConverttoYUV422, ConverttoYUV444, I feel like it was worth reporting...

Avisynth 3.7.1 Test 20 x64 CUDA

[Takoh]

Apparently it doesn't work.

gives me while doesn't crash, but process doing nothing - it simply waiting for something.

Can someone help me. I finally built AviSynthCUDAFilters and masktools-cuda, but I'm having the same problem as DKATOM I think. When I run the script my GPU just sits at 100% using about 800mb of vram. And virtualdub has a blank box.

These are the files I built.

masktools2.dll (cuda version)
KTGMC.dll
AvsCUDA.dll
KFM.dll
KDebugTool.dll
GRunT.dll
KUtil.dll

(I renamed Kmasktools to masktools2 after it was built)
I'm using the avisynth cuda build that was posted in this thread avisynth 3.7.1 test4
My script looks something like this....
SetMemoryMax(8000, type=DEV_TYPE_CUDA)

srcfile="MyCLIP.avi"
AviSource(srcfile).converttoYV12()
OnCPU(2).KTGMC(preset="Fast").OnCUDA(2)
#OnCPU(2).KTGMC(SourceMatch=3, Lossless=2, tr0=1, tr1=1, tr2=1).OnCUDA(2)


I also tried this, with the same result.
AviSource(srcfile)
onCPU(2)
KTGMC()
onCUDA(2)

When I remove onCUDA, at the bottom of Virtualdub there's the error

"Avisynth read error: [KMasktoolsFilterBase] CUDA"... and some characters in another language.


I tried removing onCPU and Virtualdub gave the error:

Device unmatch: KTGMC_Bob[CUDA] does not support [CPU] frame


I tried using an older GPU and it doesn't hang at 100% usage. Instead VirtualDub gives an error at the bottom:

Avisynth Read Error: [CUDA Error] 400: invalid resource handle @948

Any help would be GREATLY appreciated. I'm not sure what to try next :')

[Dogway]

Just a friendly reminder for my question a week ago here (https://forum.doom9.org/showthread.php?p=1953427#post1953427). Some discussion here (https://forum.doom9.org/showthread.php?p=1953668#post1953668)too.
Currently it's not possible to evaluate an expression as float which is necessary for a range of operators like pow, exp, cos, sin, etc unless we do a bunch of "range_max /" and "range_max *".

I tried with f32 decorator and scale_inputs="intf" but I think it's not supported.

[pinterf]

Just a friendly reminder for my question a week ago here (https://forum.doom9.org/showthread.php?p=1953427#post1953427). Some discussion here (https://forum.doom9.org/showthread.php?p=1953668#post1953668)too.
Currently it's not possible to evaluate an expression as float which is necessary for a range of operators like pow, exp, cos, sin, etc unless we do a bunch of "range_max /" and "range_max *".

I tried with f32 decorator and scale_inputs="intf" but I think it's not supported.
Yes, it is not handled. I could not find out in a few minutes why I put there this limitation. Anyway, you can pass a ConvertBits(32) clip there and a back conversion after.

[Dogway]

Yes, it is not handled. I could not find out in a few minutes why I put there this limitation. Anyway, you can pass a ConvertBits(32) clip there and a back conversion after.

Yes I know, it was tested and performance dropped (https://forum.doom9.org/showthread.php?p=1953701#post1953701)to less than half. Anyway thanks when you got time.

[DTL]

"ConverttoYUY2(matrix="PC.2020")"

Documentation for Convert at http://avisynth.nl/index.php/Convert looks also outdated and not list PC.2020 option at all.

[Takoh]

Apparently it doesn't work.

gives me while doesn't crash, but process doing nothing - it simply waiting for something.

Can someone help me. I finally built AviSynthCUDAFilters and masktools-cuda, but I'm having the same problem as DKATOM I think. When I run the script my GPU just sits at 100% using about 800mb of vram. And virtualdub has a blank box.

These are the files I built.

masktools2.dll (cuda version)
KTGMC.dll
AvsCUDA.dll
KFM.dll
KDebugTool.dll
GRunT.dll
KUtil.dll

(I renamed Kmasktools to masktools2 after it was built)
I'm using the avisynth cuda build that was posted in this thread avisynth 3.7.1 test4
My script looks something like this....
SetMemoryMax(8000, type=DEV_TYPE_CUDA)

srcfile="MyCLIP.avi"
AviSource(srcfile).converttoYV12()
OnCPU(2).KTGMC(preset="Fast").OnCUDA(2)
#OnCPU(2).KTGMC(SourceMatch=3, Lossless=2, tr0=1, tr1=1, tr2=1).OnCUDA(2)


I also tried this, with the same result.
AviSource(srcfile)
onCPU(2)
KTGMC()
onCUDA(2)

When I remove onCUDA, at the bottom of Virtualdub there's the error

"Avisynth read error: [KMasktoolsFilterBase] CUDA"... and some characters in another language.


I tried removing onCPU and Virtualdub gave the error:

Device unmatch: KTGMC_Bob[CUDA] does not support [CPU] frame


I tried using an older GPU and it doesn't hang at 100% usage. Instead VirtualDub gives an error at the bottom:

Avisynth Read Error: [CUDA Error] 400: invalid resource handle @948

Any help would be GREATLY appreciated. I'm not sure what to try next :')

[pinterf]

"ConverttoYUY2(matrix="PC.2020")"

Documentation for Convert at http://avisynth.nl/index.php/Convert looks also outdated and not list PC.2020 option at all.
Packed RGB<->YUY2 conversion is a totally different code part, has tricky accelerations with differently scaled internal constants other than the generic planar approach. (There is no intermediate 4:4:4 conversion like e.g. for YV16<->RGB so it is much quicker). I'd better not touch it without a proper code refactor. E.g. conversion matrix calculation code and constants are reinvented at least three or four different places in the source. Now that I've seen it again I started making it look better.

[pinterf]

Yes I know, it was tested and performance dropped (https://forum.doom9.org/showthread.php?p=1953701#post1953701)to less than half. Anyway thanks when you got time.

All these parameter were introduced at the dawn of 10+ bits introduction because there had been a zillion of scripts that were using only 8 bit constants. So our aim was to make non-8 bit clips somehow to be handled with existing expression strings. Till I return to the topic in the future - if it is still hot - please use the existing infrastructure.

[pinterf]

Can someone help me. I finally built AviSynthCUDAFilters and masktools-cuda, but I'm having the same problem as DKATOM I think. When I run the script my GPU just sits at 100% using about 800mb of vram. And virtualdub has a blank box.

These are the files I built.

(I renamed Kmasktools to masktools2 after it was built)
I'm using the avisynth cuda build that was posted in this thread avisynth 3.7.1 test4
My script looks something like this....


I also tried this, with the same result.


When I remove onCUDA, at the bottom of Virtualdub there's the error


I tried removing onCPU and Virtualdub gave the error:


I tried using an older GPU and it doesn't hang at 100% usage. Instead VirtualDub gives an error at the bottom:

Any help would be GREATLY appreciated. I'm not sure what to try next :')
Unfortunately I'm not able to help you with it.
Back in January I was able to run this script:
Avisource("MyHi8.avi").killaudio().assumefps(25,1)
AssumeBFF()
SetMemoryMax(2048, type=DEV_TYPE_CUDA)
OnCPU(2).KTGMC(SourceMatch=3, Lossless=2, tr0=1, tr1=1, tr2=1).OnCUDA(2)

I don't remember but my GTX460 was too old (GeForce drivers stopped supporting it, while CUDA SDK had a mininum requested version against the driver), then I bought a 1030 and it worked then.

Also these filters are usually limited to 8 bit YV12 and a limited set of plugin parameters.

[Dogway]

I'm aware of the situations that's why I started my modernization efforts, to bring some sanity. everything in avisynth feels patches over patches, non descriptive filter names, slow scripts or plugins, etc
I'm going to focus back on transformspack soon to deliver serious color work to avisynth and expressions there typically work on float so I wanted to avoid to fill the conversions with:
bif = bi > 16 ? "" : "range_max /"
bii = bi > 16 ? "" : "range_max *"

expr("x "+bif+" y "+bif+" "+atan2t()+" "+bii+"", "")

[videoh]

I started my modernization efforts, to bring some sanity. everything in avisynth feels patches over patches, non descriptive filter names, slow scripts or plugins, etc The Saviour has arrived. Hallellujah!

/sarc

[Dogway]

The Saviour has arrived. Hallellujah!

/sarc

Now I understand why your high post count, all talk no work. Go do something productive.

[videoh]

Script kiddie didn't do his homework.

[Dogway]

Script kiddie didn't do his homework.

Oh, the dinosaur sticking out the leg. You might be those that dislike python, julia, avisynth/vs wonderful community scripts... please some moderator, warn this guy or at least delete all his babbling posts.

[Takoh]

Unfortunately I'm not able to help you with it.

I don't remember but my GTX460 was too old (GeForce drivers stopped supporting it, while CUDA SDK had a mininum requested version against the driver), then I bought a 1030 and it worked then.

Also these filters are usually limited to 8 bit YV12 and a limited set of plugin parameters.

Alright thank you pinterf :) I'll keep trying. Going to try and build avisynth myself next. And after that I might try a different set of nvidea drivers.
Edit: Also I noticed I forgot to build KNNEDI3. It's giving an error that it can't find "..\..\Common.props" so hopefully I can get it built.
And I installed CUDA Dev 11.4 but my drivers are using 11.2 so that could be an issue. I'll try uninstalling it...
Oh and I didn't notice an error that I don't have Windows SDK 8.1 for some things ^_^; lol . I'll try and get it... (I'd rather find these mistakes than have no mistakes, so I still have hope lol.)

Edit2: Oh ok, i guess KNNEDI3 wants to be under cudafilters

[Takoh]

Hello! I finally got a script to work! :D
Not KTGMC, but still a small win, so I'm really happy!
I used MagicYUV for encoding.

1)This gave about 170fps when saving. Increasing onCUDA(x) beyond 2 didn't seem to increase the speed. (Also, SetMemoryMax type=DEV_TYPE_CUDA didn't effect the speed.) AviSource(srcfile)
onCPU()
AvsCUDA_Spline64Resize (1280,720)
onCUDA(2)

2) This gave about 250fps and was pretty consistent. AviSource(srcfile)
onCPU(43)
AvsCUDA_Spline64Resize (1280,720)
onCUDA(2)

3) This gave about 17fpsAviSource(srcfile)
onCPU(43)
Spline64Resize (1280,720)
onCPU(43)

4) This gave around 90-150fps but it bounced around a lot.AviSource(srcfile)
onCPU(43)
Spline64Resize (1280,720)
Prefetch(43)
Anyway! I just wanted to share. I'm really excited that I finally got something working lol :p
Thank you pinterf for porting it :)

EDIT: KTemporalNR , KDeband and KEdgeLevel work.
https://github-com.translate.goog/nekopanda/AviSynthCUDAFilters/wiki/Post-Processing-Filters?_x_tr_sl=ja&_x_tr_tl=en&_x_tr_hl=en-GB&_x_tr_pto=nui,sc

[videoh]

Good stuff, Takoh.

[pinterf]

Hello! I finally got a script to work! :D
Not KTGMC, but still a small win, so I'm really happy!

Nice!
Your Prefetch and OnCPU numbers are a bit too much I think.
Translated doc from the original:
http://avisynth.nl/index.php/SetFilterMTMode#OnCPU

edit:
The CUDA environment which Nekopanda created was introduced because each consecutive filter can run on GPU, there is no copy-in-out overhead.
The whole KTGMC basically runs without coming back to CPU processing. Source filter - CUDA processing - Back to real file (CPU).

Keep experimenting and write your adventures.

[tormento]

#bump: I could kill to have CudaMDegrain. :)

[magnetite]

There is a KSMDegrain (https://github-com.translate.goog/nekopanda/AviSynthCUDAFilters/blob/master/TestScripts/KSMDegrain.avsi?_x_tr_sl=ja&_x_tr_tl=en&_x_tr_hl=en-GB&_x_tr_pto=nui,sc) script under the TestScripts folder.

[Takoh]

Nice!
Your Prefetch and OnCPU numbers are a bit too much I think.
Translated doc from the original:
http://avisynth.nl/index.php/SetFilterMTMode#OnCPU


Ah ok I'll give it a try with 2.
edit3: ya, onCPU(2) gives the same performance. Thanks :)


edit:
The CUDA environment which Nekopanda created was introduced because each consecutive filter can run on GPU, there is no copy-in-out overhead.
The whole KTGMC basically runs without coming back to CPU processing. Source filter - CUDA processing - Back to real file (CPU).

Keep experimenting and write your adventures.


Ahhh I wish i could get it to work. If anyone wants they can upload their built KTGMC dll etc for me to try in case mine is wonky.

I'm not sure if i tested KSMDegrain. I'll give it a try.

Edit: Unfortunately KSMDegrain is giving that same error. "Avisynth read error: [CUDA Error] 400: invalid resource handle @948"
It looks like it uses KTGMC. I'm going to double check to see if i'm missing any plugins that KTGMC.avsi mentions are required. (I bet I am lol). Although it mentions QTGMC under getting started so not sure if the plugins listed are correct for KTGMC.

Edit2: I updated/added all of the plugins mentioned but still get the same error message ;(

When I built the cuda filters solution, a few things were asking for CUDA 8.0, so I installed it. Not sure if I was supposed to update something there to use Cuda 11.2

-------------------------------------------------------------------------------------
I tried using this:
"D3DVP
Direct3D 11 Video Processing Avisynth / AviUtl Filter
A de-interlacing filter that uses the Direct3D 11 Video API. If the driver of your GPU is properly implemented, you can de-interlace with the GPU."
https://github-com.translate.goog/nekopanda/D3DVP?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en-GB&_x_tr_pto=nui

But I get an error: CAVIStreamSynth: System exception - Access Violation... :(
He says, "Avisynth 2.6 or later? (Unknown because I have only tried it with AvisynthPlus CUDA)" so I guess it only works on his version. I've got vs2015 runtime installed as well.

[pinterf]

CUDA 8.0 rererences must be replaced to the actual SDK.afaik this had to be done before I was able to compile. Nvidia's 8.0 sdk and perhaps even 9 + Visual Studio 2019 is not supported combination.

[tormento]

There is a KSMDegrain
I think the dlls it relies upon are obsolete, to say the best.

[Takoh]

CUDA 8.0 rererences must be replaced to the actual SDK.afaik this had to be done before I was able to compile. Nvidia's 8.0 sdk and perhaps even 9 + Visual Studio 2019 is not supported combination.

OK thanks. :) I'll try and see if i can do that. I'm not very experienced with that sort of thing :D

[Dogway]

Is there such thing as clip properties as opposed to frame properties? I would like to set some properties at clip level.

[tormento]

New build.
I have incurred in the same error as described here (https://forum.doom9.org/showthread.php?p=1882153#post1882153) when enabling prefetch.

SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_48 Fantozzi\fantozzi.dgi",ct=24,cb=24,cl=0,cr=0)
ConvertBits(16)
SMDegrain (tr=6, thSAD=600, refinemotion=true, contrasharp=false, PreFilter=4, plane=4, chroma=true)
ConvertToRGB32()
AWB_Func(matrix="Rec709")
ConvertToYUV420()
fmtc_bitdepth (bits=8,dmode=8)
Prefetch(6)

Can you help me or fix it?

P.S: ConvertToYUV420() gives me a 8 bit clip. I can imagine it's because of RGB32. Perhaps it's better to move fmtc_bitdepth before ConvertToRGB32()?

[real.finder]

Is there such thing as clip properties as opposed to frame properties? I would like to set some properties at clip level.

there are old http://avisynth.nl/index.php/Clip_properties

but I think frame properties Functions (http://avisynth.nl/index.php/Internal_functions#Functions_for_frame_properties) can be updated to act like getparity() (that mean adding int "frame" to all them so they can be used outside runtime filters) similar as vs cheating (https://forum.doom9.org/showthread.php?p=1954997#post1954997)

[FranceBB]

Perhaps it's better to move fmtc_bitdepth before ConvertToRGB32()?

Definitely, you're already getting 8bit so having fmtc_bitdepth there is useless. Put it before RGB32 so that it dithers ;)

[Dogway]

@real.finder: Yes Clip properties are read only. I got the vs cheating, good one, not sure I understood the getparity() behaviour.

Also I have experienced some slowdowns in ex_edge() when replaced the inverse convolution (a few sums ands substractions) with an 'abs'. Is 'abs' optimized in Expr()? I found this question in StackOverflow (https://stackoverflow.com/questions/664852/which-is-the-fastest-way-to-get-the-absolute-value-of-a-number) which shows a few ways to avoid ternaries, one of the fastest is to use assembly or Intel intrinsics.

[wonkey_monkey]

Exrp's abs is a single fully vectorised instruction. It should be much faster than a few adds and subtracts.

[jpsdr]

And if i'm not wrong, on floating point abs is just setting the sign bit to 0, so, you can't do faster...

[Dogway]

Benchmarks for hprewitt:

425 fps Prefetch(4) 368 fps Prefetch(6)
Expr("x[1,1] A^ x[1,0] 2 * B^ x[1,-1] C^ x[-1,1] D^ x[-1,0] 2 * E^ x[-1,-1] F^ x[0,1] 2 * G^ x[0,-1] 2 * H^ " \
+"A B C + + D - E - F - abs F H C + + D - G - A - abs max","")
395 fps Prefetch(4) 376fps Prefetch(6)
Expr("x[1,1] A^ x[1,0] 2 * B^ x[1,-1] C^ x[-1,1] D^ x[-1,0] 2 * E^ x[-1,-1] F^ x[0,1] 2 * G^ x[0,-1] 2 * H^ " \
+"A B C + + D - E - F - A G D + + C - H - F - max F E D + + C - B - A - F H C + + D - G - A - max max","")


EDIT:Also I found a few (many) inconsistencies with the ConvertTo functions. You might want to have a look at them maybe.

1. All arguments names must be declared except for the first one (ie. "ConvertToXXX(mat,interlaced=false,ChromaInPlacement=cplacea,chromaresample=krn,ChromaOutPlacement=cplace)"
2. Interlaced and matrix arguments are swapped for the ConvertToYUVXXX family compared to ConvertToYV411, ConvertToPlanarRGB and ConvertToRGBXX.
3. ConvertToPlanarRGB, ConvertToRGBXX and ConvertToYUVXXX don't accept ChromaInPlacement argument for other than 420 inputs.
4. ConvertToYUVXXX doesn't accept ChromaOutPlacement argument for other than 420 outputs (in other words ConvertToYUV422, etc don't have ChromaOutPlacement argument)

[FranceBB]

ConvertToYUVXXX don't accept ChromaInPlacement argument for other than 420 inputs.

4. ConvertToYUVXXX doesn't accept ChromaOutPlacement argument for other than 420 outputs (in other words ConvertToYUV422, etc don't have ChromaOutPlacement argument)

Well, I get your point, but I think this was done because technically there's nothing like top_left alignment for anything other than 4:2:0. I mean, you can't have 4:2:2 Type 2 and 4:4:4 Type 2 as they're not standard. On this sentence "they're not standard" I would actually like to clarify myself before drawing any conclusion: I don't know if they're really not described as a standard, however I couldn't find the description of them anywhere and, as far as the delivery from third party companies is concerned, whenever I receive stuff like HDR movies in Apple ProRes or Motion JPEG2000 IMF etc in 4:2:2 or 4:4:4, they're always always always with the old classic MPEG-2 chroma placement.

Now we could ask ourselves a question: shall we introduce support for Chroma Placement for everything even if it's not formally described for the sake of having everything nice and dandy (i.e consistent)?

I don't know the answer, honestly, but I'd like to see what others think about this and where we should go.

[Dogway]

Wouldn't that be the use case for the Default() function?
We daily deal with non-standard messy clips to try to fix them, so if you know what you are doing you should be able to use a chroma placement conversion instead of a workaround.

Not that I care too much since the ConvertTo functions are being superseded by fmtconv, avsresize or my own solutions, just trying to help here.

[Dogway]

I think I found a new bug in Expr variables, possibly since the last variable bugfix in Expr() (test 11 I think).
Expr(last,"x[-1,1] A@ x[0,1] B@ x[1,1] C@ x[-1,0] D@ x[1,0] E@ x[-1,-1] F@ x[0,-1] G@ x[1,-1] H@ + + + + + + + 0.125 * W^
A B dup1 dup1 min swap2 max
C D dup1 dup1 min swap2 max
E F dup1 dup1 min swap2 max
G H dup1 dup1 min swap2 max
swap2 swap1 swap6 dup1 dup1 min swap2 max
swap2 swap1 swap4 dup1 dup1 min swap2 max
swap3 swap1 swap7 dup1 dup1 min swap2 max
swap6 swap1 swap5 dup1 dup1 min swap2 max
swap3 swap1 swap2 dup1 dup1 min swap2 max
swap3 swap1 swap6 dup1 dup1 min swap2 max
swap7 swap1 swap4 dup1 dup1 min swap2 max
swap2 swap1 swap5 dup1 dup1 min swap2 max
swap2 swap1 swap7 min
swap4 swap1 swap3 max
swap3 swap1 swap4 min
swap2 max
dup1 dup1 min swap2 max
E^ D^ H^ 50 <= x D E clip x ?","")

It has to do with remaining elements in stack and ternaries. If I explicitly assign a var to the last element it works as in:
E^ D^ H^ A^ A 50 <= x D E clip x ?","")

[FranceBB]

Wouldn't that be the use case for the Default() function?
We daily deal with non-standard messy clips to try to fix them, so if you know what you are doing you should be able to use a chroma placement conversion instead of a workaround.


Well fair enough, as long as Ferenc is willing to add them and no one has an issue with them being non standard, I don't see a problem with adding them. :)

[pinterf]

I think I found a new bug in Expr variables, possibly since the last variable bugfix in Expr() (test 11 I think).
Expr(last,"x[-1,1] A@ x[0,1] B@ x[1,1] C@ x[-1,0] D@ x[1,0] E@ x[-1,-1] F@ x[0,-1] G@ x[1,-1] H@ + + + + + + + 0.125 * W^
A B dup1 dup1 min swap2 max
C D dup1 dup1 min swap2 max
E F dup1 dup1 min swap2 max
G H dup1 dup1 min swap2 max
swap2 swap1 swap6 dup1 dup1 min swap2 max
swap2 swap1 swap4 dup1 dup1 min swap2 max
swap3 swap1 swap7 dup1 dup1 min swap2 max
swap6 swap1 swap5 dup1 dup1 min swap2 max
swap3 swap1 swap2 dup1 dup1 min swap2 max
swap3 swap1 swap6 dup1 dup1 min swap2 max
swap7 swap1 swap4 dup1 dup1 min swap2 max
swap2 swap1 swap5 dup1 dup1 min swap2 max
swap2 swap1 swap7 min
swap4 swap1 swap3 max
swap3 swap1 swap4 min
swap2 max
dup1 dup1 min swap2 max
E^ D^ H^ 50 <= x D E clip x ?","")

It has to do with remaining elements in stack and ternaries. If I explicitly assign a var to the last element it works as in:
E^ D^ H^ A^ A 50 <= x D E clip x ?","")
Eeer, this example is a little bit too long, I'm lost, what is the real problem? Should it give error and it does not, or works not as expected? I can see a W^ which is not used later but for the first sight this is all I can see. I don't know what to expect.

[Dogway]

Yes sorry, I tried to narrow down the expression to the minimum, but less than that it worked.

I rechecked again, and went from all past test versions, 12, 17, 19 and 20, this time it worked in all of them included test 20 so it seems like an obscure memory bug included recently. Read here (https://forum.doom9.org/showthread.php?p=1955158#post1955158).

I will try to pinpoint since what version it happens, most likely something related to long stack elements like in my example.

[pinterf]

Yes sorry, I tried to narrow down the expression to the minimum, but less than that it worked.

I rechecked again, and went from all past test versions, 12, 17, 19 and 20, this time it worked in all of them included test 20 so it seems like an obscure memory bug included recently. Read here (https://forum.doom9.org/showthread.php?p=1955158#post1955158).

I will try to pinpoint since what version it happens, most likely something related to long stack elements like in my example.
Thanks, I've got the access viola, no more help needed atm.

[Dogway]

Boulder was getting it with test14, so I don't think newer commits did affect that. For the time being I will work around it with the explicit var.

[pinterf]

I think the crash reason (addressing of an array's minus 1st element) was there even in the previous versions but somehow they were hidden. Lately I was not able to reproduce the addressing problem with any release version, only with my debug build.
Internal optimization of ternary operator when 'swap' was used in the condition/truecase/falsecase expressions did not like each other. Still testing.

[Dogway]

wow, well you got further than I thought. In some occasions I was also getting some errors when using swap or swap2 as ternaries outputs. Don't remember where but could be related.

Also a question, is there a limit for float values that can't get stored? In Adaptive Sharpen there's a part in the edgemask that goes up to 200.0 float or so. I need to scale it down and then back to pass it to another expression.


EDIT: BTW YPlaneMin can't return negative values (for floats)?

[pinterf]

New build: Avisynth+ 3.7.1 test build 21 (20211021) (https://drive.google.com/uc?export=download&id=1eExZZoR17JCx8G0wfp49Vfha9awz-6I9)
- Allow propGetXXX property getter functions called as normal functions, outside runtime
By default frame property values are read from frame#0 which index can be overridden by the offset parameter

Example:
Colorbars()
PropSet(last, "hello", 1) # Set to 1 for all frames
# Override to 2 with runtime function except for frameNo=1
ScriptClip("""if(current_frame!=1) {propSet("hello",2)}""")
n0 = propGetInt("hello") # same as propGetInt("hello",offset=0)
# or get the frame property from the Nth frame
n1 = propGetInt("hello",offset=1)
n2 = propGetInt("hello",offset=2)
# n0 and n2 is 2 (overridden in runtime)
# n1 will be 1 (keeps global setting)
SubTitle("n0/n1/n2=" + "{n0}/{n1}/{n2}".Format)

- Add parameter string "ChromaOutPlacement" in ConvertToYV16 and ConvertToYUV422 similar to YV12/420 conversions
4:2:2 conversions now allow ChromaInPlacement and ChromaOutPlacement parameters
"left" ("mpeg2") and "center" ("mpeg1", "jpeg").
Note 1: "top_left" and "dv" is still valid only for 4:2:0
Note 2: "mpeg2" (sale as "left") was so far the default for 4:2:0 and 4:2:2 sources as well.
- Source code: use common YUV-RGB conversion matrix values and generation throughout the project
Was: constants and calculations and inline code here and there.
New: YUY2 RGB conversions now allow matrix "PC.2020" and "Rec2020" (as a side effect)
- 4:2:0 conversions: ChromaInPlacement and ChromaOutPlacement parameters: (see http://avisynth.nl/index.php/Convert)
add "top_left" (new)
add "center" and "jpeg" (as an alternative to "mpeg1"), "left" (as an alternative to "mpeg2")
plus fixing Dogway's Expr issue (ternary+swap or variable store).

[FranceBB]

New build: Avisynth+ 3.7.1 test build 21 (20211021) (https://drive.google.com/uc?export=download&id=1eExZZoR17JCx8G0wfp49Vfha9awz-6I9)
- Allow propGetXXX property getter functions called as normal functions, outside runtime


Thank you so much for this! :D

[kedautinh12]

Thank you so much for this! :D

Reel.Deel will complain about only thanks :D

[Reel.Deel]

Reel.Deel will complain about only thanks :D

I only complain about your thanks because there was some threads that only had less than 15 posts and 60% of the thread were from you just saying "thanks" ... That was before Wilbert deleted those useless post :) ... and yet here you are again just adding noise.


New build: Avisynth+ 3.7.1 test build 21 (20211021) (https://drive.google.com/uc?export=download&id=1eExZZoR17JCx8G0wfp49Vfha9awz-6I9)
Thank you pinterf, I know you're a gentleman and always update the wiki but I'll get around to it in the next couple of days. My overworked 2 weeks are finally coming to an end.

[pinterf]

Thank you pinterf, I know you're a gentleman and always update the wiki but I'll get around to it in the next couple of days. My overworked 2 weeks are finally coming to an end.
Good observation :), no wiki on test build changes. I have planned updating wiki; now these changes seem to be final, readme changelogs can be used to do the additions. If you deal with it I thank you.

OFF:
Running season is finished for me with a 108km 3200m+ trail running race two weeks ago, you can imagine that I'm uber relaxed since then :) So I've got a bit more time nowadays.

[StainlessS]

Dont know if Deputy Doggy's edit was spotted [no mention in posted changelog].


EDIT: BTW YPlaneMin can't return negative values (for floats)?

EDIT: Is it supposed to. [if so, what then is the expected min/max limit on YPlaneMin, YPlaneMax, YPlaneMinMaxDiff]

EDIT:

Blankclip(pixel_type="YV12")
ConvertBits(32)
#info
Levels(0.0,1.0,1.0,-2.0,2.0)
SSS="""
y=YPlaneMin
RT_Debugf("%d] Y=%f",current_frame,Y)
S=RT_string("%d] Y=%f",current_frame,Y)
Subtitle(s)
return last
"""
Scriptclip(SSS)


https://i.postimg.cc/D0kLzBfb/I39587872-00.jpg (https://postimages.org/)
16.0/255.0 = 0.062745

EDIT:
Coring=False shows 0.0

Blankclip(pixel_type="YV12")
ConvertBits(32)
#info
Levels(0.0,1.0,1.0,-2.0,2.0,coring=false)
SSS="""
y=YPlaneMin
RT_Debugf("%d] Y=%f",current_frame,Y)
S=RT_string("%d] Y=%f",current_frame,Y)
Subtitle(s)
return last
"""
Scriptclip(SSS)

https://i.postimg.cc/J00VGcTM/a-00.jpg (https://postimages.org/)

EDIT: Ok, saw your next post answer, thanx P.

[pinterf]

Dont know if Deputy Doggy's edit was spotted [no mention in posted changelog].
Spotted but the solution is postponed.

Unfortunately YPlaneMin has an optional 'threshold' parameter which is a percentage, stating how many percent of the pixels are allowed above or below minimum. (Wiki)

The calculation is done by building a histogram internally.

Unfortunately 32 bit float is not a well behaving discrete type like a 10 bit format with pixels from 0 to 1023, from which we can build histograms.

So 32 bit float formats are translated (digitized) to 16 bits internally (e.g. 0..65535 for luma) and histogram is created from this converted values. This is where the clamp occurs. Or else how could we establish easily for which pixel value exceed we the 5% percent.

A possible solution is to make a real min-max when threshold is exactly zero. Planned feature.

[kedautinh12]

I meet problem when use avs cuda ver in megui. When i was indexed video .webm with directshowsource cuda ver, i meet error: "invalid index plugin". I returned avs cpu ver and problem was gone

Video sample: https://drive.google.com/file/d/1xnIgefEeOd4_gbtO2Z45E_cNqYBdMkyM/view?usp=sharing

[pinterf]

Theoretically plugins are not affected and do not differ from normal x64 version other than normal is still xp-compatible, but avisynth dll which can accept cuda filters is not. Is it the complete error message?
EDIT:
My machine was not able to read .webm with DirectShowSource (unsupported)
Using FFMS2 the clip was opened properly.

[Dogway]

Thanks a lot for the update!

So 32 bit float formats are translated (digitized) to 16 bits internally (e.g. 0..65535 for luma)
Maybe this has to do with the issue I was having, is there a limited working dynamic range for 32-bit float? It could also be useful to know if we can work with absolute HDRI's.

Also I searched in the InternalFunctions, there's no such thing as propCopy() right? to copy properties between one clip and another. (sorry to keep you busy :( )

[pinterf]

The project is request-driven, no need to excuse :)

[pinterf]

New build
Avisynth+ 3.7.1 test build 22 (20211022) (https://drive.google.com/uc?export=download&id=1M36wJoTtk15o_D3ihkhEx7FHk0DoQMaS)
There was a fix in the new propGetxxx feature.
New things:
- New function: propCopy(clip, clip [,bool 'merge'])
Copies the frame properties of the second clip to the first.
Parameter 'merge' (default false):
when false: exact copy (original target properties will be lost)
when true: keeps original properties, appends all parameters from source but overwrite if a parameter with the same name already exists.
- xxxPlaneMin xxxPlaneMax, xxxPlaneMinMaxDifference:
- 32 bit float formats: when threshold is 0 then return real values instead of 0..1 (chroma -0.5..0.5) clamped histogram-based result
- for threshold 0 they also became a bit quicker for 8-16 bit formats (~10% on i7-7700)

[Dogway]

So fast, you are in a roll! Now I have to undo my workarounds (Get+Set) hehe

Do you if it's possible to allow Format() to read array items? It's something that bothers me a lot to have to write Format(" x "+string(val[n])+" +")

[kedautinh12]

Theoretically plugins are not affected and do not differ from normal x64 version other than normal is still xp-compatible, but avisynth dll which can accept cuda filters is not. Is it the complete error message?
EDIT:
My machine was not able to read .webm with DirectShowSource (unsupported)
Using FFMS2 the clip was opened properly.

yeah, my machine can read .webm with DirectShowSource in avs+ cpu ver (use both AviSynth.dll, DevIL.dll, DirectShowSource.dll cpu ver). But can't read .webm with DirectShowSource in avs+ cuda ver (use both AviSynth.dll, DevIL.dll, DirectShowSource.dll cuda ver) and meet notice error in Megui: "Unable to render the file. You probably don't have the correct filters installed"

[guest]

I have to ask a pretty basic question, but is there a noticeable advantage in using the CUDA "build" ?

[pinterf]

It can accept specially written cuda plugins, that's all.
The other difference is the xp compatibility. (Until Visual Studio 2019 exist on my machine)

[pinterf]

yeah, my machine can read .webm with DirectShowSource in avs+ cpu ver (use both AviSynth.dll, DevIL.dll, DirectShowSource.dll cpu ver). But can't read .webm with DirectShowSource in avs+ cuda ver (use both AviSynth.dll, DevIL.dll, DirectShowSource.dll cuda ver) and meet notice error in Megui: "Unable to render the file. You probably don't have the correct filters installed"
Then it seems not an avisynth problem. Check MEGUI configuration and plugin folders or whatever it needs. Does it use its own Avisynth or the centrally installed one. Check if plugins are seen by AviSynth invoked by MEGUI. Try your script from avsmeter64 or virtualdub2 or avspmod. Wheck if your MEGUI is 32 or 64 bits. These are my ideas.

[StainlessS]

Would there be any further complications for the MeGUI Avisynth Wrapper thingy with the CUDA whotsit ?

[VoodooFX]

@pinterf
Do you get notifications from your GitHub repositories? I've made few posts at the issues...

[pinterf]

Yes, I receive notifications. Sometimes too frequently :). Which one was forgotten?

[FranceBB]

Yes, I receive notifications. Sometimes too frequently :).

If only your Avisynth contributions counted towards your Jira :p
Employee of the month, every month! :D

[VoodooFX]

Yes, I receive notifications. Sometimes too frequently :). Which one was forgotten?
Sometimes me too, if I don't have anything to answer then I add that reaction emote so user would know that I've seen a post. :)

At your masktools and AvsInpaint.

[pinterf]

For avsinpaint I surely did not get message. The other one is familiar.

[csd79]

Hi Everyone!

Using the MRestoreVect function I get the error message: Error reading source frame 0: Avisynth read error: Filter Error: Filter attempted to break alignment of VideoFrame. The script:

frange = 2

raw = FFMS2(source="src.mkv").Trim(26855,27012)
super = MSuper(raw, pel=2)
vectors = MAnalyse(super, blksize=32, search=5, delta=frange, multi=true)

MStoreVect(vectors)

# 2nd script, for variable values see above
MRestoreVect(AVISource("mv.avi"))

MDegrainN(raw, super, last, frange, thSAD=400, thSAD2=150)

I'm using AS+ 3.7.0 and mvtools 2.7.45, x64 versions. Vectors are saved using Lagarith RGBA.

I get the same error with different sources: UHD 10bit YV12 and SD 8bit YV12.

Any suggestions?

[pinterf]

Hi Everyone!

Using the MRestoreVect function I get the error message: Error reading source frame 0: Avisynth read error: Filter Error: Filter attempted to break alignment of VideoFrame. The script:

frange = 2

raw = FFMS2(source="src.mkv").Trim(26855,27012)
super = MSuper(raw, pel=2)
vectors = MAnalyse(super, blksize=32, search=5, delta=frange, multi=true)

MStoreVect(vectors)

# 2nd script, for variable values see above
MRestoreVect(AVISource("mv.avi"))

MDegrainN(raw, super, last, frange, thSAD=400, thSAD2=150)

I'm using AS+ 3.7.0 and mvtools 2.7.45, x64 versions.

I get the same error with different sources: UHD 10bit YV12 and SD 8bit YV12.

Any suggestions?
Avisynth+ is right. MRestoreVect issue. It must be made a bit smarter.
(It wants to create a subframe starting on unaligned memory position)

[csd79]

Avisynth+ is right. MRestoreVect issue. It must be made a bit smarter.
(It wants to create a subframe starting on unaligned memory position)

OK, thank you for the clarification.

[MysteryX]

A syntax feature that's long-overdue in Avisynth (unless it got added and I didn't know) is default parameter values.


function MyFunc(bool "param" = True)
{
}


instead of

function MyFunc(bool "param")
{
param = Default(param, True)
}

[MysteryX]

ConvertToYUV444 ...

chromaresample allows setting any resizer, but is there a way to specify b and c parameters of Bicubic? Or are the default b=1/3 c=1/3 highly recommended over using b=0 c=0.5 ?

[wonkey_monkey]

Or are the default b=1/3 c=1/3 highly recommended over using b=0 c=0.5 ?

b=1/3, c=1/3 will soften the image. b=0, c=0.5 is more preferable in that it doesn't change the pixels under any "null" transformation.

This choice of defaults has previously caused issues: https://forum.doom9.org/showthread.php?p=1849901

[MysteryX]

Yet there's currently no way of upsampling using b=0 c=0.5 with ConvertToYUV444 -- which means that function should be avoided altogether? Or use Spline36.

That's the kind of feature that's way overdue in Avisynth.

[wonkey_monkey]

Spline16 will probably be closer to Bicubic.

[MysteryX]

Spline16 will probably be closer to Bicubic.
It depends on the usage. Spline16 is an unbalanced sharpening kernel. (Default Bicubic is unbalanced blurring kernel)

I'm looking for numerically-accurate data for better processing in YUV444, and Bicubic(0, .5) is the most numerically accurate.

Spline36 looks pretty similar to Bicubic(0, .5)

[poisondeathray]

ConvertToYUV444 ...

chromaresample allows setting any resizer, but is there a way to specify b and c parameters of Bicubic? Or are the default b=1/3 c=1/3 highly recommended over using b=0 c=0.5 ?

Another option would be avsresize

z_ConvertFormat (pixel_type=, resample_filter_uv = , filter_param_a_uv = , filter_param_b_uv= ) ...

[DTL]

It depends on the usage. Spline16 is an unbalanced sharpening kernel. (Default Bicubic is unbalanced blurring kernel)

I'm looking for numerically-accurate data for better processing in YUV444, and Bicubic(0, .5) is the most numerically accurate.


Avisynth converters are of medium quality for everyday consumer use. And using simple linear resizers can not fix the design bugs of subsampled colour systems. For some better quality you anyway need highly non-linear transforms. So if built-in quality is not enough - it is better use custom designed script with planes separating/combining or external plugin. The only simple way to avoid these bugs is not going from good 4:4:4 to subsampled.

[MysteryX]

In my script I'm offering 3 upsampling methods: bicubic(0, .5), nnedi3 and ChromaFeconstructor_faster

In any case, built-in methods should allow configuring b and c parameters when upsampling. It wouldn't cause any backwards incompatibility. Meanwhile I can use FMTC to do the job.

btw is there a nnedi3 resampling script written in Avisynth?

[kedautinh12]

Here:
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.6%20and%20up/nnedi3_resize16.avsi

[Dogway]

A few things I observed, converting from RGB to YUV in float bitdepth.
ConvertToyuv420(matrix="PC.709")
Output is greyscale, not sure if this is intended.

I was also benchmarking my metrics function to see if it had a slowdown compared to vanilla and observed it was like 30% slower the culprit being propCopy(). I'm not sure it can be sped up, a simple (yet too verbose) Get+Set has no speed penalty. I think propCopy also wants identical clips (bitdepth, resolution, format)?

[DTL]

In my script I'm offering 3 upsampling methods: bicubic(0, .5), nnedi3 and ChromaFeconstructor_faster

In any case, built-in methods should allow configuring b and c parameters when upsampling.

I'm looking for numerically-accurate data for better processing in YUV444, and Bicubic(0, .5) is the most numerically accurate.


There are minimum 2 different types of subsampled to 4:4:4 convertors - for intermediate processing work (with packing back to subsampled at output) and for final result (for displaying without additional distortions). It is because of design of subsampled colour systems with conditioning of luma data at source side but conditioning of chroma data at receiver side. It is to decrease loss of chroma sharpness at the many generations of subsampled<->full_band conversions.
But 'final' result is when frame samples number is not changed. If resizing is performed - it may be better to use 'final' conversion before resize and may be other processing.
So 4:4:4 output of 'intermediate' type convertor will be 'distorted'.

The 'intermediate' type 'linear' convertto444 have 'flat' chroma channels frequency response (close to sinc-resize) and 'final' have 'slow rolloff' type like gauss-resize (or better). The worst news here is that the standard frequency response of the 'final/displaying' conditioning filter of convertto444 not exist. It either supposed to be known by designer (from common DSP) or still up to designer's of current conversion taste. It also mean the 'perfect convertto_subsampled' still can not be designed because it depends on the backward conversion.

The current defaults for convertto444 may be selected to be in-between of 'intermediate' and 'final' conversions so average user without many generations of conversion from and to subsampled will not got more chroma blurring but also if trying to use it as 'final' conversion - do not got more over/undershooting and/or ringing. So it may be sort of 'fail-safe' defaults for average low-experienced user.

[MysteryX]

DTL, I don't think ConvertToYUV444 is anywhere that sophisticated in terms of "balance". It does a plain resize, with whatever resizer you want, whether Spline36, Gauss or Bicubic. Although it allows changing the resizer, it just doesn't allow changing Bicubic values and keeps the defaults. I think that's all there is to it.

[DTL]

"ConvertToYUV444"

It not named ConvertToFin444() and ConvertToImm444(), so it have some hidden default inside. And the description allow to select any of available built-in resampler's kernel just to pick possibly better for current task. And typically the request to pass arguments to parametrical kernels looks like very rare (because it looks most of users possibly even not need different from default kernel).

I personally hope the awful shadows of the poor past with subsampled chroma compression will be deprecated at some day so we finally can forgot these conversions. It is already put to grave interlace-compression so I hope in some day the subsampled-compression and even transfer-function-compression will also be killed.

Also this conversion is sort of built-in macro for a sequence of separate-resample-combine planes operations. And possibly the user-defined function will have close performance and can use any required resampler. As I see programmers do not likes an idea to pack long macro into one function with tons of params for each used in macro simple function.

The sad truth of this function is hidden - correct name is not convert but DecompressColourTo(Final/Immediate)444(). So when you use modern (U)HDTV in 2021 not in 4:4:4 you still see ugly ancient colour-compressed system designed half+ century ago.

Nowdays as Avisynth progress it start to have many _clip properties to assist of auto-options selection to prevent gross-errors by low experienced users. So it is good to set property of colour-decompressed data in 4:4:4 format to be 'flat' or 'conditioned' (and band-limited to 1/2h and or 1/2v) colour-difference spectrum. So the hint-aware next processing software can use it. Like resize after decompressto444 with 'flat, (1/2v), 1/2h' chroma-diff spectrum properties need additional colour-difference conditioning inside 1/2 allowed frequency band operation before resampling.

[MysteryX]

Here:
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.6%20and%20up/nnedi3_resize16.avsi
What parameter to change subsampling?

[MysteryX]

I personally hope the awful shadows of the poor past with subsampled chroma compression will be deprecated at some day so we finally can forgot these conversions.
Even if it got deprecated, you'd still have to deal with it in the majority of videos already produced.

And with 4K and 8K videos, we're heading towards more compression, not less compression. Bandwidth is becoming even more important than before. Reducing Chroma compression would mean losing Luma details. That's the reason subsampling was created to begin with. And for 8K footage, subsampling becomes so tiny that it has much less impact than it used to be; yet huge compressibility benefits.

Although. With the move from Pixel-based videos to Vector-based videos, subsampling might indeed become a thing of the past? I still doubt it.

[DTL]

"And with 4K and 8K videos, we're heading towards more compression, not less compression. Bandwidth is becoming even more important than before. Reducing Chroma compression would mean losing Luma details. That's the reason subsampling was created to begin with. And for 8K footage, subsampling becomes so tiny that it has much less impact than it used to be; yet huge compressibility benefits."

If current (progressing) civilization claims to make better quality video systems it is good to deprecate old poor and ugly compression systems. The progress is very slow - to 2021 only 2:1 compression of interlacing scan finally going to death.
Also it is good to exchange poor old compression methods to new higher quality. So 4:2:0/4:2:2 chroma-compression put distortions that can not be fixed by MPEG compression stage. But current MPEG put less distortions into 4:4:4 footage with same output bitrate as 4:2:0+MPEG. MPEG have much more advanced colour-treatment and compression methods in compare with poor old ugly '2:1 subsampling'. And MPEG have tons of updates at half of century interval (from MPEG-1 to current 265+). But poor ugly old chroma-subsampling compression still not completely documented and have zero updates.
I make tests with 4:4:4-version of x264 - it creates even less output bitrate if feeded with 4:4:4 version (same crf-value encoding) - not 2x larger as may be expected. But 4:4:4 x264 is not compatible with poor-people viewing hardware.

[DTL]

"move from Pixel-based"

It is one source why Avisynth is poor in many places for processing of professional samples-based motion pictures data. It is directly connected to the resampling and 4:2:x to 4:4:4 too.

It looks was started by computer pixel-based programmers. And lately more or less supplemented with sample-based methods of processing. But still far from complete.

[kedautinh12]

What parameter to change subsampling?

What do you mean?? It's right??
https://github.com/realfinder/AVS-Stuff/blob/cd4e6e41ba7c50be2e824a22d45935bb50986cb7/avs%202.6%20and%20up/nnedi3_resize16.avsi#L190

[MysteryX]

What do you mean?? It's right??
https://github.com/realfinder/AVS-Stuff/blob/cd4e6e41ba7c50be2e824a22d45935bb50986cb7/avs%202.6%20and%20up/nnedi3_resize16.avsi#L190
Chroma placement is something else. I need to say "upscale this clip from 420 to 444". Then it does need to know whether "420" has MPEG1 or MPEG2 chroma position before converting it to "444". NNEDI3 can work well for chroma upsampling because it's an exact double resolution.

[FranceBB]

Then it does need to know whether "420" has MPEG1 or MPEG2 chroma position before converting it to "444".

Or the new hellish top_left (i.e 4:2:0 type 2) and honestly we would have been living happily without it nowadays. I still don't understand why they decided to change it...

[pinterf]

A few things I observed, converting from RGB to YUV in float bitdepth.
ConvertToyuv420(matrix="PC.709")
Output is greyscale, not sure if this is intended.

I was also benchmarking my metrics function to see if it had a slowdown compared to vanilla and observed it was like 30% slower the culprit being propCopy(). I'm not sure it can be sped up, a simple (yet too verbose) Get+Set has no speed penalty. I think propCopy also wants identical clips (bitdepth, resolution, format)?
Greyscale is definitely not intended. Slowdown: don't know why, worth to understand its reason. Problem registered.
EDIT: greyscale problem is fixed on my test bench.

[DTL]

Also why ConvertTo444 is of not best quality: It is better to make scaling in 'linear domain'. But CbCr data typically (and by standard) obtained from R'G'B' non-linear data. So to make simple upscaling of chroma data for 4:4:4 it is required long quest (and still not perfect): Make Y' half size (it is typically not delivered to decompressor - but inside Avisynth environment it is possible in theory). Y' is not-linear so simple dowscaling to 1/2 is not perfect. To dematrix half size chroma with Y' half size to get R'G'B' half size - de-transfer to linear RGB - finally upscale half size RGB to half-band full-size RGB - transfer to R'G'B' and matrix to Y'CbCr full size for combine to output 4:4:4.
The simple upscale of CbCr in its delivering non-linear domain is only partially good idea.

[DTL]

I found one place of large enough memalloc in DeviceManager.cpp -> CPUDevice class and put simple enough patch:

class CPUDevice : public Device {
public:
CPUDevice(InternalEnvironment* env)
: Device(DEV_TYPE_CPU, 0, 0, env)
{
HANDLE hToken = NULL;
TOKEN_PRIVILEGES tp;

// Enable this priveledge for the current process
if (!OpenProcessToken(GetCurrentProcess(), TOKEN_QUERY | TOKEN_ADJUST_PRIVILEGES, &hToken))
{
env->ThrowError("LargePages: Can not open process token");
return;
}

tp.PrivilegeCount = 1;
tp.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;

if (!LookupPrivilegeValue(NULL, SE_LOCK_MEMORY_NAME, &tp.Privileges[0].Luid))
{
env->ThrowError("LargePages: LookupPrivilegeValue failed.");
return;
}

BOOL result = AdjustTokenPrivileges(hToken, FALSE, &tp, 0, (PTOKEN_PRIVILEGES)NULL, 0);
DWORD error = GetLastError();

if (!result || (error != ERROR_SUCCESS))
{
env->ThrowError("LargePages: AdjustTokenPrivileges failed.");
return;
}

// Cleanup
CloseHandle(hToken);
hToken = NULL;

}

virtual int SetMemoryMax(int mem)
{
// memory_max for CPU device is not implemented here.
env->ThrowError("Not implemented ...");
return 0;
}

virtual BYTE* Allocate(size_t size, int margin)
{
size += margin;
#ifdef _DEBUG
// large pages
SIZE_T stLPGranularity = GetLargePageMinimum();
size_t iNumLPUnits = size / stLPGranularity;
SIZE_T stSizeToAlloc = (iNumLPUnits + 1) * stLPGranularity;

BYTE* data = (BYTE*)VirtualAlloc(0, stSizeToAlloc, MEM_LARGE_PAGES | MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE);
DWORD error = GetLastError();

if (error != ERROR_SUCCESS)
env->ThrowError("LargePages alloc error. GetLastError returned: %d\n", error);

// BYTE* data = new BYTE[size + 16];
int *pInt = (int *)(data + size);
pInt[0] = 0xDEADBEEF;
pInt[1] = 0xDEADBEEF;
pInt[2] = 0xDEADBEEF;
pInt[3] = 0xDEADBEEF;

static const BYTE filler[] = { 0x0A, 0x11, 0x0C, 0xA7, 0xED };
BYTE* pByte = data;
BYTE* q = pByte + size / 5 * 5;
for (; pByte < q; pByte += 5)
{
pByte[0] = filler[0];
pByte[1] = filler[1];
pByte[2] = filler[2];
pByte[3] = filler[3];
pByte[4] = filler[4];
}
return data;
#else
// return new BYTE[size + 16];
// large pages
SIZE_T stLPGranularity = GetLargePageMinimum();
size_t iNumLPUnits = size / stLPGranularity;
SIZE_T stSizeToAlloc = (iNumLPUnits + 1) * stLPGranularity;

BYTE* data = (BYTE*)VirtualAlloc(0, stSizeToAlloc, MEM_LARGE_PAGES | MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE);
DWORD error = GetLastError();

if (error != ERROR_SUCCESS)
env->ThrowError("LargePages alloc error. GetLastError returned: %d\n", error);
return data;
#endif
}

virtual void Free(BYTE* ptr)
{
if (ptr != nullptr) {
// large pages here
VirtualFree(ptr, 0, MEM_RELEASE);
// delete[] ptr;
}
}


It works but not enough - somewhere still many 'standard' allocations:

Current stat at 4K MDegrainN with tr=6:
About 1 GB in large pages, about 300 Mb in standard pages, PageTable size of all AVSMeter process about 1.7 MB.
Standard AVS 3.6 (latest download from repository 3.7-master do not builds in Release at me with VS2019) - total standard pages about 1.6 GB and PageTable size about 3.7 MB. I use RAMMap tool to check - https://docs.microsoft.com/ru-ru/sysinternals/downloads/rammap .

For compare - xmrig with 2 GB total memory have PageTable 5 MB with standard pages and 300K with large pages.

Maybe someone with more knowlege in AVS core can point to other places of residual memory allocations (in avs core ?) ? I try to search 'new/delete' pairs now.

[FranceBB]

Is Prefetch() NUMA Node aware?
I can't seem to be able to make use of two CPUs at the same time, no matter how many threads I set :(
In a dual socket motherboard, it always uses either CPU0 or CPU1.
This way I can only use up to 28c/56th but I have 2 CPUs 28c/56th each, so I'd like to make use of 56c/112th :(

[kedautinh12]

Rich boy :D

[FranceBB]

Rich boy :D

When your company has the "kaching"

https://media2.giphy.com/media/hrQnFqUKTXwSSvH8AI/200.gif

[DJATOM]

Is Prefetch() NUMA Node aware?
I can't seem to be able to make use of two CPUs at the same time, no matter how many threads I set :(
In a dual socket motherboard, it always uses either CPU0 or CPU1.
This way I can only use up to 28c/56th but I have 2 CPUs 28c/56th each, so I'd like to make use of 56c/112th :(

I did observe that effect with x265. For dual CPU systems, it's loading all 16 threads on CPU0 and CPU1 is mostly idling. Maybe something is wrong in the Windows server OS, I don't know that.

[DTL]

Is Prefetch() NUMA Node aware?
I can't seem to be able to make use of two CPUs at the same time, no matter how many threads I set :(
In a dual socket motherboard, it always uses either CPU0 or CPU1.
This way I can only use up to 28c/56th but I have 2 CPUs 28c/56th each, so I'd like to make use of 56c/112th :(

As followed from https://gcc.gnu.org/bugzilla/attachment.cgi?id=44273 the new application wanting to run > 64threads at > 64 processors must check number of processors groups and number of processor threads (in each group) and expicitly assign > 64 threads to different processors groups. It is significant multithreading re-write of application. The 'free/auto' multuthreading by OS support is limited to 64 processors of 1 group. No more 'auto-sheduling' of the 1 easy to create threadpool between > 64 processors. Now application need to create different sets of threads with max 64 threads in each set and assign to different processors groups. The new multi-processor-groups AVS version is needed.

And it still not guaranteed any good performance scaling with > 64 threads because typical MT model assumes about equal access of different threads to units of data in memory and different processor group may have (and possibly will have) not equal. So it is required to re-design MultiThreaded data processign functions to be MultiGroup-MultiThreaded aware for best performance.

[Dogway]

Maybe ConvertBits() could write to frameprops, it might be useful for example for AvSpmod which doesn't know bitdepth of output clip, so it can show proper info.

Also chained comparisons are not supported in Expr() right? (3 <= x <= 5) Not sure if that would save any performance though.

Yesterday I was playing with Prefetch(), since it is well known that convolutions perform better with physical cores and long expressions better with logical I tested to run something like:
Convolution
Prefetch(4)
Expression
Prefetch(6) # normally better than 8 for my 4C/8T CPU

It happened to perform worse than just running everything with Prefetch(6). Not sure what to extract from it, just some insight.

[pinterf]

VapourSynth developers were experimenting with large pages since 2018. Finally the idea was abandoned. Check commit history and search for 'page'.
'Add implementation', 'add hack to disable', 'disable if Windows is broken', 'disable in release build'. I think I trust them.

[pinterf]

Maybe ConvertBits() could write to frameprops, it might be useful for example for AvSpmod which doesn't know bitdepth of output clip, so it can show proper info.

? AvsPMod knows everything about the clip format.


Also chained comparisons are not supported in Expr() right? (3 <= x <= 5) Not sure if that would save any performance though.

Not supported of course.

[DTL]

Yes - large page option is not for everyday use of typical user. It is still server-class feature even in the win10 and most easy to use in system-startup services. With application it still may easy fail allocation with 'not enough resources' error after some time after system startup. So even win10 memory manager still not like to defragment memory. Also at different versions of win10 may not allowed even to have >1 processes with large page using. Need to close previous large page using application to start new. So it not comfortable for everyday use feature and only may be useful for external plugins with random enough use of large memory arrays. Most of AVS core functions like RGBAdjust() uses sequential scan and 1 access per sample for process so possibly will not any gain from large pages.
Unfortunately some external plugins like MVtools do not use internal memory management and were specially re-written for AVS memory management and caching so the only way to use large pages in MVtools is to have AVS core modified.

[tormento]

Yes - large page option is not for everyday use of typical user. It is still server-class feature even in the win10 and most easy to use in system-startup services.
If my memory recalls it correctly, 7-Zip has a large page option since years and I am currently using it on Win 11 Enterprise with marginal differences.

[Dogway]

? AvsPMod knows everything about the clip format.
Then I might have had the wrong impression from this post (https://forum.doom9.org/showthread.php?p=1942719#post1942719)

[DTL]

large page option"

It can try but if allocation call fails - perform silent fallback to standard 4kB pages. I still not know if it possible to force windows to perform memory defragmentation and avoid reboot if it returns 'out of resources' return code. Avisynth looks like do not have log messages output for standard user so it may be not possible to display if it use large pages or not. The only indirect way is to check memory usage - the large page allocations typically not visible in most of memory using display software (AVSMeter also do not see it). So if it really used the displayed amount of memory in taskanager starts to become low (in most of available columns). Even RAMmap tool do not display number of largepages per process - only total number of largepages in the system.

It looks the application mostly benefits from large page when it uses small number of large size structures. Avisynth partially meet the requirement when uses small number of large video frame buffers of several MB in size. But it looks only part of total used memory (about 2/3). Other part looks like some structures of cache that may be uses copy of C++ objects made by even not AVS core but some external C++ library. I still do not found the place where it allocates lots of memory after parsing the script and walking through Manalyse and Mdegrain and finally after the Preftech(). It looks it somewhere at preparation lots of copies of objects for multithreading. To get full advantage of large pages the PageTable need to be small enough - the 2GB memory in 2 MB pages and 8byte per PageTable entry can be only 8 kByte in size. But now Avisynth uses about 1.7 MB PageTable for about 300 MB allocation outside the CPUDevice video buffers.

I need to found some memory display debug software to show each memory allocation by software and its size - to view how many total separated allocations and to pick largest first. But if Avisynth programmers were too lazy and uses lots of very small allocations - it will also overload PagesTable because it require at least one page per allocation and one entry in PT. And uses 2MB pages per very small allocations will overuse the memory. So to full benefit from large pages programmers need to be more smart to uses large structures up to 2MB page size in small amout of structures.
As i remember there exist some tools for detecting memory leaks and can display all allocations by current process. But may be something already exist in VisualStudio (I currently trying to set 2019 at home because it looks like have better CMake support for AVS compiling) ?

[FranceBB]

Then I might have had the wrong impression from this post (https://forum.doom9.org/showthread.php?p=1942719#post1942719)

AVSPmod knows about high bit depth, however due to the python library implementation, the preview will always be 8bit, therefore, if you work in 16bit and you use a color picker on the AVSPmod preview, every value will be scaled down to its corresponding 8bit value. It's always been like this: it was like this with 16bit stacked and interleave and it's like this with 16bit planar. Unfortunately there isn't much we can do. It's actually also the reason why we can't have BT2020 displayed properly despite using Windows 10 or Windows 11 and a BT2020 capable display. :(

[Dogway]

I'm fine with 8-bit preview but the color values in the status bar show 235 for white instead of 255 (ie. scaled from 16-bit). I mean if AvsPmod knows about frameprops and its values, then it knows about bitdepth and can scale those values according (by bitshift or full stretch) if one wants to show them in 16-bits or another bitdepth, no need for python support. We should also support bitdepth scale in frameprops to round it down so all clip information is within frame properties.

As for BT2020 I don't think there are capable monitors to show its gamut, it's mostly Display-P3 but that is enough for most non pure saturated real surface colors.

[gispos]

Then I might have had the wrong impression from this post (https://forum.doom9.org/showthread.php?p=1942719#post1942719)
We've had that topic before.
I'm just not smart enough to implement that.
It is correct that python only returns 8bit from the display, but I am now totally overwhelmed how that would behave with a real 10bit display ... no idea.

However, the values of the pixels can also be calculated, but only for 8 bits.
And now to all the clever minds, below is the code for YUV and RGB calculations for 8bit.
POST me the code for other color depths, I'll be happy to implement that.

https://forum.doom9.org/showthread.php?p=1879721#post1879721

[VoodooFX]

I'm writing one function using YPlaneMinMaxDifference() ect... it would help if those runtime functions could be masked.

[Dogway]

And now to all the clever minds, below is the code for YUV and RGB calculations for 8bit.
POST me the code for other color depths, I'll be happy to implement that.

I have a 10bit monitor, but I keep all HDR stuff disabled as I see no point for working, unless it's a master monitor. Python has bitshift so as far as you know the clip bitdepth you can do:

32-bit float: 8-bit / 255 (then for UV planes shift to minus -0.5)
This will only show from 0 to 1 though, when 32-bit float in avs+ is capable of -+128.000000 (stored in 16-bit container)

10-16-bit int: Python can do bitshift. n << (bitdepth-8)

10-16-bit int: For full scale you don't have access to that information, unless pinterf says otherwise. But if you know I did it like so in ex_bs().
round(n << (bitdepth-8) + (n << (bitdepth-8)) / 256) (ie. ((16<<8)+(16<<8)/256))
You can also do
round((257/256) * (n << (bitdepth-8) ))

[pinterf]

We've had that topic before.
I'm just not smart enough to implement that.
It is correct that python only returns 8bit from the display, but I am now totally overwhelmed how that would behave with a real 10bit display ... no idea.

However, the values of the pixels can also be calculated, but only for 8 bits.
And now to all the clever minds, below is the code for YUV and RGB calculations for 8bit.
POST me the code for other color depths, I'll be happy to implement that.

https://forum.doom9.org/showthread.php?p=1879721#post1879721

I think of something like this, see IsPlanar + Y-only case.
(may not work out-of-box but the idea is there)
def GetPixelYUV(self, x, y):
x = x * self.component_size

if self.IsPlanar:
indexY = x + y * self.pitch
if self.IsY:
if self.component_size == 1: #8 bit
return (self.ptrY[indexY], -1, -1)
elif self.component_size == 2: # up to 16 bits
buffer = [self.ptrY[indexY], self.ptrY[indexY + 1]]
return (int.from_bytes(buffer, byteorder='little'), -1, -1)
else: # 32 bit float
buffer = [self.ptrY[indexY], self.ptrY[indexY + 1], self.ptrY[indexY + 2], self.ptrY[indexY + 3]]
buf = bytearray(buffer)
return (struct.unpack('<f', buf) , -1, -1)

x = x >> self.WidthSubsampling
y = y >> self.HeightSubsampling
indexU = indexV = x + y * self.pitchUV
elif self.IsYUY2:
indexY = (x*2) + y * self.pitch
indexU = 4*(x/2) + 1 + y * self.pitch
indexV = 4*(x/2) + 3 + y * self.pitch
else:
return (-1,-1,-1)
return (self.ptrY[indexY], self.ptrU[indexU], self.ptrV[indexV])

[gispos]

I think of something like this, see IsPlanar + Y-only case.
(may not work out-of-box but the idea is there)
def GetPixelYUV(self, x, y):
x = x * self.component_size

if self.IsPlanar:
indexY = x + y * self.pitch
if self.IsY:
if self.component_size == 1: #8 bit
return (self.ptrY[indexY], -1, -1)
elif self.component_size == 2: # up to 16 bits
buffer = [self.ptrY[indexY], self.ptrY[indexY + 1]]
return (int.from_bytes(buffer, byteorder='little'), -1, -1)
else: # 32 bit float
buffer = [self.ptrY[indexY], self.ptrY[indexY + 1], self.ptrY[indexY + 2], self.ptrY[indexY + 3]]
buf = bytearray(buffer)
return (struct.unpack('<f', buf) , -1, -1)

x = x >> self.WidthSubsampling
y = y >> self.HeightSubsampling
indexU = indexV = x + y * self.pitchUV
elif self.IsYUY2:
indexY = (x*2) + y * self.pitch
indexU = 4*(x/2) + 1 + y * self.pitch
indexV = 4*(x/2) + 3 + y * self.pitch
else:
return (-1,-1,-1)
return (self.ptrY[indexY], self.ptrU[indexU], self.ptrV[indexV])

Thank you Ferenc but unfortunately no success.

When looking at the code I thought that it could work with it, especially the non Y section seemed logical to me.
But very often 0,0,0 is returned, or values only within the byte range.
I looked at other relevant parts and didn't notice anything that could change the values to byte.

I had tried RGB before and thought that there shouldn't be any problems with it.
bytes = self.vi.bytes_from_pixels(1) should return the correct size to multiply regardless of the color depth (I thought)
and then just multiply, indexB = (x * bytes) + (self.Height - 1 - y) * self.pitch
Should be the right position for BGR and should return the right value, but unfortunately not either.

[pinterf]

Meanwhile, powered up by the 3rd Covid vaccine :)
Avisynth+ 3.7.1 test build 23 (20211103) (https://drive.google.com/uc?export=download&id=1UfjtV3FMqzCjI8s-A864KzWUO_dAdaAR)
Well, this time the source suffered significant changes and additions. Let's hope the best. Anyway, the first Avisynth version which really sets _some_ common frame properties.
And a fix after Dogway's issue, thanks for reporting.
20211103 WIP
------------
- frame propery support: preliminary _Matrix and _ColorRange in various filters

Summary:

_Matrix constants - as seen in propShow() Constants will probably appear for developers in a header file. Not in Avisynth script.

AVS_MATRIX_RGB 0
AVS_MATRIX_BT709 1
AVS_MATRIX_UNSPECIFIED 2
AVS_MATRIX_FCC 4
AVS_MATRIX_BT470_BG 5 (BT601)
AVS_MATRIX_ST170_M 6 (practically same as 5)
AVS_MATRIX_ST240_M 7
AVS_MATRIX_YCGCO 8 (not supported by internal converters)
AVS_MATRIX_BT2020_NCL 9
AVS_MATRIX_BT2020_CL 10 (same as 9)
AVS_MATRIX_CHROMATICITY_DERIVED_NCL 12 (not supported by internal converters)
AVS_MATRIX_CHROMATICITY_DERIVED_CL 13 (not supported by internal converters)
AVS_MATRIX_ICTCP 14 (not supported by internal converters)

_ColorRange constants:

AVS_RANGE_FULL = 0
AVS_RANGE_LIMITED = 1

string "matrix" parameter possible values and their mapping (used in YUV-RGB converters)

"rgb" AVS_MATRIX_RGB
"709" AVS_MATRIX_BT709
"unspec" AVS_MATRIX_UNSPECIFIED
"170m" AVS_MATRIX_ST170_M
"240m" AVS_MATRIX_ST240_M
"470bg" AVS_MATRIX_BT470_BG
"fcc" AVS_MATRIX_FCC
"ycgco" AVS_MATRIX_YCGCO not supported
"2020ncl" AVS_MATRIX_BT2020_NCL
"2020cl" AVS_MATRIX_BT2020_CL same as 2020ncl
"chromacl" AVS_MATRIX_CHROMATICITY_DERIVED_CL not supported
"chromancl" AVS_MATRIX_CHROMATICITY_DERIVED_NCL not supported
"ictcp" AVS_MATRIX_ICTCP not supported
"601" AVS_MATRIX_BT470_BG compatibility alias
"2020" AVS_MATRIX_BT2020_NCL compatibility alias

the above "matrix" parameters can be followed by a "full" or "f" and "limited" or "l" or "auto" marker after a ":"
e.g. "709:f" means the same as the old "PC.709"
When there is no limited-ness marker, or is set to "auto" then value of _ColorRange frame property is used

old-style "matrix" parameters are kept, their name indicate the full/limited
For memo and the similar new string
"rec601" same as "170m:l"
"rec709" "709:l"
"pc.601" and "pc601" "170m:f"
"pc.709" and "pc709" "709:f"
"average" - kept for compatibility, really it has no standard _Matrix equivalent
"rec2020" "2020cl:l"
"pc.2020" and "pc2020" "2020cl:f"

- RGB<->YUV (YUY2) conversions: frame property support _Matrix and _ColorRange
Unlike smart external plugins, in Avisynth there is a single "matrix" parameter,
since the function names explicitely tell whether we are converting from RGB or to RGB.

New: additional "matrix" parameter values: see table above.
With a new syntax "170m", "240m", "fcc" are newly available matrixes.
New-style matrix name can be:
matrix name
or
matrix name : full_or_limited_marker
"auto" can appear on before and after the ":" character, e.g. "auto:full" will take matrix from frame property or default
When converting to RGB the _Matrix parameter is set to 0 ("rgb")
- ConvertBits: frame property support: _ColorRange
When parameter "fulls" is not specified, whether the source clip is full or limited is decided on _ColorRange frame property.
When no property available, then RGB clips are treated as fulls=true, while YUV are fulls=false.
If not specified, "fulld" parameter will inherit the value of the established fulls
- ColorBars: frame property support: writes _Matrix and _ColorRange.
RGB: _ColorRange = 1 ("limited") - ColorBars is using studio RGB values
_Matrix = 0 ("rgb")
- ColorBarsHD: frame property support:
_ColorRange = 1 ("limited"), _Matrix = 1 ("709")
- BlankClip: frame property support:
RGB: _ColorRange = 0 ("full"), _Matrix = 1 ("709")
YUV: _ColorRange = 1 ("limited"), _Matrix = 6 ("170m")

- Fix: Planar RGB 32 bit -> YUV matrix="PC.709"/"PC.601"/"PC.2020" resulted in greyscale image

[FranceBB]

Awesome!
Well done, Ferenc! :D

https://c.tenor.com/ZWopsXeO7tQAAAAd/clapping-applause.gif

I look forward to test those, especially 'cause this makes the burden of setting frame properties less big for the user.
A very nice step in the right direction, bring it on! :D

[gispos]

I have a 10bit monitor, but I keep all HDR stuff disabled as I see no point for working, unless it's a master monitor. Python has bitshift so as far as you know the clip bitdepth you can do:

32-bit float: 8-bit / 255 (then for UV planes shift to minus -0.5)
This will only show from 0 to 1 though, when 32-bit float in avs+ is capable of -+128.000000 (stored in 16-bit container)

10-16-bit int: Python can do bitshift. n << (bitdepth-8)

10-16-bit int: For full scale you don't have access to that information, unless pinterf says otherwise. But if you know I did it like so in ex_bs().
round(n << (bitdepth-8) + (n << (bitdepth-8)) / 256) (ie. ((16<<8)+(16<<8)/256))
You can also do
round((257/256) * (n << (bitdepth-8) ))
Hello Dogway, take a look at this and tell me if this can be so right

From left to right and top to bottom the first two lines (13 color fields)

ColorBarsHD(width=1280, height=720, pixel_type="YV24")
ConvertBits(bits=16, truerange=true, fulls=true, fulld=true)

*rgb=(26728,26471,26728) *yuv=(26925,32842,32862)
*rgb=(49601,49087,49601) *yuv=(46444,32917,32957)
*rgb=(49601,49087,257) *yuv=(41608,11255,36460)
*rgb=(257,49087,49344) *yuv=(33737,40107,11313)
*rgb=(257,49087,0) *yuv=(28901,18445,14816)
*rgb=(49601,0,49858) *yuv=(21729,47314,51002)
*rgb=(49601,0,514) *yuv=(16893,25652,54506)
*rgb=(257,0,49601) *yuv=(9022,54504,29359)
*rgb=(26728,26471,26728) *yuv=(26925,32842,32862)
*rgb=(0,65535,65535) *yuv=(43548,42467,3998)
*rgb=(27242,10794,257) *yuv=(16562,25707,40736)
*rgb=(49601,49087,49601) *yuv=(46444,32917,32957)
*rgb=(514,0,65535) *yuv=(10650,61461,28340)


ColorBarsHD(width=1280, height=720, pixel_type="YV24")
ConvertBits(bits=16, truerange=false, fulls=false, fulld=false)

*rgb=(26214,26214,26214) *yuv=(26613,32768,32768)
*rgb=(49087,49087,49087) *yuv=(46261,32768,32768)
*rgb=(49087,49087,0) *yuv=(41451,11218,36253)
*rgb=(0,49087,48830) *yuv=(33621,39920,11237)
*rgb=(0,49087,0) *yuv=(28835,18483,14703)
*rgb=(49087,0,49344) *yuv=(21547,47165,50813)
*rgb=(49087,0,257) *yuv=(16736,25615,54298)
*rgb=(0,0,49087) *yuv=(8906,54317,29282)
*rgb=(26214,26214,26214) *yuv=(26613,32768,32768)
*rgb=(0,65278,65535) *yuv=(43418,42541,4092)
*rgb=(26728,10794,0) *yuv=(16405,25671,40529)
*rgb=(49087,49087,49087) *yuv=(46261,32768,32768)
*rgb=(257,0,65535) *yuv=(10584,61499,28227)


Or this one (without averaging)

ColorBarsHD(width=1280, height=720, pixel_type="YV24")
ConvertBits(bits=16, truerange=false, fulls=false, fulld=false)

*rgb=(26112,26112,26112) *yuv=(26526,32768,32768)
*rgb=(48896,48896,48896) *yuv=(46097,32768,32768)
*rgb=(48896,48896,0) *yuv=(41305,11302,36239)
*rgb=(0,48896,48640) *yuv=(33506,39892,11320)
*rgb=(0,48896,0) *yuv=(28739,18539,14774)
*rgb=(48896,0,49152) *yuv=(21479,47109,50743)
*rgb=(48896,0,256) *yuv=(16687,25643,54215)
*rgb=(0,0,48896) *yuv=(8887,54233,29296)
*rgb=(26112,26112,26112) *yuv=(26526,32768,32768)
*rgb=(0,65024,65280) *yuv=(43265,42503,4204)
*rgb=(26624,10752,0) *yuv=(16357,25698,40499)
*rgb=(48896,48896,48896) *yuv=(46097,32768,32768)
*rgb=(256,0,65280) *yuv=(10559,61388,28245)

[johnmeyer]

I am unable to get AVISynth+ to open anything under Windows 7 64-bit SP1.

I downloaded and installed "AviSynthPlus_3.7.0_20210111_vcredist.exe" using the defaults, and running as Administroator. I then tried to open video frameserved from Vegas using this one-line script:

AVISource("C:\Users\User\Documents\fs.avi")

AVISynth throws the error "couldn't locate a decompressor"

I then tried:DirectShowSource("C:\Users\User\Documents\fs.avi")but that gave me all sorts of errors, including "could not open as video or audio," and "Pins cannot connect due to not supporting the same transport." AVISource still throws the same error.

I tried opening a dozen other video files, rather than the framerserver signpost. I uninstalled and re-installed. I spent over an hour searching both this forum and the Internet at large.

I did note that the installation installed TWO Microsoft Visual C++, redistributables, one for the 32-bit and one for the 64-bit.

I did try to install just the 64-bit, but the installation did not complete because it did not ask me to re-boot.

Sadly, this means I won't be able to work on the road the next five days.

Any suggestions, I'd sure appreciate it.

edit: I uninstalled everything and then installed just the 64-bit version. Using DirectShow I get an error message "cannot load a 32-bit DLL in 64-bit AVISynth ... AGC.DLL"

[kedautinh12]

Try L-SMASH source, can open .avi file
https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/releases

[johnmeyer]

Try L-SMASH source, can open .avi file
https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/releasesThanks for that. I've downloaded it, but I found out the problem: 64-bit AVISynth+ just doesn't work on my system. I removed everything, and installed only the 32-bit AVISynth+. It didn't work quite right until I also used the 32-bit version of VirtualDub.

So, as long as I stay with everything at 32-bit, everything works just like it has on my other computers. Since I could care less about 64-bit, I'll just proceed with what I have working now.

[FranceBB]

Even if you're not on XP, what about the XP build for x64? We don't know why it didn't run on your system, but the XP build uses v141_xp so it should be more compatible.

[pinterf]

Decompressor and avi opening problems swow that Avisynth works but a specific codec which knows your avi internal format is missing.

My test versions are still good for xp I think.
Install 3.7.0 then overwrite the files with the unzipped ones.

32 bit dll among the 64 bit ones is definitely user error, move that file away from your plugin folder. 64 bit Avisynth (or anything) is not able to consume 32 bit dlls.

[Reel.Deel]

@johnmeyer

I think you may be doing some things wrong. I have 64-bit AviSynth+ installed on my Windows 7 PC and also on a 11 year old MacBook Pro running Windows 7 and both work flawlessly. And if I'm not mistaken there are some users using 64-bit AVS+ even on Windows XP.

With that being said, based on the information you provided it seems you are trying to use some 32-bit plugins on a 64-bit environment. HDRAGC is one of the few plugins that is still used that's only in 32-bit because it's closed source. It will only work with 32-bit AVS+ and a 32-bit host. You can also use MP_Pipeline (http://avisynth.nl/index.php/MP_Pipeline) to load a 32-bit plugin in a 64-bit environment.

Some suggestions:

Install the AIO Repack for latest Microsoft Visual C++ Redistributable Runtimes (https://github.com/abbodi1406/vcredist/releases)
Install VirtualDub2 (https://sourceforge.net/projects/vdfiltermod/) - it includes both 32/64-bit versions. Not sure if you have this already or not, I did not see you mention it.
Start with a clean AviSynth+ install and only install plugins that you know are 64-bit into the appropriate install locations. An almost complete list of 64-bit plugins can be found here: http://avisynth.nl/index.php/Category:Plugins_x64 - (only shows 200 per page so you have to click next to view all)

[Dogway]

I get invalid arguments with the next call, still on test22:
a=last
b=propClearAll()

scriptclip(b, function [a,b] () {
propcopy(b,a,"_Matrix")
} )

"Invalid arguments to function 'propcopy'"
Maybe I missed something, sometimes it works for me others doesn't. 'a' has the attribute, also tested with custom properties.

@gispos: That's a totally different topic. I don't know why you use truerange=false, that will assume 16-bit regardless of int bitdepth. Also converting from RGB to YUV is not simple if we start to take into account range, bitdepth scale, etc. With the latest changes it's even more complicated since 'fulld' is broken. I'm about to upload a bunch of updates so maybe tomorrow I can have a deeper look.

[pinterf]

I get invalid arguments with the next call, still on test22:
a=last
b=propClearAll()

scriptclip(b, function [a,b] () {
propcopy(b,a,"_Matrix")
} )

"Invalid arguments to function 'propcopy'"
Maybe I missed something, sometimes it works for me others doesn't. 'a' has the attribute, also tested with custom properties.

@gispos: That's a totally different topic. I don't know why you use truerange=false, that will assume 16-bit regardless of int bitdepth. Also converting from RGB to YUV is not simple if we start to take into account range, bitdepth scale, etc. With the latest changes it's even more complicated since 'fulld' is broken. I'm about to upload a bunch of updates so maybe tomorrow I can have a deeper look.
There is no selective property copy.
You can copy (clone) all, or merge them.

[pinterf]

Thank you Ferenc but unfortunately no success.

When looking at the code I thought that it could work with it, especially the non Y section seemed logical to me.
But very often 0,0,0 is returned, or values only within the byte range.
I looked at other relevant parts and didn't notice anything that could change the values to byte.


Probably you made it right, seeing your last results.

Anyway, I refreshed my memories about AvsPMod development, reinstalled everything, so here is my working version. (I see the project is still on 2.7, Microsoft has silently removed their VcPython27 compiler, you can find only on some peoples' github repo.)

The code was try-except-pass guarded so it hid all internal errors e.g. using things which 2.7 did not know about.


import struct

def GetPixelYUV(self, x, y):
if self.bits_per_component > 8:
if self.bits_per_component == 32:
x = x * 4 # 32 bit float
else:
x = x * 2 # 10-16 bits
# if a resize filter used in the preview filter. CRASH if not check here
if self.DisplayWidth != self.Width or self.DisplayHeight != self.Height:
return (-1,-1,-1)
if self.IsPlanar:
indexY = x + y * self.pitch
if self.IsY8:
return (self.ptrY[indexY], -1, -1)
x = x >> self.WidthSubsampling
y = y >> self.HeightSubsampling
indexU = indexV = x + y * self.pitchUV
elif self.IsYUY2:
indexY = (x*2) + y * self.pitch
indexU = 4*(x/2) + 1 + y * self.pitch
indexV = 4*(x/2) + 3 + y * self.pitch
else:
return (-1,-1,-1)
if self.bits_per_component == 8:
return (self.ptrY[indexY], self.ptrU[indexU], self.ptrV[indexV])
if self.bits_per_component <= 16:
# struct.unpack needs import struct, and returns a single element tuple
# =H: unsigned short (2 bytes), native byte order
bufferY = [self.ptrY[indexY], self.ptrY[indexY + 1]]
valY = struct.unpack('=H', bytearray(bufferY))[0]
bufferU = [self.ptrU[indexU], self.ptrU[indexU + 1]]
valU = struct.unpack('=H', bytearray(bufferU))[0]
bufferV = [self.ptrV[indexV], self.ptrV[indexV + 1]]
valV = struct.unpack('=H', bytearray(bufferV))[0]
return (valY, valU, valV)
#float # =f: float (4 bytes), native byte order
bufferY = [self.ptrY[indexY], self.ptrY[indexY + 1], self.ptrY[indexY + 2], self.ptrY[indexY + 3]]
valY = struct.unpack('=f', bytearray(bufferY))[0]
bufferU = [self.ptrU[indexU], self.ptrU[indexU + 1], self.ptrU[indexU + 2], self.ptrU[indexU + 3]]
valU = struct.unpack('=f', bytearray(bufferU))[0]
bufferV = [self.ptrV[indexV], self.ptrV[indexV + 1], self.ptrV[indexV + 2], self.ptrV[indexV + 3]]
valV = struct.unpack('=f', bytearray(bufferV))[0]
return (valY, valU, valV)

And the calling/test part in avsp.py

avsYUV = script.AVI.GetPixelYUV(x, y)
if avsYUV != (-1,-1,-1):
Y,U,V = avsYUV
cY = ''
if script.AVI.bits_per_component == 8:
hexcolor = '$%02x%02x%02x' % (Y,U,V)
elif script.AVI.bits_per_component <= 16:
hexcolor = '$%04x,%04x,%04x' % (Y,U,V) # comma separated is more visible
else: # 32 bit
hexcolor = '%.5f,%.5f,%.5f' % (Y,U,V) # no reason for 32 bit float in hex

[pinterf]

@gispos: That's a totally different topic. I don't know why you use truerange=false, that will assume 16-bit regardless of int bitdepth. Also converting from RGB to YUV is not simple if we start to take into account range, bitdepth scale, etc. With the latest changes it's even more complicated since 'fulld' is broken. I'm about to upload a bunch of updates so maybe tomorrow I can have a deeper look.
Yes, please stop using truerange, I didn't even know what is still there (there was a huge ConvertBits refactor in the source last week) and someone is still using it. Probably it was a workaround in the very early HBD era, when some filters stored 10 bit data in a stacked-16-bit container and someone asked me to solve the problem (?) I was about to remove it, but at least I'm gonna give error when it is used other than the default value.

Q2:
How is 'fulld' broken?

Q3: Are you implementing another conversion engine besides z_convertformat, fmtconv?

[DTL]

" _ColorRange constants:"

The most strange why only color range. It typically data range including luma too and the luma errors may be most visible. Expected _Range or _DataRange.

Also at time of addition different metadata (moving pictures object properties) it is also good to add spectrum properties:

(for each plane separately or for luma and chroma separately in YUV, equal for planes of RGB formats) -
1. spectrum shape - rectangular (unconditioned), anti-gibbs conditioned.
2. target (preffered) scaling domain - linear, transfer-function converted.
3. type of chroma anti-gibbs filter for final conversion to 4:4:4 - (1,2,3,...).

For each property additional state 'undefined' if no information available.

[pinterf]

What is strange? This is only the beginning.
There are 'standardized' properties and values which are used since years in VapourSynth, zimg and lately in many Avisynth plugins.
I'm not gonna introduce or in-house-standardize brand new features or properties, I'm just trying to catch the train and would like to implement what exists presently before the end of this year.
Keeping conventions helps developers and plugin porting between vs and avs as well.

If there is no constant, then the "no information available" means that the given property does not exist.

[Dogway]

Q2:
How is 'fulld' broken?
Now I got your attention : P 'fulld' requires same value than 'fulls' so for example can't convert from 14-bit bitshift scale to 16-bit full scale. fmtc_bitdepth() allows different values, albeit linked to the color range (which I don't see the relation with, but don't want another argument).
Q3: Are you implementing another conversion engine besides z_convertformat, fmtconv?
I started TransformsPack (https://github.com/Dogway/Avisynth-Scripts/blob/master/TransformsPack.v1.0.RC22.avsi)(big update coming tomorrow) in May before ExTools, color is my area of interest but didn't like the current solutions for avisynth; terms, limitations, accesibility... It's also an expansion of my old LinearResize() so I integrated more resizers, scaling in different "spaces" like gamma, log or sigmoid, added different color models (reversible YUV, reversible YCoCg, OPP, YcCbcCrc, IPT, OkLab, HSV, Lab, Luv, Duv, etc) and plan to add a preset system, full ACES model (including the RRT) and gamut compression. Later (2.0 onwards) for HDR color models and transfer functions if I didn't lose my head already. I would like to see avisynth back being competent in a production environment. Watched a lecture a few months ago of Netflix' encoder David Ronca talking about how he used avisynth in the company's early days, things like that.

[DTL]

Standard naming in ITU-R BT.2100 table 9: https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.2100-2-201807-I!!PDF-E.pdf

Table 9 describes two different signal representations, “narrow” and “full”. The narrow range representation is in widespread use and is considered the default. (page 9)

Table 9:
Parameters: Quantization of R', G', B', Y', I (resulting values that exceed the video data range should be clipped to the video data range) - Narrow range , Full range .
Quantization of C'B, C'R, CT, CP (resulting values that exceed the video data range should be clipped to the video data range) - Narrow range , Full range .

So 'ITU-R standard' naming of essence is 'video data range' and its values 'narrow' and 'full'.

The word 'color' may cause error thinking it is only for colour data like C'B, C'R, CT, CP and for luma data separate property required.

It looks other programs like fmtc try to solve the issue using 'not-full' value. Do not call it 'limited' or 'narrow'. Just 'full' or 'not-full'.

In the 'broadcasting world' there was no need to name this essence before BT.2100 introducing 'full' for DolbyVision PQ. All 'pro broadcast' range was 'narrow' only.

[Dogway]

I also saw it as 'SMPTE range', whether 'legal' or not. Sounds more technical. Some read here (https://nick-shaw.github.io/cinematiccolor/full-and-legal-ranges.html) by Nick Shaw, a fellow from ACES Central.

[DTL]

Also for colour video data properties the 'Primaries' property required. For colour conversions for example between SDR ('standard' colour gamut) and WCG (and more precise SD colour conversions between PAL and NTSC).

[johnmeyer]

@johnmeyer

I think you may be doing some things wrong. I have 64-bit AviSynth+ installed on my Windows 7 PC and also on a 11 year old MacBook Pro running Windows 7 and both work flawlessly. And if I'm not mistaken there are some users using 64-bit AVS+ even on Windows XP.

With that being said, based on the information you provided it seems you are trying to use some 32-bit plugins on a 64-bit environment. I'm sure I'm using 32-bit plugins, since my main, useful but old, system in WinXP 32-bit, and I just copied over all those plugins.

This will get me through the trip and let me do some work. When I get back, I'll be using my old trusty XP computer that may not be modern, but it works every time (except for browsing which is getting tougher, since no XP-compatible browser supports modern scripts, etc.). Even the latest Firefox spinoff is no longer being developed.

[FranceBB]

(except for browsing which is getting tougher, since no XP-compatible browser supports modern scripts, etc.). Even the latest Firefox spinoff is no longer being developed.

Updated XP Compatible Browser from the Windows XP Forever community I'm part of in MSFN ;)

http://rtfreesoft.blogspot.com/search/label/browser

[pinterf]

Also for colour video data properties the 'Primaries' property required. For colour conversions for example between SDR ('standard' colour gamut) and WCG (and more precise SD colour conversions between PAL and NTSC).
As for _Primaries and _Transfer: they are well known and used. I'm on halfway with frameprops integration, I'm not there at the moment. I see how they are used in z_ConvertFormat but in Avisynth core they have zero history.

[pinterf]

Fresh daily stuff (still not on git, your feedback and cleanup needed)
Avisynth+ 3.7.1 test build 24 (20211104) (https://drive.google.com/uc?export=download&id=1LtBvg0gR6KLtdwzO1lquZvLzszL0_sFS)

20211104 WIP
------------
- frame property support: _ChromaLocation in various filters (e.g. ConvertToYUV422)
New location parameter values: "top", "bottom_left", "bottom", "auto"
"ChromaInLocation" rules:
- if source has _ChromaLocation frame property it will be used else the default is "mpeg2" ("left")
- if parameter is "auto" or not given at all, ChromaInLocation will be set to the above mentioned default value
- if parameter is explicitely given, it will be used
"ChromaOutLocation" rules:
- default is "mpeg2" ("left")
- if parameter is "auto" or not given at all, ChromaOutLocation will be set to the above mentioned default value
- if parameter is explicitely given, it will be used

Accepted values for "ChromaInLocation" and "ChromaOutLocation" (when source/target is a chroma subsampled format)
(full list):
- "left" or "mpeg2"
- "center" or "jpeg" or "mpeg1"
- "top_left"
- "dv"
- "top"
- "bottom_left"
- "bottom"

_ChromaLocation constants - as seen in propShow()

AVS_CHROMA_LEFT = 0
AVS_CHROMA_CENTER = 1
AVS_CHROMA_TOP_LEFT = 2 (4:2:0 only)
AVS_CHROMA_TOP = 3 (4:2:0 only)
AVS_CHROMA_BOTTOM_LEFT = 4 (4:2:0 only)
AVS_CHROMA_BOTTOM = 5 (4:2:0 only)
AVS_CHROMA_DV = 6 Special to Avisynth

_ChromaLocation property will be cleared when the result clip is not a chroma subsampled format (4:4:4 or RGB)

[gispos]

Probably you made it right, seeing your last results.

Anyway, I refreshed my memories about AvsPMod development, reinstalled everything, so here is my working version. (I see the project is still on 2.7, Microsoft has silently removed their VcPython27 compiler, you can find only on some peoples' github repo.)

The code was try-except-pass guarded so it hid all internal errors e.g. using things which 2.7 did not know about.


Excellent!
Thank you for the effort.
I hardly dare to ask:), what about RGB.

[pinterf]

Excellent!
Thank you for the effort.
I hardly dare to ask:), what about RGB.
Planar rgb is the very same logic.
Then there remains rgb48 and 64 which follows the two-byte logic as well. But we pisition with X*8 (rgb64) and x*6 (rgb48) instead of x*4 and x*3 then access bgr(a) with +0 +2 +4 +6 instead of +1 +2 +3 inside the 3 or 4 pixel blocks

[gispos]

Planar rgb is the very same logic.
Then there remains rgb48 and 64 which follows the two-byte logic as well. But we pisition with X*8 (rgb64) and x*6 (rgb48) instead of x*4 and x*3 then access bgr(a) with +0 +2 +4 +6 instead of +1 +2 +3 inside the 3 or 4 pixel blocks
Ok, thanks, 16bit seems to be returning the correct values.

bytes = self.vi.bytes_from_pixels(1)
if self.bits_per_component == 16:
if BGR:
indexB = (x * bytes) + (self.Height - 1 - y) * self.pitch
bufferB = [self.ptrY[indexB], self.ptrY[indexB + 1]]
valB = struct.unpack('=H', bytearray(bufferB))[0]
bufferG = [self.ptrY[indexB+1], self.ptrY[indexB + 2]]
valG = struct.unpack('=H', bytearray(bufferG))[0]
bufferR = [self.ptrY[indexB+2], self.ptrY[indexB + 3]]
valR = struct.unpack('=H', bytearray(bufferR))[0]
return (valR, valG, valB)
else:
indexR = (x * bytes) + y * self.pitch
bufferR = [self.ptrY[indexR], self.ptrY[indexR + 1]]
valR = struct.unpack('=H', bytearray(bufferR))[0]
bufferG = [self.ptrY[indexR+1], self.ptrY[indexR + 2]]
valG = struct.unpack('=H', bytearray(bufferG))[0]
bufferB = [self.ptrY[indexR+2], self.ptrY[indexR + 3]]
valB = struct.unpack('=H', bytearray(bufferB))[0]
return (valR, valG, valB)


Edit: It's not right, I have to sleep on it first.

[pinterf]

IndexB+2 indexB+3
indexB+4 indexB+5
The 2nd and 3rd pixel

[Dogway]

Is there a way for the next to don't crash? maybe something along RequestLinear(), Prefetch, optSingleMode or some trick?
for (i=1,48,1) {
Expr("x[0,1] x[-1,0] min x[1,0] min x[0,-1] min","")
}

[pinterf]

Is there a way for the next to don't crash? maybe something along RequestLinear(), Prefetch, optSingleMode or some trick?
for (i=1,48,1) {
Expr("x[0,1] x[-1,0] min x[1,0] min x[0,-1] min","")
}

"Stack overflow". Interesting.
EDIT: stopped using local variables for large arrays, dynamic std::vector is better for them. Many ~1-2kbytes per Expr on stack chained for 48 Expr calls consumed available stack area.
Fix appears in next build.

[Dogway]

Thanks for the update pinterf!

I'm giving the last touches for the latest update of TransformsPack that is setting all the frameprops values and let me tell you (rant coming), what a disaster!
So for primaries or matrix constants they are mixing color models with color spaces, going to the extent of (check table E.5 of ITU H.265 page 429) defining '1' for xvYCC -and a small 709 subscript- and defining '5' for xvYCC -and a small 601 subscript-, rhetorical? Then sYCC (the 's' comes from sRGB) is defined along Rec601 and 470BG, that is '5' again when it should be '1' (check table E.1 at value 1 entry)!

To be honest and this is my opinion, the "matrix" concept is outdated as it tries to fit constants into a model (YCbCr) that is flawed and outdated. So you see 0, 8, 11, 12, 13, 14 (and what is not listed) as 'see equations' because some models don't rely on the YCbCr's Kr,Kb,Kg typical constant system. I think if something should be valuable is to set 'models', where matrix constants are a subset of the YCbCr model which can hold several color spaces (with color primaries).

[gispos]

IndexB+2 indexB+3
indexB+4 indexB+5
The 2nd and 3rd pixel
Hadn't read your posting and just pressed Quote.
So I tried without reading your posting with the two-byte logic... and of course without success. :o
This is how it now works for RGB 48,64 16bit
Thanks again for your help.

If I still have to pay attention to something then out with it.:)

def GetPixelRGB(self, x, y, BGR=True):
if self.IsRGB:
# if a resize filter used in the preview filter. CRASH if not check here
if self.DisplayWidth != self.Width or self.DisplayHeight != self.Height:
return (-1,-1,-1)
bytes = self.vi.bytes_from_pixels(1)

if self.bits_per_component == 16:
if BGR:
indexB = (x * bytes) + (self.Height - 1 - y) * self.pitch
bufferB = [self.ptrY[indexB], self.ptrY[indexB+1]]
valB = struct.unpack('=H', bytearray(bufferB))[0]
bufferG = [self.ptrY[indexB+2], self.ptrY[indexB+3]]
valG = struct.unpack('=H', bytearray(bufferG))[0]
bufferR = [self.ptrY[indexB+4], self.ptrY[indexB+5]]
valR = struct.unpack('=H', bytearray(bufferR))[0]
return (valR, valG, valB)
else:
indexR = (x * bytes) + y * self.pitch
bufferR = [self.ptrY[indexR], self.ptrY[indexR+1]]
valR = struct.unpack('=H', bytearray(bufferR))[0]
bufferG = [self.ptrY[indexR+2], self.ptrY[indexR+3]]
valG = struct.unpack('=H', bytearray(bufferG))[0]
bufferB = [self.ptrY[indexR+4], self.ptrY[indexR+5]]
valB = struct.unpack('=H', bytearray(bufferB))[0]
return (valR, valG, valB)

if self.bits_per_component > 8:
return (-1,-1,-1)

if BGR:
indexB = (x * bytes) + (self.Height - 1 - y) * self.pitch
indexG = indexB + 1
indexR = indexB + 2
else:
indexR = (x * bytes) + y * self.pitch
indexG = indexR + 1
indexB = indexR + 2
return (self.ptrY[indexR], self.ptrY[indexG], self.ptrY[indexB])
else:
return (-1,-1,-1)


Edit:
Is there an alpha channel even with 16 bit?
Is there a function like IsRGBA for testing?

Edit2: found
HasAlpha and IsPlanarRGBA or num_components

[pinterf]

Happy Friday!
Avisynth+ 3.7.1 test build 25 (20211105) (https://drive.google.com/uc?export=download&id=1xQGN9i3ecJqzwZTfDFNCQ_F1efZfekc-)

20211105 WIP
------------
- ConvertBits: allow dither from 32 bits to 8-16 bits (through an internal 16 bit immediate clip)
- ConvertBits: allow different fulls fulld when converting between integer bit depths
- ConvertBits: allow 32 bit to 32 bit conversion

ColorbarsHD()
# another method for converting to full range
ConvertToRGB(matrix="709:l")
ConvertToYUV444(matrix="709:f")
# 8 to 32 bits
ConvertBits(32, fulls=true, fulld=false)
ConvertBits(32, fulld=true) # fulls=false: auto from frame prop _ColorRange
ConvertBits(8, fulld=false, dither = 1, dither_bits=1) # low dither_bits just for fun :)
Histogram("levels")

- Expr: consume less bytes on stack. 48x Expr call in sequence caused stack overflow

I've just found again this dither = 1, dither_bits=1 (2, 3, ...) option, which I made for curiousity. Apply to your favorite video clip and forget this high-bit-depth hype :)

[Dogway]

Awesome! I can barely keep up with all the changes.
The dither_bits can come useful for posterization which I have some ideas for.

[FranceBB]

Awesome, Ferenc!
Have a lovely thoughts-free/stress-free weekend, you deserve it! :D

[cretindesalpes]

I've just found again this dither = 1, dither_bits=1 (2, 3, ...) option, which I made for curiousity. Apply to your favorite video clip and forget this high-bit-depth hype :)

Funnily a few days ago I found a formula to fix the EOTF effect (gamma thing) when dithering with a low bitdepth:

# Gamma-aware color quantization
# https://www.desmos.com/calculator/qyyjugcbbt

b = 2
q = Int (Pow (2, b))

source ()
ref = ConvertBits (8, fulls=true, fulld=true)

x = " x 0.000001 max "
f = " 2.2 ^ " # EOTF transfer function (expects input value on the stack)
flr = " 0.5 - round "
u = x + String (q) + " * " + flr + String (q) + " / "
v = x + String (q) + " * " + flr + " 1 + " + String (q) + " / "
remap = x + f + u + f + " - " + v + f + u + f + " - / " + v + u + " - * " + u + " + "
fixed = mt_lut (remap, u=3, v=3)

Interleave (last, fixed)
ConvertBits(8, fulls=true, fulld=true, dither=1, dither_bits=b)

StackVertical (SelectEvery (2, 0), SelectEvery (2, 1), ref)

Function source ()
{
BlankClip (pixel_type="RGB24", width=1280, height=64)
ConvertToPlanarRGB ().ConvertToFloat ()
mt_lutspa (mode="relative", expr="x", u=3, v=3)
}

https://i.postimg.cc/Y9zQ2wBf/dithering-gamma.png

[wonkey_monkey]

I'm writing a plugin and accidently did the following:

env->AddFunction("TrackingReduce", "c[bool]translate[bool]scale[bool]rotate", Create_TrackingReduce, 0);

The parameter string is completely the wrong format. Instead of getting a useful error, this instead causes a System Violation, but only if you try to use any of the functions of the plugin in question. Otherwise it's silent (initialisation of that plugin DLL just aborts).

Obviously it was my silly error, but is it worth considering some validation for parameter strings that could throw a warning instead of leading to a somewhat confusing System Violation?

[StainlessS]

In my experience [I think], many errors in parameter string produce error exception,
if problem in calling constructor [ie before frameserving starts], then good idea to check params string [first].

however, this error did not seem to produce any problems:- https://forum.doom9.org/showthread.php?p=1954448#post1954448
And Plugins [just a few in Plugins dir] : NOTE, it actually found a bug in ApparentFPS params list (top line, used '[' instead of closing ']', does not seem to cause problems though)
EDIT: Actually 46 plugins auto loaded, only 2 of which cause errors (AssumeFPS, Grunt).

00001704 1.30285561 RT_DebugF: ApparentFPS "c[DupeThresh]f[FrameRate]f[Samples]i[ChromaWeight]f[Prefix]s[Show]b[Verbose]b[Debug]b[Mode]i[Matrix]i[BlkW]i[BlkH[i[oLapX]i[oLapY]i"

[Dogway]

pinterf, I noticed a few inconsistencies with color related frame props.

-Matrix string Rec601 in test23 defaults to ID 5 (470BG), that's PAL Rec601 which is much less common than NTSC Rec601 (170M).
-FCC corresponds to 470M in ITU to keep the naming convention (170M, 240M, 470BG)
-YCgCo seems to be also much less used than YCoCg (86K versus 16K in Google search), YCoCg is also the preferred term in the Wikipedia and in the Colour python module.
-Also noticed that Rec2020 and 2020 are supported as 2020CL and 2020NCL, but not Rec2020CL or Rec2020NCL.
-I couldn't use the "auto" matrix type. From YCbCr, ConvertToPlanarRGB("auto:auto") or ConvertToPlanarRGB("auto") triggers an "Unknown matrix" error.
-Quote (https://forum.doom9.org/showthread.php?p=1956500#post1956500): "When converting to RGB the _Matrix parameter is set to 0 ("rgb")". Yet for internal RGB clips _Matrix is set to 1(?)
- BlankClip: frame property support:
RGB: _ColorRange = 0 ("full"), _Matrix = 1 ("709")

-Also just tested the new dither_bits arg, too bad it doesn't work with dither=-1 for posterization, not sure if it uses the same code path as it can be very useful.

-Found another issue, how do you correct the bitdepth scale of source without changing the range? Example, load a sample image which by nature is PC range (ie with JPEGSource which has a great builtin deblocker), render the YPlaneMax, it reads 65280. How do we fix this with internal filters? Currently it's only possible with Expr("x 257 256 / *")

[vcmohan]

I am testing for speed my plugin functions for avs+ using run video analysis on vdub. I am using vseditor benchmark for testing my vapoursynth functions. I find that speed of functions on avs+ are about a third of what they are for vapoursynth. In my script I am not using filter MT_NICE_FILTER and prefetch commands as I presume it will use the type declared in the function itself. I have specified in cacheHints MT_NICE_FILTER.
On checking performance of computer I find for avs+ about 25% usage, while for vapoursynth I get 90+% usage. Looks only one cpu is being used out of 4 on my Intel i5 11th gen computer with windows 10.
Is this a problem in scripting or vdub video analysis?
If this question was already dealt with, I am sorry to have posted this, but would like to get a link to it.

[VoodooFX]

Is this a problem in scripting or vdub video analysis?

Try benchmarking with AVSMeter (https://forum.doom9.org/showthread.php?t=174797).

[gispos]

Hello Ferenc, can you please look over there again. With 10 to 16 bit color depth, the correct values seem to be returned only with YUV444.
And please take another look at 32bit.
Thanks in advance.

Edit:
The fact that only 0 is displayed at 32bit is my fault (everything is formatted as an integer), but I also get negative values after float formatting.

Probably you made it right, seeing your last results.

Anyway, I refreshed my memories about AvsPMod development, reinstalled everything, so here is my working version. (I see the project is still on 2.7, Microsoft has silently removed their VcPython27 compiler, you can find only on some peoples' github repo.)

The code was try-except-pass guarded so it hid all internal errors e.g. using things which 2.7 did not know about.


import struct

def GetPixelYUV(self, x, y):
if self.bits_per_component > 8:
if self.bits_per_component == 32:
x = x * 4 # 32 bit float
else:
x = x * 2 # 10-16 bits
# if a resize filter used in the preview filter. CRASH if not check here
if self.DisplayWidth != self.Width or self.DisplayHeight != self.Height:
return (-1,-1,-1)
if self.IsPlanar:
indexY = x + y * self.pitch
if self.IsY8:
return (self.ptrY[indexY], -1, -1)
x = x >> self.WidthSubsampling
y = y >> self.HeightSubsampling
indexU = indexV = x + y * self.pitchUV
elif self.IsYUY2:
indexY = (x*2) + y * self.pitch
indexU = 4*(x/2) + 1 + y * self.pitch
indexV = 4*(x/2) + 3 + y * self.pitch
else:
return (-1,-1,-1)
if self.bits_per_component == 8:
return (self.ptrY[indexY], self.ptrU[indexU], self.ptrV[indexV])
if self.bits_per_component <= 16:
# struct.unpack needs import struct, and returns a single element tuple
# =H: unsigned short (2 bytes), native byte order
bufferY = [self.ptrY[indexY], self.ptrY[indexY + 1]]
valY = struct.unpack('=H', bytearray(bufferY))[0]
bufferU = [self.ptrU[indexU], self.ptrU[indexU + 1]]
valU = struct.unpack('=H', bytearray(bufferU))[0]
bufferV = [self.ptrV[indexV], self.ptrV[indexV + 1]]
valV = struct.unpack('=H', bytearray(bufferV))[0]
return (valY, valU, valV)
#float # =f: float (4 bytes), native byte order
bufferY = [self.ptrY[indexY], self.ptrY[indexY + 1], self.ptrY[indexY + 2], self.ptrY[indexY + 3]]
valY = struct.unpack('=f', bytearray(bufferY))[0]
bufferU = [self.ptrU[indexU], self.ptrU[indexU + 1], self.ptrU[indexU + 2], self.ptrU[indexU + 3]]
valU = struct.unpack('=f', bytearray(bufferU))[0]
bufferV = [self.ptrV[indexV], self.ptrV[indexV + 1], self.ptrV[indexV + 2], self.ptrV[indexV + 3]]
valV = struct.unpack('=f', bytearray(bufferV))[0]
return (valY, valU, valV)

And the calling/test part in avsp.py

avsYUV = script.AVI.GetPixelYUV(x, y)
if avsYUV != (-1,-1,-1):
Y,U,V = avsYUV
cY = ''
if script.AVI.bits_per_component == 8:
hexcolor = '$%02x%02x%02x' % (Y,U,V)
elif script.AVI.bits_per_component <= 16:
hexcolor = '$%04x,%04x,%04x' % (Y,U,V) # comma separated is more visible
else: # 32 bit
hexcolor = '%.5f,%.5f,%.5f' % (Y,U,V) # no reason for 32 bit float in hex

[johnmeyer]

Updated XP Compatible Browser from the Windows XP Forever community I'm part of in MSFN ;)

http://rtfreesoft.blogspot.com/search/label/browserI'm late in replying, but thank you for that link. Very useful.

[pinterf]

pinterf, I noticed a few inconsistencies with color related frame props.

-Matrix string Rec601 in test23 defaults to ID 5 (470BG), that's PAL Rec601 which is much less common than NTSC Rec601 (170M).
-FCC corresponds to 470M in ITU to keep the naming convention (170M, 240M, 470BG)
-YCgCo seems to be also much less used than YCoCg (86K versus 16K in Google search), YCoCg is also the preferred term in the Wikipedia and in the Colour python module.
-Also noticed that Rec2020 and 2020 are supported as 2020CL and 2020NCL, but not Rec2020CL or Rec2020NCL.
-I couldn't use the "auto" matrix type. From YCbCr, ConvertToPlanarRGB("auto:auto") or ConvertToPlanarRGB("auto") triggers an "Unknown matrix" error.
-Quote (https://forum.doom9.org/showthread.php?p=1956500#post1956500): "When converting to RGB the _Matrix parameter is set to 0 ("rgb")". Yet for internal RGB clips _Matrix is set to 1(?)


-Also just tested the new dither_bits arg, too bad it doesn't work with dither=-1 for posterization, not sure if it uses the same code path as it can be very useful.

-Found another issue, how do you correct the bitdepth scale of source without changing the range? Example, load a sample image which by nature is PC range (ie with JPEGSource which has a great builtin deblocker), render the YPlaneMax, it reads 65280. How do we fix this with internal filters? Currently it's only possible with Expr("x 257 256 / *")
Thanks for the feedback, I'm looking into them.

[pinterf]

I am testing for speed my plugin functions for avs+ using run video analysis on vdub. I am using vseditor benchmark for testing my vapoursynth functions. I find that speed of functions on avs+ are about a third of what they are for vapoursynth. In my script I am not using filter MT_NICE_FILTER and prefetch commands as I presume it will use the type declared in the function itself. I have specified in cacheHints MT_NICE_FILTER.

You have to set Prefetch(4) (for example) manually, usually at the end of the script. Or else the whole script runs is single threaded. When a plugin does not specify the default is MT_MULTI_INSTANCE. Even if your filter is totally reentrant and does not have internal states so it is a nice filter, you could test whether MT_MULTI_INSTANCE is better or not speedwise.

[pinterf]

Hello Ferenc, can you please look over there again. With 10 to 16 bit color depth, the correct values seem to be returned only with YUV444.
And please take another look at 32bit.
Thanks in advance.

Edit:
The fact that only 0 is displayed at 32bit is my fault (everything is formatted as an integer), but I also get negative values after float formatting.
Full float chroma range is -0.5 .. 0.5 so negative values are normal.

EDIT: Ohh, my bad, did not try on a subsampled clip.

- Fix offset for chroma subsampling
- Fix 10+ bit greyscale
In class AvsClipBase you must uncomment two fields:

self.num_components = None # PF 20211109 go live
self.component_size = None # PF 20211109 go live


def GetPixelYUV(self, x, y):
if self.bits_per_component == 8:
component_size = 1;
elif self.bits_per_component == 32:
component_size = 4 # 32 bit float
else:
component_size = 2 # 10-16 bits
# if a resize filter used in the preview filter. CRASH if not check here
if self.DisplayWidth != self.Width or self.DisplayHeight != self.Height:
return (-1,-1,-1)
if self.IsPlanar:
indexY = x * component_size + y * self.pitch
# IsY8 does not detect Y10..Y16,Y32
# Probably IsY is not implemented, so we use num_components
if self.num_components == 1:
if self.bits_per_component == 8:
return (self.ptrY[indexY], -1, -1)
elif self.bits_per_component <= 16:
bufferY = [self.ptrY[indexY], self.ptrY[indexY + 1]]
valY = struct.unpack('=H', bytearray(bufferY))[0]
return (valY, -1, -1)
else:
bufferY = [self.ptrY[indexY], self.ptrY[indexY + 1], self.ptrY[indexY + 2], self.ptrY[indexY + 3]]
valY = struct.unpack('=f', bytearray(bufferY))[0]
return (valY, -1, -1)
x = x >> self.WidthSubsampling
y = y >> self.HeightSubsampling
indexU = indexV = x * component_size + y * self.pitchUV
elif self.IsYUY2:
indexY = (x*2) + y * self.pitch
... rest is the same

[pinterf]

pinterf, I noticed a few inconsistencies with color related frame props.
Thank you.
As a reference, I'm using z_ConvertFormat.
Avsresize wiki: http://avisynth.nl/index.php/Avsresize
VapourSynth doc: http://www.vapoursynth.com/doc/functions/video/resize.html

But I think I must check VapourSynth resizer behaviour as well, they have a bit longer history.


-Matrix string Rec601 in test23 defaults to ID 5 (470BG), that's PAL Rec601 which is much less common than NTSC Rec601 (170M).
testing z/avs parallel:
ColorBars(pixel_type="RGBP8")
clip1=z_convertformat(pixel_type="yuv420p8").SubTitle("z_conv",y=120)
clip2=ConvertToYUV420().SubTitle("Avs",y=120)
Interleave(clip1, clip2)
propShow()

z_ConvertFormat defaults _Matrix=6 (170M) while I implemented 5 (470BG)
I like when there is a consent, so I'm gonna change it to 6.


FCC corresponds to 470M in ITU to keep the naming convention (170M, 240M, 470BG)

Both avsresize and VapourSynth is using "fcc". The "470m" appears only in transfer characteristic and color primaries.


-YCgCo seems to be also much less used than YCoCg (86K versus 16K in Google search), YCoCg is also the preferred term in the Wikipedia and in the Colour python module.

The base examples are using YCgCo terminology, so I kept it.
(in my opinion when we - here at doom9 or at github - agree on a change then it would be welcomed if everybody change/extend it in their plugin/system.)


-Also noticed that Rec2020 and 2020 are supported as 2020CL and 2020NCL, but not Rec2020CL or Rec2020NCL.

Old matrix constants are kept, but there is no a "rec" or PC" prefix one for every new one.
The preferred strings are the new one, which do not implicitely define full/limited range, but has to specify it after them 2020:f or 2020:full or 2020ncl:l or 2020ncl::limited.
As avsresize wiki says, "2020" is a compatibility alias for "2020ncl", I have implemented the very same convention.


-I couldn't use the "auto" matrix type. From YCbCr, ConvertToPlanarRGB("auto:auto") or ConvertToPlanarRGB("auto") triggers an "Unknown matrix" error.

Thanks, I'm gonna check it.


-Quote (https://forum.doom9.org/showthread.php?p=1956500#post1956500): "When converting to RGB the _Matrix parameter is set to 0 ("rgb")". Yet for internal RGB clips _Matrix is set to 1(?)

What is "internal RGB clips"? Value of "1" is surely wrong for them.


-Also just tested the new dither_bits arg, too bad it doesn't work with dither=-1 for posterization, not sure if it uses the same code path as it can be very useful.

Yes, this dither code has limits.


-Found another issue, how do you correct the bitdepth scale of source without changing the range? Example, load a sample image which by nature is PC range (ie with JPEGSource which has a great builtin deblocker), render the YPlaneMax, it reads 65280. How do we fix this with internal filters? Currently it's only possible with Expr("x 257 256 / *")
JPEGSource return 65280 which is FF00, lower byte is simply zero. This is not a range-question, rather than how JPEGsource is returning you this 16 bit data. (If I understand correctly)

[Dogway]

FCC is the very old term before ITU standardized it, it's like calling 470BG as EBU Tech. I don't think avsresize should be the golden book for reference on color matters, the colour (https://github.com/colour-science/colour)module or other color science projects are more reliable if you don't take my word for it. But I don't have anything against if still you decide to keep it.

Thanks also for the standard matrix naming convention, I will adhere to them.


What is "internal RGB clips"? Value of "1" is surely wrong for them.
I quoted your example in the test23 release post (https://forum.doom9.org/showthread.php?p=1956500#post1956500)where you post BlankClip in RGB with _Matrix set to 1

JPEGSource return 65280 which is FF00, lower byte is simply zero. This is not a range-question, rather than how JPEGsource is returning you this 16 bit data. (If I understand correctly)
Yes, it can be seen in two ways, a source loader flaw or a lack of options in avs+. I will try to check what loaders fail, unfortunately I think JPEGSource is closed source.

[pinterf]

FCC is the very old term before ITU standardized it, it's like calling 470BG as EBU Tech. I don't think avsresize should be the golden book for reference on color matters, the colour (https://github.com/colour-science/colour)module or other color science projects are more reliable if you don't take my word for it. But I don't have anything against if still you decide to keep it.

VapourSynth and avsresize both are based on zimg library, and since I'm much less familiar in this area than the creators I rely on them, and learn and read.


I quoted your example in the test23 release post (https://forum.doom9.org/showthread.php?p=1956500#post1956500)where you post BlankClip in RGB with _Matrix set to 1

Ah, I see, it is a typo in docs. Fixed it in original post as well.

EDIT:
FCC is the very old term before ITU standardized it, it's like calling 470BG as EBU Tech. I don't think avsresize should be the golden book for reference on color matters, the colour module or other color science projects are more reliable if you don't take my word for it. But I don't have anything against if still you decide to keep it.

Golden standard because their official constants (matrix, primaries, transfer, chroma location) are from ITU-T H.265, download from here: https://www.itu.int/rec/T-REC-H.265-202108-I
See Table E.5, at value 4 I cannot see any better hint why we not to keep there "fcc". (Unlike transfer and primary)

[Dogway]

I see, well I won't be taking the zimg route but colour science reference and terms. I have all ITU papers I could find and more but for example the linked Colour repo disregards FCC in favour of 470-525 also called 470M by ITU which is an international standard while FCC is an USA only committe.
I already ranted before of inconsistencies on those papers and many color scientists seem to agree for a reason.

On another note related to image loading:
What is the recommended procedure to convert an image (usually PC range) into HBD?

Expr("255","128") # PC range YUV image
ConvertBits(16,fulls=false,fulld=false) # Now we have PC range but with bitshift scale and _ColorRange of 1 (limited?)
or
ConvertBits(16,fulls=false,fulld=true) # it does a TV range to PC range conversion

Another option is true,true, but then look at this:
Expr("255","128")
ConvertBits(16,fulls=true,fulld=true)
ConvertBits(16,fulls=true,fulld=false)
scriptclip("subtitle(string(UPlaneMax),x=30,y=10)") # output is 32880 instead of 32768

[pinterf]

Pc range = full range. Both fulls and fulld is true. Like rgb

[qyot27]

For consistency's sake, using the same names that FFmpeg uses internally makes sense, considering that the two main general-purpose source filters that would be setting frame properties both use FFmpeg's libraries, and that it would also be the thing most likely to be used to playback/encode from a script with those properties set.

[pinterf]

Another option is true,true, but then look at this:
Expr("255","128")
ConvertBits(16,fulls=true,fulld=true)
ConvertBits(16,fulls=true,fulld=false)
scriptclip("subtitle(string(UPlaneMax),x=30,y=10)") # output is 32880 instead of 32768

But this works:
Expr("255","128")
ConvertBits(32,fulls=true,fulld=true) # 8->32 bits: OK
ConvertBits(16,fulls=true,fulld=false) # 32->16 bits: OK: 32768

This is because when converting to and from 32 bit float the chroma channel is handled specially because of the 0 chroma center at 32 bit float formats.

While in your example
ConvertBits(16,fulls=true,fulld=true)
the internal Avisynth code has no special chroma handling case. The range 0-255 is multiplied by 65535/255 (=257) to stretch the range. :(

It is not correct, because 128 is not the real center of the 0-255 range. Cb Cr is a signed quantity and it is only a technical thing that they are biased and stuffed into an unsigned byte (or 16 bit word) 16-240 is -112..+112 in reality, and its full range equivalent must be -127..+127.

I was thinking earlier that a correct method would be good to implement.

Integer-to-Integer full-full range originated chroma conversion must first subtract 128 (8 bit case) from source, then upscale by a factor then add the new bitdepth's center.

U8 to U16: (U-128) / 127 * 32767 + 32768
1 -> 1
128 -> 32768 (center is O.K.!)
255 -> 65535

In general:
half_src = 2^(M-1) where M is src bit depth
half_dest = 2^(N-1) where N is target bit depth
U_dest = (U_src - half_src) / (half_src-1) * (half_target - 1) + half_target

Note: theoretically a full range 8 bit chroma must have in the range of 1-255 (128 center is kept) 0 is invalid value in any bit depth.

Yes. I have to go this way.

[Dogway]

This issue propagates into internal RGB conversion with:
ConvertBits(16,fulls=true,fulld=true)
ConverttoplanarRGB("PC.709")
scriptclip("subtitle(string(GPlaneMax),x=30,y=10)") # outputs 65451

The same happens with z_ConvertFormat and fmtc_matrix. My current (not uploaded yet) ConvertFormat version converts to RGB correctly without this happening in 16-bit (grey gradient ramp is achromatic), but still there are other issues I have to iron out before saying I got a solution, also I did the YUV to RGB implementation months ago so I have to revisit it.

The described formula looks right, I tested over other bitdepths, they don't match my HBD constants so I might have a second look to them.
This is 12-bit:

U=240
(U-128) / 127 * 2047 + 2048
Output: 3853,228

[gispos]

EDIT: Ohh, my bad, did not try on a subsampled clip.

- Fix offset for chroma subsampling
- Fix 10+ bit greyscale

Thanks again, seems to be working.:)

In class AvsClipBase you must uncomment two fields:

IsY, component_size, num_components
It is available in the newer versions and is only set to None during initialization.

Just for my better understanding IsY would then replace num_components?
IsY recognizes all color depths with a single Y component and IsY8 only recognizes 8-bit single Y components?

[pinterf]

I see, well I won't be taking the zimg route but colour science reference and terms. I have all ITU papers I could find and more but for example the linked Colour repo disregards FCC in favour of 470-525 also called 470M by ITU which is an international standard while FCC is an USA only committe.
I already ranted before of inconsistencies on those papers and many color scientists seem to agree for a reason.

You are argueing on how we should call the conversion filter parameter string which would select "Table E.5 Matrix code 4", which has KR = 0.30; KB = 0.11 and we cannot call it neither 470M nor any other because they have different coefficients?

Perhaps let we name it "fcc_title_47_code_2003" instead of simple "fcc" named after its informative remark
"FCC Title 47 Code of Federal Regulations (2003) 73.682 (a) (20)"?

[pinterf]

IsY recognizes all color depths with a single Y component and IsY8 only recognizes 8-bit single Y components?
Yes. I was not sure that IsY was implemented at the Python C interface, this is why I didn't use.

[pinterf]

This issue propagates into internal RGB conversion with:
ConvertBits(16,fulls=true,fulld=true)
ConverttoplanarRGB("PC.709")
scriptclip("subtitle(string(GPlaneMax),x=30,y=10)") # outputs 65451


Yes, and when conversion is OK (going through 32 bit - which is correct), then you get 65535.

ColorbarsHD() # just for YV24 format
Expr("255","128")
ConvertBits(32,fulls=true,fulld=true)
ConvertBits(16,fulls=true,fulld=true)
ConverttoplanarRGB("PC.709")
scriptclip("subtitle(string(GPlaneMax),x=30,y=10)") # outputs 65535 Yeah


I'm gonna do it right for direct 8-16 conversions.

[Dogway]

You are argueing on how we should call the conversion filter parameter string which would select "Table E.5 Matrix code 4", which has KR = 0.30; KB = 0.11 and we cannot call it neither 470M nor any other because they have different coefficients?

Perhaps let we name it "fcc_title_47_code_2003" instead of simple "fcc" named after its informative remark
"FCC Title 47 Code of Federal Regulations (2003) 73.682 (a) (20)"?

All of a sudden ^^ I don't care how you call it, it was an educated suggestion. I will do my own thing.

[pinterf]

All of a sudden ^^ I don't care how you call it, it was an educated suggestion. I will do my own thing.
Dear Dogway, you were right.
I read that pdf for the Nth time, and did not recognize the zillion sub-option letters after BT.470-6. I've changed the Avisynth code accordingly. I'm gonna thank you with a glass of red wine in the evening (Kadarka) :)

[Dogway]

Last recommendation (https://www.itu.int/rec/R-REC-BT.1701-1-200508-I/en) in 2005 (in force). 1701-M anyone? NTSC-M also seems legit.

[pinterf]

Last recommendation (https://www.itu.int/rec/R-REC-BT.1701-1-200508-I/en) in 2005 (in force). 1701-M anyone? NTSC-M also seems legit.
I'm lost and not seeing in this document the relevant info.

Plenty of papers.

However this paper from 2015 (Report ITU-R BT.2380-0 07/2015 Television colorimetry elements) is the most detailed one I found so far.
https://www.itu.int/dms_pub/itu-r/opb/rep/R-REP-BT.2380-2015-PDF-E.pdf

In TABLE 2.8 matrix_coefficient=4 says:

- US NTSC 1953 Recommendation for transmission standards for colour television (only MPEG-2 Video, MPEG-4 Visual, MPEG HEVC)
- US FCC Title 47 Code of Federal Regulations (2004) 73.682 (a) (20) (only MPEG-4/AVC)
- Recommendation ITU-R BT.470-6 system M (only MPEG-H HEVC)

As a shortcut both bt470m, fcc or fcc47 and probably ntsc1953 or ntsc-m would be valid, when we want to hint [0.30, 0.59, 0.11] matrix.
For historical reasons we had "fcc" for that, then now there is a "bt470m".

I'd like to make propShow to write a human-friendly short descriptions when displaying _Matrix (_ColorRange, ...)

Other topic:
I'm undereducated on these areas: primaries and transfer characteristics, O.K., I can look into the sources of zimg and fmtconv.
Is there any good and clean documentation I can learn from or google is my friend?

[Dogway]

ITU-R BT.2380-0 07/2015 seems to focus on digital television (SD and HD) except for the tables where it mentions some analogue formats. The tables repeat in several papers, also in T-REC-H.265-201911 and T-REC-H.273-201612. These are codec specific papers whereas BT.2380 looks like a typical standard recommendation.

The current paper (in effect) for analogue formats is R-REC-BT.1701-1-200508 where you can find the specifications for the different types of PAL and NTSC, which some of them we have to deal with. I don't know why they still refer to R-REC-BT.470-6-199811 in these newer papers. I'm not going to make a fuss about cosmetics calling a matrix x or y. 470 or PAL/NTSC with the hyphen+letter seems the most common, shorter and logic to follow, but x265 still uses 'fcc' some whatever pleases you.

The current digital NTSC matrix/primaries (they call it simply 525) is based on 170M, aka SMPTE-C aka Rec601 in Doom9.


As I understand the difference of the matrix naming for the different codecs is because the codec spec is not updated to latest recommendations, so in AVC it's called what fcc, in MPEG2 NTSC 1953 and in HEVC 470M. We are newer so we can call it 1701M ^^

EDIT: Last year I made a project for legacy formats (where I have some knowledge but barely for newer HDR formats), and besides these recommendations and some random papers and books I found a very valuable one called "Video Demystified" by Keith Jack. You can find excerpts and PDFs on google.

[jpsdr]

@pinterf
This is an "old" version, two updates has been made since.
Last is : https://www.itu.int/pub/R-REP-BT.2380-2-2018

You can read the PDF i've made with my HDRTools, i've tried to explain some things. I don't know if i've succed...

[FranceBB]

I don't know if i've succed...

You have. That document has been translated in Italian by me and has been in our intranet ever since you made it years ago (with your credits of course) eheheheh

[Dogway]

Is the 'neg' operator working in Expr? I tried several combinations without success.

[pinterf]

Well, this build took a longer time to develop.
A _lot_of ConvertBits tweaks (almost a total rewrite in the background), another Dogway wish in Expr, adventures with Intel C++ compiler.

What I enjoyed most was playing with the low dither_bit option both in ordered and in Floyd type of dithering.

Avisynth+ 3.7.1 test build 26 (20211116) (https://drive.google.com/uc?export=download&id=13_UFB4KL_KKFi4pC_G5HBVCRixeEvBqL)

20211116 WIP
------------
- Expr: add "neg": negates stack top: a = -a
- Floyd dither ("dither"=1)
- add native fulls-fulld support, add special chroma handling when full-range = true involved
- valid "dither_bits" parameter 1 to 16 (similar to ordered dither)
- special handling of low (1-7 bits) "dither_bits" => result looks nice, same as at ordered dither
- more optimized to frequently used source and dither target bits differences: 2,4,6 and 8
(covers typical 16->8, 10->8, 16->10 bit conversions; others have ~-10% speed, less than 8 bit targets are -20-25% )
- (fix YV411 to and from conversion - regression since recent chroma placement addition)
- ConvertBits: Support YUY2 (by autoconverting to and from YV16), support YV411
- ConvertBits: "bits" parameter is not compulsory, since the dit depths can stay as it was before. One can call like ConvertBits(fulld=true)
- ConvertBits: "dither" parameter: type changed to integer. Why was it float? :) valid values were 0 and 1
- ConvertBits: source: dither almost full refactor
- ConvertBits: allow dithering down from 8 bit sources (use case: specify parameter "dither_bits" less than 8)
Example: My8bitVideo.ConvertBits(8, fulls=true, fulld=true, dither = 0, dither_bits=1)
- ConvertBits: ordered dither (dither_type=0) new features
- add AVX2
- allow odd dither_bits values, 1-16 bits (was: 2,4,6,8,..). The difference is still maximum 8, so dither_bits=1 is available
only for 8 bit sources. (memo: for Floyd (dither=1) the minimum remained 1, allowed range is 1-16)
- correct conversion of full-range chroma at 8-16 bits, keeping center
- fulls-fulld mix support (conversion - if any - happens before dithering)
- when dither target bitdepth is less than 8, then special measures are taken in order to show 'nice' output;
using dither_bits=1 would be especially ugly without this. (dither table is treated as signed float, autocorrect levels)
Why autocorrect? Ordered dither produces (2^dither_bits) different pixel values.
e.g. dither_bits=1 results in pixel values 0 and 1; dither_bits=2 => 0 to 3, and so on, dither_bits=7 => 0 to 127
When these dithered pixel values are scaled back to 8 bits, Avisynth stretches the upper extremes to 255 (8 bit case).
At dither_bits=1 instead of 0, 128 we get 0 and 255. Or at dither_bits=2 the values 0, 64, 128, 192 are translated to 0, 85, 170, 255.
Note: for low dither targets RGB definitely looks better.

- Use _Matrix name "bt470m" for value=4 ("fcc" is still kept)
Source: Rename AVS_MATRIX_FCC to AVS_MATRIX_BT470_M
- ConvertBits: Correct conversion of full-range chroma at 8-16 bits, keeping center (32 bit float was O.K.) (ditherless case)
- ConvertBits: Direct, much quicker conversions between 8-16 bit formats when either source or target is full range, avx2 support (ditherless case)
Special even quicker case: 8->16 bit fulls=true, fulld=true (simply *257)
- ConvertBits: Fix: fulls=true->fulld=true 16->8 bit missing rounding
- CMake/source: Intel C++ Compiler 2021 and Intel C++ Compiler 19.2 support
With the help of CMake GUI:
- Generator: "Visual Studio 16 2019"
- Optional toolset to use (-T option): (type to the editbox)
For LLVM based icx: Intel C++ Compiler 2021
For classic 19.2 icl: Intel C++ Compiler 19.2
- Specify native compilers (choose radiobutton),
then browse for the appropriate compiler executable path. For example:
icx: C:\Program Files (x86)\Intel\oneAPI\compiler\latest\windows\bin\icx.exe
icl: C:\Program Files (x86)\Intel\oneAPI\compiler\latest\windows\bin\intel64\icl.exe
There are some bugs in the Intel-VS integration:
If you have errors like "xilink: : error : Assertion failed (shared/driver/drvutils.c, line 312" then
as a workaround you must copy clang.exe (by default it is located in C:\Program Files (x86)\Intel\oneAPI\compiler\latest\windows\bin)
to the folder beside xilink (for x64 configuration it is in C:\Program Files (x86)\Intel\oneAPI\compiler\latest\windows\bin\intel64).
- CMake/source: Intel C++ Compiler 2021 and Intel C++ Compiler 19.2 support

[tormento]

What I enjoyed most was playing with the low dither_bit option both in ordered and in Floyd type of dithering.
Thank you so much!

Would it be difficult to implement the dithering modes of fmtconv? At least dmode=8, i.e. Void and cluster halftone dithering. I find it really effective and compression efficient.

[Dogway]

Huge changelog. Thanks a lot!

EDIT: cretindesalpes also suggests Sierra dithering for encoding, I guess these 2 are worthy of being ported

[pinterf]

Today's news:
Avisynth+ 3.7.1 test build 27 (20211117) (https://drive.google.com/uc?export=download&id=1tyZCCC96arWeXaZ0EKMSiKkS6My9LxRR)

- Expr: atan2 to SSE2 and AVX2. Up to 20x speed.
See notes in readme

[pinterf]

@pinterf
This is an "old" version, two updates has been made since.
Last is : https://www.itu.int/pub/R-REP-BT.2380-2-2018

You can read the PDF i've made with my HDRTools, i've tried to explain some things. I don't know if i've succed...

Thanks for the materials. For the first sight: I'm glad that all this happens in a 3rd party plugin by you :)

[Dogway]

Thanks, you are turning my functions into legacy ^^
Is 'sign' operator possible? I see many cases of where the only alternative is to use ternaries " 1 -1 ? " or division "A B /" to get the sign, both are slow.

EDIT: Also noticed that ColorYUV(levels="TV->PC") doesn't write to frameprops. I don't use it but many users do.

[hello_hello]

pinterf,
Just a minor thing, out of curiosity....
Is there a reason why zero is never displayed as a negative number by the subtitle function when it's an integer, but it can for float? I don't think Avisynth 2.6 did that.

Subtitle(string(-0)) # displays zero
SubTitle(string(float(-0))) # displays zero
Subtitle(string(-0.0)) # displays minus zero
Subtitle(string(-float(0))) # displays minus zero

[Dogway]

Another issue (probably) with new scriptclip and frameproperties.
This works:
ScriptClip( """
st = propGetAsArray("_MinMax")
subtitle(string(st[0])) """ )
But this doesn't
ScriptClip( function [] () {
st = propGetAsArray("_MinMax")
subtitle(string(st[0])) } )

[StainlessS]

HH,
There is no such thing as -ve zero for int, leastwise not on x86 using Two's Complement arithmetic, [Two's Complement]:- https://en.wikipedia.org/wiki/Two%27s_complement ]
Some machines [sometimes older mainframes], might use Ones Complement, which does have -ve zero.
Some machines [sometimes mainframes] may use Sign and Magnitude, where a sign bit denotes sign, and remainder of bits the magnitude, these can have -ve zero.
[Signed number representations]:- https://en.wikipedia.org/wiki/Signed_number_representations

IEEE 754 floats use a sign bit, and so can also represent -ve zero. [IEEE754]:- https://en.wikipedia.org/wiki/IEEE_754

EDIT: From long time ago [2011]
I came upon this snippet, in the v2.6 header.

#if 0
#define MAX_INT 0x7fffffff
#define MIN_INT -0x7fffffff // ::FIXME:: research why this is not 0x80000000
#endif

This seems to go back some time and possibly written by Ben Rudiak-Gould, himself.
(EDIT: apart from the ::FIXME:: )


Signed Int System Equivalent meaning Result of
under system -0x7FFF,FFFF
of 0x8000,0000

Twos Complement (-MAX_INT) - 1 0x8000,0001 ie -MAX_INT

Ones Complement -MAX_INT 0x8000,0000 ie -MAX_INT

Sign & Magnitude -0 0xFFFF,FFFF ie -MAX_INT


The above "#define MIN_INT -0x7fffffff" is portable across all three positional [EDIT: See above in BLUE]
number systems regarding signed integers whereas the 0x8000,0000 one is not.

It is only when "Sign & Magnitude" is considered that the reasoning becomes evident.

EDIT: Under Twos Complement, 0x8000,0000 is troublesome anyway with
-(0x80000000) == 0x80000000

Above, eg 0x8000,0000 is intended to be read as 0x80000000 : intended to make it easier to read.

EDIT: MAX_INT and MIN_INT were present in AVISYNTH_VERSION 3 header [v2.58] and earlier versions of v2.60 header,
but removed from final v2.60 header [AVISYNTH_VERSION 5], not that long after above quoted post. [although IanB made no comment on the post]

[hello_hello]

StainlessS,
Thanks for the info. I get the gist of it, at least.
Maybe something's different for Avisynth+ in respect to the way the string function converts negative zero to a string, because I checked Avisynth 2.6 to see if I was remembering correctly, and both of these display as zero.

Subtitle(string(-0.0))
Subtitle(string(-float(0)))

It's just a little thing but it puzzled me for a function that does the math with positive numbers but ultimately displays them as negative (right and bottom resizer cropping, for example). So to prevent negative zero....

A = 1.0 - 1.0
A = (A == 0) ? 0 : A
Subtitle(string(-A))

Although as it turns out, both Avisynth 2.6 and Avisynth+ round negative numbers to negative zero, or it's the string function rounding that way. This displays as negative zero, even though there's no decimal places.

Subtitle(string(-0.00001, "%.0f"))

So for either version preventing the subtitle function displaying negative zero involved something like...

A = 1.0000001 - 1.0
A = (A <= 0.0005) ? 0 : A
Subtitle(string(-A), "%.3f")

[pinterf]

Another issue (probably) with new scriptclip and frameproperties.
This works:
ScriptClip( """
st = propGetAsArray("_MinMax")
subtitle(string(st[0])) """ )
But this doesn't
ScriptClip( function [] () {
st = propGetAsArray("_MinMax")
subtitle(string(st[0])) } )

This one works for me.
propSet("_MinMax",[12,32])
ScriptClip( function [] () {
st = propGetAsArray("_MinMax")
subtitle(string(st[0])) } )

[pinterf]

Thanks, you are turning my functions into legacy ^^
Is 'sign' operator possible? I see many cases of where the only alternative is to use ternaries " 1 -1 ? " or division "A B /" to get the sign, both are slow.

EDIT: Also noticed that ColorYUV(levels="TV->PC") doesn't write to frameprops. I don't use it but many users do.
"sgn" as 1, 0, -1 is done.

ColorYUV: under construction.
Obviously, this filter with its existing parameter values cannot use input frame properties. Explicite "PC->TV" and "TV->PC" will override any input properties. But setting the output range is O.K.

[Dogway]

Sorry, it must be then a specific issue with ShowChannels. Back compatibility might be broken.
ShowChannels(SetVar=True,show=false)
# global SC_LMn_0 = 2
ScriptClip( function [] () {
subtitle(string(SC_LMn_0)) } )


EDIT: a random question, should masks (scalar by nature) should be in fullscale bitdepth values (ie. 0-65535), or it depends on the bitdepth scale it's going to be used for?

EDIT2: And I know this is not going to be very popular, but I really miss comments in expression blocks, for example ignore everything after # until end-of-line or better #this_is_is_a_comment. So I can add some notes in complex ones like medians, adaptive_sharpen, etc.

[LigH]

Is there a reason why zero is never displayed as a negative number by the subtitle function when it's an integer, but it can for float?

Yes, it's caused by the binary representation of the numbers.

Integer numbers are discrete positions in a range. There is only exactly one number with the value zero in an integer type of a specific resolution = value range. Because there is already one zero, there cannot be another "negative zero".

Wikipedia: Two's complement (https://en.wikipedia.org/wiki/Two%27s_complement)

IEEE 754 Float numbers have a different representation. They store a mantissa (fraction) and an exponent (scaling factor). And they have an explicit sign bit. This makes it possible to store both a positive and a negative number zero.

Even worse, some numbers do not even have a unique representation: If the mantissa equals zero, the exponent does not matter, it could be any, but the whole number still means zero.

Wikipedia: Signed zero (https://en.wikipedia.org/wiki/Signed_zero)

[wonkey_monkey]

Even worse, some numbers do not even have a unique representation: If the mantissa equals zero, the exponent does not matter, it could be any, but the whole number still means zero.



The mantissa has an implicit leading "1" (unless it's a denormal number) so that is not the case.

[pinterf]

Sorry, it must be then a specific issue with ShowChannels. Back compatibility might be broken.
ShowChannels(SetVar=True,show=false)
# global SC_LMn_0 = 2
ScriptClip( function [] () {
subtitle(string(SC_LMn_0)) } )

Sorry, what is ShowChannels?

But without knowing that, variable visibility - especially writing global variables - is very strict inside a function which is inside a ScriptClip.
See:
http://avisynth.nl/index.php/ConditionalFilter#ScriptClip
Avisynth+ specialities: parameter 'local'



EDIT: a random question, should masks (scalar by nature) should be in fullscale bitdepth values (ie. 0-65535), or it depends on the bitdepth scale it's going to be used for?


Maximum mask value is (2^N)-1 for integer bit depths and 1.0 for float. 255, 1023, 4095, 16383, 65535


EDIT2: And I know this is not going to be very popular, but I really miss comments in expression blocks, for example ignore everything after # until end-of-line or better #this_is_is_a_comment. So I can add some notes in complex ones like medians, adaptive_sharpen, etc.
Considering.

[StainlessS]

ShowChannels:- https://forum.doom9.org/showthread.php?t=163829

EDIT:


void __stdcall ShowChannels::CallSetVar(bool init,int n,int chan,AVSValue avs,IScriptEnvironment* env) {
// Create/Update Global variable
const char*p,*nam[]={"Ave","Min","Max","LMn","LMx","AAve","AMin","AMax","ALMn","ALMx","Visited"};
char bf[256],*d;
for(d=bf,p=Prefix;*d++=*p++;); // strcpy Prefix eg "SC_"
--d; // back up 1, to point at nul term
for(p=nam[n];*d++=*p++;); // strcat variable name part
if(chan >= 0) { // if NOT "Visited" variable
d[-1]='_'; // append channel suffix eg "_0".
*d++ = chan + '0';
*d='\0'; // nul term again
}
if(init) { // In constructor ONLY , use SaveString
env->SetGlobalVar(env->SaveString(bf), avs);
} else { // SaveString not needed as already exists (Created by Constructor)
env->SetGlobalVar(bf, avs);
}
}

[Dogway]

Thanks for the notes. I read that a few days before but didn't quite grasp it. So I have to assume that ShowChannels() is one of those plugins written in the assumption that local=false, hence it doesn't work in new scriptclip with either local=false or true.
I see a mention on UseVar(), but didn't find an example, it might be useful in this case(?).

About masks, here an example:
convertbits(16, fulls=true, fulld=false)
mt_binarize(60) # output 65535
scriptclip("subtitle(string(YPlaneMax),x=30,y=10)")

So here's the situation:
convertbits(16, fulls=true, fulld=false)
a=last
msk=mt_binarize(60)
# ( (0~65280)/65535 * (0|65535)/65535 ) * 65280
expr(a, msk, "x range_max / y range_max / * range_max *","")

If clip source is white 60160, for the white part (pass) of the mask we get:
((60160/65535) * (65535/65535) ) * 65535 = 60160

If the mask has the same bitdepth scale than source:
((60160/65535) * (60160/65535) ) * 65535 = 55225.84

Is this ok (one or the other)?

In my tools I map range_max to bitdepth scale including masks, so this is the situation:
((60160/65280) * (65280/65280) ) * 65280 = 60160

Is this wrong?


On another note I found myself testing 32-bit filtering and getting some warning messages, because I always revert back to input bitdepth (using dithering) when filtering in a higher bitdepth. But now this occurs (not a fan of warning messages):


convertbits(32)
blah()

function blah(clip a) {
a.convertbits(32)
# Required 32-bit filtering
convertbits(BitsPerComponent(a), dither=1)}

[pinterf]

meanwhile
Avisynth+ 3.7.1 test build 28 (20211124) (https://drive.google.com/uc?export=download&id=1Syz8js7R3_8376WAyYKzmwKP40gu3wHy)
Changes since test27
20211124 WIP
------------
- Language syntax: accept arrays in the place of "val" script function parameter type regardless of being named or unnamed.
(Note: "val" is "." in internal function signatures)
Example:
BlankClip(pixel_type="yv12")
r([1, 2, 3])
r(n=[10,11,[12,13]])
r("hello")
function r(clip c, val "n")
{
if (IsArray(n)) {
if (IsArray(n[2])) {
return Subtitle(c, String(n[2,1]), align=8) #13 at the top
} else {
return Subtitle(c, String(n[2]), align=2) #3 at the bottom
}
} else {
return Subtitle(c, String(n), align=5) #hello in the center
}
}

- Histogram "Levels": more precise drawing when bit depth is different from histogram's resolution bit depth
- Expr: no more banker's rounding when converting back float result to integer pixels. Using the usual truncate(x+0.5) rounding method
- ColorYUV: fix 32 bit float output
- ColorYUV: More consistent and accurate output across different color spaces, match with ConvertBits fulls-fulld conversions
- ColorYUV: set _ColorRange frame property
levels = "TV->PC" -> full
levels = "PC->TV" or "PC->TV.Y" or "TV" -> limited
levels = (not given) and _ColorRange property exists -> keeps _ColorRange
levels = (not given) and no _ColorRange property -> full range (old default behaviour)
- ColorYUV: when no hint is given by parameter "levels" then use _ColorRange (limited/full) frame property for establishing source range
If _ColorRange does not exist, it treats input as full range (old default behaviour)
Why: when there is no limited<->full conversion, but gamma is provided then this info is still used in gamma calculation.
- ColorYUV: fixes for showyuv=true:
- fix display when bits=32
- "showyuv_fullrange"=true case: U and V range is chroma center +/- span (1..max) for integer bit depths instead of 0..max
Shown ranges:
For bits=8: 128 +/- 127 (range 1..255 is shown) (UV size is 255x255 -> 510x510 image YV12)
bits=10: range 512 +/- 511 (UV size is 1023x1023 -> 2046x2046 image YUV420P10)
bits=12: range 2048 +/- 2047 (UV size is same as 10 bits 1023x1023 -> 2046x2046 image YUV420P12)
bits=14: range 8192 +/- 8191 (UV size is same as 10 bits 1023x1023 -> 2046x2046 image YUV420P14)
bits=16: range 32768 +/- 32767 (UV size is same as 10 bits 1023x1023 -> 2046x2046 image YUV420P16)
bits=32: range 0.0 +/- 0.5 (UV size is same as 10 bits 1023x1023 -> 2046x2046 image YUV420PS)
In general: chroma center is 2^(N-1); span is (2^(N-1))-1 where N is the bit depth
- propShow: display _Matrix, _ColorRange and _ChromaLocation constants with friendly names
- Info on Wincows XP compatibility (Microsoft side)
Avisynth+ can be build to be XP compatible (VS2019): v141_xp toolset and -Z-threadSafeInit flag.
But in order to work, a _compatible_ (=not latest) Visual C++ runtime is still needed (XP support has been stopped by MS meanwhile)
As experienced here: https://github.com/AviSynth/AviSynthPlus/issues/241
The latest XP compatible version is probably 14.28.29213.0.
Links to official installers for last XP compatible Microsoft Visual C++ 2015-2019 Redistributable (version 14.28.29213):
x64 - https://download.visualstudio.microsoft.com/download/pr/566435ac-4e1c-434b-b93f-aecc71e8cffc/B75590149FA14B37997C35724BC93776F67E08BFF9BD5A69FACBF41B3846D084/VC_redist.x64.exe
x86 - https://download.visualstudio.microsoft.com/download/pr/566435ac-4e1c-434b-b93f-aecc71e8cffc/0D59EC7FDBF05DE813736BF875CEA5C894FFF4769F60E32E87BD48406BBF0A3A/VC_redist.x86.exe

- Expr: new function "sgn". Returns -1 when x is negative; 0 if zero; 1 when x is positive

[pinterf]

Thanks for the notes. I read that a few days before but didn't quite grasp it. So I have to assume that ShowChannels() is one of those plugins written in the assumption that local=false, hence it doesn't work in new scriptclip with either local=false or true.

There was no change in behaviour in ScriptClip called with string parameter. Theoretically when ScriptClip is called with the function-syntax then local=false would help, I think it can even write back global variables with global varname = value syntax. (But varname = value may not work, don't know)

[pinterf]

If clip source is white 65280, for the white part (pass) of the mask we get:
((65280/65535) * (65535/65535) ) * 65280 = 65025.99

I see. The problem is that 65280 (255 << 8) is not a standard white in either 16 bit logic. Too big to be a limited range value. If it were full-range then it must be 65535.

edit: with the new 'sgn' you can convert 0..max-1 mask values into 0.0 and 1.0 and use it for multiplication

[Dogway]

Sorry, I borked at many places with my examples. I will give it a though and edit it.

EDIT: ok, 65280 is a defect, it shouldn't exist at first place or fix prior to anything.

[pinterf]

ShowChannels:- https://forum.doom9.org/showthread.php?t=163829

Thanks, downloaded, I'm gonna give it a try.

[FranceBB]

Thanks Ferenc! Keep them coming! :D

About XP, you should both (you and qyot) be aware already that the C++ Redistributable shipped with the installer were not XP Compatible 'cause I raised the "issue" several months ago, saying that reverting to the old C++ Redistributable worked.
I'm gonna quote myself 'cause it's always funny ehehehe
Cherry picked from this very same thread, posted on 5th February 2021:

Hi Ferenc,
I have just a thing to report.

The AviSynth+ 3.7.0 release installer for Windows XP has been shipped with the "wrong" version of C++ Redistributable.
Let me explain.

There are:

- AviSynthPlus_3.7.0_20210111_vcredist_xp.exe
- AviSynthPlus_3.7.0_20210111_xp.exe

Both builds work just fine on Windows XP and they've been compiled to run on XP targeting v141_xp correctly, so no problem, however the "_vcredist_xp.exe" version isn't shipping the XP compatible Microsoft C++ Redistributable 2015-2019 installer, so what will happen is that the C++ Redistributable that is gonna be installed won't work on XP, hence it will make impossible to use Avisynth.
In order to make it work, I've installed the "vcredist" version shipped with AviSynth+ 3.6.1 and then I installed "AviSynthPlus_3.7.0_20210111_xp.exe" without re-installing the C++ Redist and it worked like a charm.
I think you should be shipping the vcredist version from AVS 3.6.1 for XP ;)

And Stephen said:

Then Microsoft changed the vcredist silently, because it was the standard 2015-2019 vcredist download link.

to which I replied:

Yep...
I just checked and it seems that VC++ 2019 version 14.28.29213.0 (August 2020) is the last version compatible with Windows XP... :( C++ Redistributable AIO (XP Compatible) (https://github.com/abbodi1406/vcredist/releases/download/v0.35.0/VisualCppRedist_AIO_x86_x64_35.zip)
I've just archived it... For future XP releases, we should always include that one I think.


So... when you're gonna release 3.7.1 stable, you know what to do :P

[pinterf]

Thanks Ferenc! Keep them coming! :D

About XP, you should both (you and qyot) be aware already that the C++ Redistributable shipped with the installer were not XP Compatible 'cause I raised the "issue" several months ago, saying that reverting to the old C++ Redistributable worked.


I know exactly what I'd like to do with XP support but I won't do that at the moment. :)

As written in the changelog:
- Info on Windows XP compatibility (Microsoft side)
Avisynth+ can be build to be XP compatible (VS2019): v141_xp toolset and -Z-threadSafeInit flag.
But in order to work, a _compatible_ (=not latest) Visual C++ runtime is still needed (XP support has been stopped by MS meanwhile)
As experienced here: https://github.com/AviSynth/AviSynthPlus/issues/241
The latest XP compatible version is probably 14.28.29213.0.
Links to official installers for last XP compatible Microsoft Visual C++ 2015-2019 Redistributable (version 14.28.29213):
x64 - https://download.visualstudio.microsoft.com/download/pr/566435ac-4e1c-434b-b93f-aecc71e8cffc/B75590149FA14B37997C35724BC93776F67E08BFF9BD5A69FACBF41B3846D084/VC_redist.x64.exe
x86 - https://download.visualstudio.microsoft.com/download/pr/566435ac-4e1c-434b-b93f-aecc71e8cffc/0D59EC7FDBF05DE813736BF875CEA5C894FFF4769F60E32E87BD48406BBF0A3A/VC_redist.x86.exe

[pinterf]

(I was just playing some hours with an interesting mod - proof of concept -, namely 64 bit double and int64 support. Of course this works only on x64. But hey, I've just displayed 7FFFFFFFFFFFFFFF in hex after getting sqrt(2) in double :))

[FranceBB]

I know exactly what I'd like to do with XP support

:scared:


but I won't do that :)


I'm sure manolito, hello_hello, Katie and others are gonna thank you for this eheheheh

[manolito]

I'm sure manolito, hello_hello, Katie and others are gonna thank you for this eheheheh

You can drop me from this list now... :scared:
About 6 months ago I woke up one morning after having an overnight stroke without even waking up. Life goes on, but I am no longer the smart person I used to be. I cannot even decipher code I wrote myself right before this stroke.

I still try to read the forum threads I used to be interested in, but it is a bit frustrating. As far as computers are concerned, I am happy that I can manage my emails and my bank account.

So I would like to thank all the folks from this forum, I had a great time here as long as it lasted.

Take care and Cheers
manolito

[kedautinh12]

Bye manolito, see you again in future :D

[ryrynz]

You can drop me from this list now... :scared:


I'm sure you'd still like XP support to die lol.

Life is too short to focus on what was, I hope you can enjoy those things you can do with as much time as you have left. A pleasure having you around and all the best to you.

[VoodooFX]

About 6 months ago I woke up one morning after having an overnight stroke without even waking up. Life goes on, but I am no longer the smart person I used to be. I cannot even decipher code I wrote myself right before this stroke.

Ooh, scary stuff, sad to hear... I hope you'll get better. :scared:

[pinterf]

Hey manolito, sad things happened to you, get better. Avisynth x86 test versions are still compiled with SSE requirement only, traditionally, just because of you.

[tormento]

About 6 months ago I woke up one morning after having an overnight stroke without even waking up.
I have a rare genetic mutation (EDS syndrome) that gives me sometimes excruciating pain and the risk of a sudden death every single day.

Unfortunately the doctors discovered it when I was 45 already, with all that goes with it. No treatment possible, at least for the next decade or so.

I toss the coin every morning and every day is a new day of a new life.

I can't work anymore (and welfare is not helping me at all) but I go to swimming pool, diving (when I have enough money) and I do try my best.

My best wishes for your "new" everyday life.

[FranceBB]

About 6 months ago I woke up one morning after having an overnight stroke without even waking up. Life goes on, but I am no longer the smart person I used to be. I cannot even decipher code I wrote myself right before this stroke.

I still try to read the forum threads I used to be interested in, but it is a bit frustrating. As far as computers are concerned, I am happy that I can manage my emails and my bank account.

So I would like to thank all the folks from this forum, I had a great time here as long as it lasted.

Take care and Cheers
manolito

I'm really sorry to hear that. :(
Take care and try to rest a bit and not think about this.
That must have been scary...
Anyway, your words are always gonna stay here in this forum and I've had a good time with you and the others too.
I consider Doom9 as a sort of family which just shows how people can be respectful and cooperative. :)
I hope you're gonna recover and get back to be active again here one day, but in any case, it's been a pleasure having you around. :)


I have a rare genetic mutation (EDS syndrome) that gives me sometimes excruciating pain and the risk of a sudden death every single day.

Unfortunately the doctors discovered it when I was 45 already, with all that goes with it. No treatment possible, at least for the next decade or so.

I toss the coin every morning and every day is a new day of a new life.

Yet the fact that you wake up every morning and that you also find time to post here, test scripts, deal with new stuff etc just shows how much you care about encoding. :') I mean, other people in your situation would have just dropped everything... If we won't hear back from you one day, we'll know what happened... :(

[tormento]

If we won't hear back from you one day, we'll know what happened... :(
[emoji1591] (sorry, I think Italians only can understand).

I prefer to think that those days when you don’t hear me, it’s because I am at the sea.

[kedautinh12]

So sad for you two

[Gavino]

Best wishes, manolito!

Coraggio, tormento!

[FranceBB]

[emoji1591] (sorry, I think Italians only can understand).


hahahahaha "le corna" / "tocca ferro" is impossible to translate ehehehe I guess the closest thing we can say to translate it would be something like 'touch wood' :P


I prefer to think that those days when you don’t hear me, it’s because I am at the sea.

yeah enjoying the sun and the sea, but hopefully not in France this time so you won't have to remember *that person* eheheheh

[VoodooFX]

hahahahaha "le corna" / "tocca ferro" is impossible to translate ehehehe I guess the closest thing we can say to translate it would be something like 'touch wood' :P

I know that English is a poor language, but they understand "knocking on wood". :P

[StainlessS]

but they understand "knocking on wood"
Yes they do, and we send best wishes to Mani and Tormento and feel for the pair of you,
you have both made for an indelible presence on the forum and if you choose to spend less time here
you will be sorely missed.

[pinterf]

Avisynth+ 3.7.1 test build 29 (20211126) (https://drive.google.com/uc?export=download&id=1gxEaQDQYoLBiZdFc1HOJKWgh8o7zVLf5)
Plus new sections in http://avisynth.nl/index.php/ConditionalFilter#ScriptClip after experimenting with ScriptClip, functions, local and global variables.
20211126 WIP
------------
- New: ArrayAdd(a, b): appends b to the end of a (a is array, b is a value which can be another array)
Example:
a = []
a=ArrayAdd(a,[1,2]) # [[1,2]]
a=ArrayIns(a,3,0) # [3,[1,2]]
a=ArrayAdd(a,"s1") # [3,[1,2],"s1"]
a=ArrayAdd(a,"s2") # [3,[1,2],"s1","s2"]
a=ArrayDel(a,2) # [3,[1,2],"s2"]
- New: ArrayIns(a, b, n): inserts b into a to position n (a is array, b is a value (but can be another array), n is a zero based index. 0: inserts at the beginning, array_size: inserts after the last element)
- New: ArrayDel(a, n): removes the n-th element from a (a is array, n is a zero based index, must be a valid index between 0 and arraysize-1)
- Enhancement: xPlaneMin/Max/Median/MinMaxDifference runtime functions to accept old packed formats (RGB24/32/48/64 and YUY2)
(By autoconverting them to Planar RGB or YV16)
- New runtime function: PlaneMinMaxStats(clip, float "threshold", int "offset", int "plane", bool "setvar")
Returns an 5-element array with [min,max,thresholded minimum,thresholded maximum,median]
Parameters:
float 'threshold': a percent number between 0.0 and 100.0%
int 'offset': if not 0, they can be used for pulling statistics from a frame number relative to the actual one
int 'plane' (default 0):
0, 1, 2 or 3
for YUV inputs they mean Y=0,U=1,V=2,A=3 planes
for RGB inputs R=0,G=1,B=2 and A=3 planes
bool 'setvar' (default false):
when true then it writes a global variables named
"PlaneStats_min" "PlaneStats_max" "PlaneStats_thmin" "PlaneStats_thmax" "PlaneStats_median"

Note: using global variables are thread safe from ScriptClip only when used with 'function'-syntax call with its default 'local'=true

Example:

# function-syntax ScriptClip + runtime function call + dedicated global var demo
# Here 'local'=true (for the sake of the demo; this is the default for this mode).
# 'local'=true makes a dedicated global variable area, in which 'last' and 'current frame'
# 'c' is a parameter which must be passed to the function. Name is not important, it moves the actual clip into function's scope.
# This is why we can SubTitle on it.
# A function can see only global variables. 'last' and 'current_frame' are available here - they are global variables which were
# set by ScriptClip after creating a safe global variable stack.
# PlaneMinMaxStats writes five global variables "PlaneStats_min", "PlaneStats_max", "PlaneStats_thmin", "PlaneStats_thmax", "PlaneStats_median"
ScriptClip( function [] () {
x=PlaneMinMaxStats(threshold=30, offset=0, plane=1, setvar=true)
subtitle("min=" + string(PlaneStats_min) + " thmax" + String(PlaneStats_thmax) + " median = " + String(PlaneStats_Median) + " median_too=" + String(x[4]))
} , local = true)

[StainlessS]

Note: using global variables are thread safe from ScriptClip only when used with 'function'-syntax call with its default 'local'=true
Does that mean that globals using standard scriptclip are not thread safe in avs + MT ? [or specifically only in 'function'-syntax]

In my plugs setting globs, I usually use a PreFix arg (string), where default prefix might be eg "PlaneStats_", and subnames postfixed to that.
What if you wanna sample frames, previous, current, and next [using offset], and you only got single non prefix option.
[I guess you could sample, and transfer to locals between calls].
Also, if only of use in function'-syntax call, would setting locals be better [some of mine set globals, some locals, has depended on user feedback or plugin request and such].

and thanks for the new one, installing it in a few minutes.

[Dogway]

oooh thanks a lot! Out of dispair I was at the brink of writing a script based, obviously unoptimized, PlaneStats() function. Thanks for the addition.

I plan to add some kind of rolling average to the values, will check tomorrow how to do it but looks tricky.

[goorawin]

Thanks for the continued updates, however one thing I have noticed is that all these test builds (64bit) have been unable to run MP_Pipeline. The following error occurs:
MP_Pipeline: Unable to create slave process. Message: (slave_common.cpp) ReadFile failed,code=109

[pinterf]

There has been an update on mp_pipeline some months ago, do you have the latest one?

[pinterf]

StainlessS: I'm gonna return to it later, it takes more time for me to explain it, what I found is already added to the linked wiki entry.

[goorawin]

There has been an update on mp_pipeline some months ago, do you have the latest one?
Thanks pinterf that did it. I thought I had the lastest, but turns out I didn't

[VoodooFX]

20211126 WIP
- Enhancement: xPlaneMin/Max/Median/MinMaxDifference runtime functions to accept old packed formats (RGB24/32/48/64 and YUY2)
I guess that's after my recent complaint somewhere. You want me to drop v2.6 support. :D

Tested new PlaneMinMaxStats():

ScriptClip( function [] () {
Y=PlaneMinMaxStats(plane=0, setvar=true)
YMinMaxDiff = Y[1] - Y[0]
U=PlaneMinMaxStats(plane=1, setvar=true)
UMinMaxDiff = U[1] - U[0]
V=PlaneMinMaxStats(plane=2, setvar=true)
VMinMaxDiff = V[1] - V[0]
subtitle("Y=" +string(int(YMinMaxDiff)) +" U=" +String(int(UMinMaxDiff)) +" V=" +String(int(VMinMaxDiff)))
} , local = true)

YV12
https://i.imgur.com/Ho43aaO.png

Despite "for RGB inputs R=0,G=0,B=0 and A=3", RGB24 works too:

https://i.imgur.com/M1E1X7a.png


EDIT:
I'm wondering, what is good corresponding chroma plane minmaxdiff value to YPlaneMinMaxDifference value to get the somewhat solid looking frames?
Now I'm using half of Y for some reason, maybe I should use same?

[pinterf]

I guess that's after my recent complaint somewhere. You want me to drop v2.6 support. :D

Not exactly. Some other runtime functions supported them already in the same autoconverting way so I copy pasted those lines.

Tested new PlaneMinMaxStats():

ScriptClip( function [] () {
Y=PlaneMinMaxStats(plane=0, setvar=true)
YMinMaxDiff = Y[1] - Y[0]
U=PlaneMinMaxStats(plane=1, setvar=true)
UMinMaxDiff = U[1] - U[0]
V=PlaneMinMaxStats(plane=2, setvar=true)
VMinMaxDiff = V[1] - V[0]
subtitle("Y=" +string(int(YMinMaxDiff)) +" U=" +String(int(UMinMaxDiff)) +" V=" +String(int(VMinMaxDiff)))
} , local = true)

YV12
https://i.imgur.com/Ho43aaO.png

Despite "for RGB inputs R=0,G=0,B=0 and A=3", RGB24 works too:

https://i.imgur.com/M1E1X7a.png


EDIT:
I'm wondering, what is good corresponding chroma plane minmaxdiff value to YPlaneMinMaxDifference value to get the somewhat solid looking frames?
Now I'm using half of Y for some reason, maybe I should use same?
R=0 G=1 B=2 of course. (and I noticed other typos in the changelog.) If you were able to pass plane=3 for RGB24 then it seems that I do not have proper check for valid plane indexes.

Note: when you do not need global variables, setvar=true is not necessary. You'll get the result array without it.

[Dogway]

@pinterf, now after_frame=true, is not required right?

[Gavino]

now after_frame=true, is not required right?
I suppose you are referring to the example in pinterf's post #1589 above.

I wouldn't expect the behaviour of after_frame to have changed.
In the recent example that required after_frame=true, the call to ShowChannels() was outside the ScriptClip() call (before it).
But in pinterf's example, we have PlaneMinMaxStats() inside ScriptClip's run-time script. The order of events inside the run-time script is not affected by after_frame, which only controls whether the run-time script as a whole is to be evaluated before or after fetching the current input frame.

However... (catch 22), evaluating the run-time script can itself sometimes cause an input frame to be fetched at some point, so the order of events is often not at all obvious!
See here for more hairy details:
Runtime filter evaluation sequence rather more complex than documented

[StainlessS]

Any runtime script that directly samples/force_fetches a frame [YPlaneMin, Averageluma or even PlaneMinMaxStats(), etc], does not need after_frame=true.
[any script code following the sampling will act as if after_frame=true {the sampling will force a fetch of the frame and then sample it, any script code following that acts as if after_frame=true}]

EDIT:

SSS="""
x = 42 # no force fetch of frame for this line of code, processed before frame is fetched.
Return Last # or implicit return of Last, forces fetch of frame from previous filter, (after above script code already executed)
"""
Scriptclip(SSS,after_frame=FALSE)



SSS="""
x = 42 # As After_frame=true, so frame already requested from previous filter even before SSS script is executed.
Return Last
"""
Scriptclip(SSS,after_frame=TRUE)

[Dogway]

Thanks both, yes this is not my area of expertise, but at least I could see the difference since ShowChannels() was called outside the runtime env.

I'm having another issue, with the goal to fetch past runtime variables I coded this (snippet from modded ScSelect_HBD() ):

ScriptClip( function [] () {

n = current_frame
propSet("_SceneChangePrevLast", n == 0 || SC != 0 ? 0 : current_frame, 0) # Last detected Start frame

SCP = n != 0 && SC != 0 ? PropGetInt(last,"_SceneChangePrevLast",offset=-1) : current_frame
propSet(last,"_SceneChangePrevLast",SCP, 0)
} )

This is kind of a readback, but I don't know if it's possible. Or I have to get out of the scriptclip, or even the function's scope. Basically I'm checking if current frame is not a SC, if so copy the previous frameproperty to current. Basically I want to hold the last SC frame number in "_SceneChangePrevLast" property until a new SC. I'm trying to avoid globals as generally they are not advised.


@pinterf: OMG, at last (https://github.com/AviSynth/AviSynthPlus/commit/e35f52e40a158c264e8a94bcc11f2025d006126d)?! Are you holding back for Xmas present :rolleyes: or is it still WIP? I predict a lot of filtering in 14-bits in the future if my 32Gb of RAM permit lol.

[pinterf]

Thanks both, yes this is not my area of expertise, but at least I could see the difference since ShowChannels() was called outside the runtime env.

I'm having another issue, with the goal to fetch past runtime variables I coded this (snippet from modded ScSelect_HBD() ):

ScriptClip( function [] () {

n = current_frame
propSet("_SceneChangePrevLast", n == 0 || SC != 0 ? 0 : current_frame, 0) # Last detected Start frame

SCP = n != 0 && SC != 0 ? PropGetInt(last,"_SceneChangePrevLast",offset=-1) : current_frame
propSet(last,"_SceneChangePrevLast",SCP, 0)
} )

This is kind of a readback, but I don't know if it's possible. Or I have to get out of the scriptclip, or even the function's scope. Basically I'm checking if current frame is not a SC, if so copy the previous frameproperty to current. Basically I want to hold the last SC frame number in "_SceneChangePrevLast" property until a new SC. I'm trying to avoid globals as generally they are not advised.


@pinterf: OMG, at last (https://github.com/AviSynth/AviSynthPlus/commit/e35f52e40a158c264e8a94bcc11f2025d006126d)?! Are you holding back for Xmas present :rolleyes: or is it still WIP? I predict a lot of filtering in 14-bits in the future if my 32Gb of RAM permit lol.
Yes, Luts are on the workbench. Some other cleanups are in progress.
Till then: presently I put an error if 1D or 2D Lut is not supported for a given bit depth. I suppose it's better if no fatal error is given but do the usual realtime Expr instead.

[Dogway]

I'm not a fan of error messages as they are disruptive, but in any case I already adapted ExTools to select proper lut int for given bitdepth. I'm very curious to see what kind of performance upgrades it brings, specially when there's pixel addressing involved in the code.

[DTL]

Do SetMaxCPU() support also AVX512 flags (disabling) ?

The file cpuid.h in the /avs includes to latest official MVtools https://github.com/pinterf/mvtools/blob/mvtools-pfmod/DePan/include/avs/cpuid.h lists

CPUF_AVX512F = 0x100000, // AVX-512 Foundation.
CPUF_AVX512DQ = 0x200000, // AVX-512 DQ (Double/Quad granular) Instructions
CPUF_AVX512PF = 0x400000, // AVX-512 Prefetch
CPUF_AVX512ER = 0x800000, // AVX-512 Exponential and Reciprocal
CPUF_AVX512CD = 0x1000000, // AVX-512 Conflict Detection
CPUF_AVX512BW = 0x2000000, // AVX-512 BW (Byte/Word granular) Instructions
CPUF_AVX512VL = 0x4000000, // AVX-512 VL (128/256 Vector Length) Extensions
CPUF_AVX512IFMA = 0x8000000, // AVX-512 IFMA integer 52 bit
CPUF_AVX512VBMI = 0x10000000,// AVX-512 VBMI


Wiki still only lists up to AVX2 - http://avisynth.nl/index.php/Internal_functions#SetMaxCPU

So exact question: if at CPU with AVX512F (and may be more) I set SetCPUMax(avx2) in the script - will it return IscriptEnviroment->GetCPUFlags() max AVX2 to the plugin ? And if not call this script function - will it return all AVX512 flags found ?

I do not have debugger at the system with AVX512F cpu and want to test plugin with AVX512 functions enabled and disabled (without creating special builds and adding more params).

[pinterf]

Do SetMaxCPU() support also AVX512 flags (disabling) ?

The file cpuid.h in the /avs includes to latest official MVtools https://github.com/pinterf/mvtools/blob/mvtools-pfmod/DePan/include/avs/cpuid.h lists

CPUF_AVX512F = 0x100000, // AVX-512 Foundation.
CPUF_AVX512DQ = 0x200000, // AVX-512 DQ (Double/Quad granular) Instructions
CPUF_AVX512PF = 0x400000, // AVX-512 Prefetch
CPUF_AVX512ER = 0x800000, // AVX-512 Exponential and Reciprocal
CPUF_AVX512CD = 0x1000000, // AVX-512 Conflict Detection
CPUF_AVX512BW = 0x2000000, // AVX-512 BW (Byte/Word granular) Instructions
CPUF_AVX512VL = 0x4000000, // AVX-512 VL (128/256 Vector Length) Extensions
CPUF_AVX512IFMA = 0x8000000, // AVX-512 IFMA integer 52 bit
CPUF_AVX512VBMI = 0x10000000,// AVX-512 VBMI


Wiki still only lists up to AVX2 - http://avisynth.nl/index.php/Internal_functions#SetMaxCPU

So exact question: if at CPU with AVX512F (and may be more) I set SetCPUMax(avx2) in the script - will it return IscriptEnviroment->GetCPUFlags() max AVX2 to the plugin ? And if not call this script function - will it return all AVX512 flags found ?

I do not have debugger at the system with AVX512F cpu and want to test plugin with AVX512 functions enabled and disabled (without creating special builds and adding more params).
No distinct values, but when you set AVX2 then all AVX512 flags are disabled.

[pinterf]

I'm not a fan of error messages as they are disruptive, but in any case I already adapted ExTools to select proper lut int for given bitdepth. I'm very curious to see what kind of performance upgrades it brings, specially when there's pixel addressing involved in the code.
I don't think pixel addressing is compatible with LUT theory.

[pinterf]

I do not have debugger at the system with AVX512F cpu and want to test plugin with AVX512 functions enabled and disabled (without creating special builds and adding more params).
btw, probably in a month I will have an i7 11th gen with AVX512 support. It is a beast. In this article https://www.anandtech.com/show/16535/intel-core-i7-11700k-review-blasting-off-with-rocket-lake/2 they actually had ~225-275W load which is rather huge for a 125W TDP unit.

[DTL]

btw, probably in a month I will have an i7 11th gen with AVX512 support.

It is very good for test plugins. At i5-11500 I today got strange error - can not found AVX2 for MVtools (using Avisynth+ 3.6.1 of the end of 2020). May be if AVX512 present it somehow clear AVX2 flag ? And no debugger at that system to check what is wrong. So currently simply disable that check.

"they actually had ~225-275W load"

For 2 kBytes register file and wide 512bit execution units in AVX512 mode it is really required more power. The register file of AVX2 is only 512 bytes so the AVX512 register file require about 4 times more power for switching. But it is the fastest memory on chip and very lovely for its performance.
Even at 10nm chip or thinner. So peaks power at AVX512 load was up to 290 W. I currently trying to make AVX512 version (mm512_dbsad) of lovely for many HD/UHD users block size 16x16 faster in compare with AVX2 (mm256_mpsadbw). At least already found - gathering load of ref and src is slower in compare with standard SIMD load. So need to redo AVX512 version from easy but slow gather load to standard SIMD load. 16x16 8bit block with ref and sad results fit freely in 2 kBytes AVX512 register file so I hope will be faster in compare with AVX2 version with reloading half of block from cache for process.

" when you set AVX2 then all AVX512 flags are disabled."

Addition: It looks SetCPUMax("avx2") works at i5-11500 as expected. At least AMD uProf can not disassemble AVX512 function and shows difference between enabled SetCPUMax("avx2") (use AVX2 version of function) and commented out SetCPUMax("avx2") string in the script. Looks like flags signalling about AVX512 working and the AVX512 version of function used. Unfortunately AMD uProf still can not load even symbols at the non-build system to display used functions names even with provided .pdb file and all equal paths to sources.

[pinterf]

A ColorBars.Info() will show you the detected CPU flags

[pinterf]

It's December.
Broken test 30 replaced with 32
With qyot27's interface additions
Avisynth+ 3.7.1 test build 32 (20211202) (https://drive.google.com/uc?export=download&id=1xv8zwctgJ16XThopha4TBsU-aw_FTUUP)
20211202 WIP
------------
- Fix: MinMax runtime filter family: check plane existance (e.g. error when requesting RPlaneMinMaxDifference on YV12)
- Fix: prevent x64 debug AviSynth builds from crashing in VirtualDub2 (opened through CAVIStreamSynth)
- ExtractY/U/V/R/G/B/A, PlaneToY: delete _ChromaLocation property. Set _ColorRange property to "full" if source is Alpha plane
- AviSynth interface additions: extend queryable internal environment properties.
Since Interface version 8 IScriptEnvironment::GetEnvProperty (Avisynth.h) and avs_get_env_property (avisynth_c.h)
interface functions can query some specific internal properties of AviSynth core. Thread count, etc..
These are mainly for internal use but some can be useful for plugins and external applications.
Each requested property has an identification number, they are found in avisynth.h and avisynth_c.h

This addition brought new properties to query: host system's endianness, interface version and bugfix subversion.
Relevant enum names start with AEP_ (cpp) or AVS_AEP_ (c) (AEP stands for Avisynth Environment Property)

Details:

AEP_HOST_SYSTEM_ENDIANNESS (c++) AVS_AEP_HOST_SYSTEM_ENDIANNESS (c)
Populated by 'little', 'big', or 'middle' based on what GCC and/or Clang report at compile time.

AEP_INTERFACE_VERSION (c++) AVS_AEP_INTERFACE_VERSION (c)
for requesting actual interface (main) version. An long awaited function.
So far the actual interface version could be queried only indirectly, with trial and error, by starting from e.g. 10 then
going back one by one until CheckVersion() did not report an exception/error code.

Even for V8 interface this was a bit tricky, the only way to detect was the infamous
has_at_least_v8 = true;
try { env->CheckVersion(8); } catch (const AvisynthError&) { has_at_least_v8 = false; }
method.

Now (starting from interface version 8.1) a direct version query is supported as well.
Of course this (one or two direct call only) is the future.
Programs or plugins which would like to identify older systems still must rely partially on the CheckVersion method.

CPP interface (through avisynth.h).

IScriptEnvironment *env = ...
int avisynth_if_ver = 6;
int avisynth_bugfix_ver = 0;
try {
avisynth_if_ver = env->GetEnvProperty(AEP_INTERFACE_VERSION);
avisynth_bugfix_ver = env->GetEnvProperty(AEP_INTERFACE_BUGFIX);
}
catch (const AvisynthError&) {
try { env->CheckVersion(8); avisynth_if_ver = 8; } catch (const AvisynthError&) { }
}
has_at_least_v8 = avisynth_if_ver >= 8; // frame properties, NewVideoFrameP, other V8 environment functions
has_at_least_v8_1 = avisynth_if_ver > 8 || (avisynth_if_ver == 8 && avisynth_bugfix_ver >= 1);
// 8.1: C interface frameprop access fixed, IsPropertyWritable/MakePropertyWritable support, extended GetEnvProperty queries
has_at_least_v9 = avisynth_if_ver >= 9; // future

C interface (through avisynth_c.h)

AVS_ScriptEnvironment *env = ...
int avisynth_if_ver = 6; // guessed minimum
int avisynth_bugfix_ver = 0;
int retval = avs_check_version(env, 8);
if (retval == 0) {
avisynth_if_ver = 8;
// V8 at least, we have avs_get_env_property but AVS_AEP_INTERFACE_VERSION query may not be supported
int retval = avs_get_env_property(env, AVS_AEP_INTERFACE_VERSION);
if(env->error == 0) {
avisynth_if_ver = retval;
retval = avs_get_env_property(env, AVS_AEP_INTERFACE_BUGFIX);
if(env->error == 0)
avisynth_bugfix_ver = retval;
}
}
has_at_least_v8 = avisynth_if_ver >= 8; // frame properties, NewVideoFrameP, other V8 environment functions
has_at_least_v8_1 = avisynth_if_ver > 8 || (avisynth_if_ver == 8 && avisynth_bugfix_ver >= 1);
// 8.1: C interface frameprop access fixed, IsPropertyWritable/MakePropertyWritable support, extended GetEnvProperty queries
has_at_least_v9 = avisynth_if_ver >= 9; // future


AEP_INTERFACE_BUGFIX (c++) AVS_AEP_INTERFACE_BUGFIX (c)
Denotes situations where there isn't a breaking change to the API,
but we need to identify when a particular change, fix or addition
to various API-adjacent bits might have occurred. Could also be
used when any new functions get added.

Since the number is modelled as 'changes since API bump' and
intended to be used in conjunction with checking the main
AVISYNTH_INTERFACE_VERSION, whenever the main INTERFACE_VERSION
gets raised, the value of INTERFACE_BUGFIX should be reset to zero.

The BUGFIX version is added here with already incremented once,
both because the addition of AVISYNTH_INTERFACE_BUGFIX_VERSION
itself would require it, but also because it's intended to signify
the fix to the C interface allowing frame properties to be read
back (which was the situation that spurred this define to exist
in the first place).

- CMake build environment:
While we do need the compiler to support C++17 features, we can
get by on older GCC using CMake 3.6 and -std=c++-1z with some other fixes.
CMAKE_CXX_STANDARD can be raised intelligently to 17 based on whether we detect CMake 3.8 or higher.
- Add AVISYNTHPLUS_INTERFACE_BUGFIX_VERSION

- Avisynth programming interface V8.1 or V9(?)):
Add 'MakePropertyWritable' to the IScriptEnvironment (CPP interface), avs_make_property_writable (C interface)
Add 'VideoFrame::IsPropertyWritable' (CPP interface), avs_is_property_writable (C interface)
(AviSynth interface version will be stepped to V9 in the release version?)

bool env->MakePropertyWritable(PVideoFrame *);
bool VideoFrame::IsPropertyWritable();

'MakePropertyWritable' is similar to 'MakeWritable' but it does not copy all bytes of the frame content in order to have a writable property set.

Reason: 'propSet' is a filter which does not alter frame content, but sets the given frame property in its each GetFrame.
So far it used MakeWritable to obtain a safely modifiable copy of frame properties, however - as a side-effect - full copy of frame content was performed.
(env->getFramePropsRW alone does not ensure a uniquely modifiable property set, it just obtains a pointer which can be used in the property setter functions)
(Note: frame properties of frames obtained by NewVideoFrame, MakeWritable and SubFrame are still safe to modify)
- Expr: when actual bit depth is too large for building LUT table, fallback to realtime mode.
lut_x 1D (realtime when 32 bit)
lut_xy 2D (realtime when 16 or 32 bits)
- Expr: allow 'f32' as internal autoscale target (was: i8, i10, i12, i14, i16 were accepted, only integers)
affects: 'scale_inputs' when "int", "intf", "all", "allf"
more on that (todo: refresh docs) http://avisynth.nl/index.php/Expr
- Expr: fix conversion factor (+correct chroma scaling) when integer-to-integer full-scale automatic range scaling was required
- New: Expr: new parameter integer 'lut'
integer 'lut' (default 0)
0: realtime expression
1: expression is converted to 1D lut (lut_x)
2: expression is converted to 2D lut (lut_xy)
Valid bit depths: lut=1 : 8-16 bits. lut=2 : 8-14 bits. Note: a 14 bit 2D lut needs (2^14)*(2^14)*2 bytes buffer in memory per plane (~1GByte)
In lut mode some keywords are forbidden in the expression: sx, sy, sxr, syr, frameno, time, relative pixel addressing

[VoodooFX]

Enhancement: xPlaneMin/Max/Median/MinMaxDifference runtime functions to accept old packed formats (RGB24/32/48/64 and YUY2)
Yesterday evening I was puzzled by the script's weirdness, but thanks to the Sandman I dreamed where bug is: unknowingly I was feeding YV12 to RPlaneMinMaxDifference, but there were no errors from it.

EDIT:
Actually I didn't checked it in the reality, didn't got my coffee yet, but I trust my dreams [it seems that there I can code better]. :D

[pinterf]

Yesterday evening I was puzzled by the script's weirdness, but thanks to the Sandman I dreamed where bug is: unknowingly I was feeding YV12 to RPlaneMinMaxDifference, but there were no errors from it.
:D
True, the sad truth is that this errorless behaviour is very true, but I fixed, and next time you'll get an error sized as big as an elephant :)

[VoodooFX]

I see that an elephant (https://github.com/AviSynth/AviSynthPlus/commit/13219ff9568bee29b4c1e7532db81f3ae0f40d88) is ready to be seen. :)
Is there any ETA on the official 3.7.1 release?

[pinterf]

I see that an elephant (https://github.com/AviSynth/AviSynthPlus/commit/13219ff9568bee29b4c1e7532db81f3ae0f40d88) is ready to be seen. :)
Is there any ETA on the official 3.7.1 release?
Not really. When it's done. Probably in month or two. Cleanups, documentation, tests. We'll discuss it at the project meeting in a pub. Eeeer, not. :)

[Dogway]

Thanks for the update!! Lots of core changes.

I was eager to test the lut calculations but I get slower speeds.

ConvertBits(14)
a=FlipHorizontal()
ex_blend(last,a,"overlay",0.5) # 246fps
Prefetch(6)

ConvertBits(16)
a=FlipHorizontal()
ex_blend(last,a,"overlay",0.5) # 310fps (270fps with Preftech(6) )
Prefetch(4)

Following I went to test ex_binarize(mode="otsu") which should see great improvements given I use a bunch of ex_lutxy() for the bins. Well, avspmod froze. I understand that some pre-scanning is to be made but I waited a few minutes for a 8-bit clip. Setting lut=0, or even deleting the lut args didn't make things better. Back to test29, no issues.

# 1080p@8-bits
ex_binarize(mode="otsu")

[DTL]

A ColorBars.Info() will show you the detected CPU flags

It is great idea. At i5-11500 it detects almost all possible AVX-512 extensions:
https://i3.imageban.ru/out/2021/12/01/ad931b334df8029caeb951ae475d3ecd.png

For 8/16bit samples most wanted is BW and VL for addition to F.

[pinterf]

Thanks for the update!! Lots of core changes.

I was eager to test the lut calculations but I get slower speeds.

ConvertBits(14)
a=FlipHorizontal()
ex_blend(last,a,"overlay",0.5) # 246fps
Prefetch(6)

ConvertBits(16)
a=FlipHorizontal()
ex_blend(last,a,"overlay",0.5) # 310fps (270fps with Preftech(6) )
Prefetch(4)

Following I went to test ex_binarize(mode="otsu") which should see great improvements given I use a bunch of ex_lutxy() for the bins. Well, avspmod froze. I understand that some pre-scanning is to be made but I waited a few minutes for a 8-bit clip. Setting lut=0, or even deleting the lut args didn't make things better. Back to test29, no issues.

# 1080p@8-bits
ex_binarize(mode="otsu")
O.k. this is why it is called test.
I noticed too that Lut is not necessarily quicker, probably because of the extra memory round? Probably with larger e.g. 4K clip sizes?

EDIT: thanks for the report.
Though I wrote the world's quickest infinite loop, it didn't manage to return from a function :) New build is being produced.

[Dogway]

Ah no problem, take your time, this is a long due feature, as long as it works like in masktools2 I'm fine.

[pinterf]

Test30 replacement
Test31 removed until it really works. Thanks for Dogway for the feedback

[pinterf]

Thanks for the update!! Lots of core changes.

I was eager to test the lut calculations but I get slower speeds.

ConvertBits(14)
a=FlipHorizontal()
ex_blend(last,a,"overlay",0.5) # 246fps
Prefetch(6)

ConvertBits(16)
a=FlipHorizontal()
ex_blend(last,a,"overlay",0.5) # 310fps (270fps with Preftech(6) )
Prefetch(4)


Are you sure ex_blend is using lutx or lutxy?

[Dogway]

Yes, lutxy, the lower block is for reference. Anyway I haven't updated GradePack for some months, so better test with ex_binarize(mode="otsu") for now. Will check test31.

Testing with test31:
expr("x 80 > 255 0 ?","",lut=1)
Ok with lut=0

[pinterf]

Next round with qyot27's interface additions
Avisynth+ 3.7.1 test build 32 (20211202) (https://drive.google.com/uc?export=download&id=1xv8zwctgJ16XThopha4TBsU-aw_FTUUP)
20211202 WIP
------------
- Fix: MinMax runtime filter family: check plane existance (e.g. error when requesting RPlaneMinMaxDifference on YV12)
- Fix: prevent x64 debug AviSynth builds from crashing in VirtualDub2 (opened through CAVIStreamSynth)
- ExtractY/U/V/R/G/B/A, PlaneToY: delete _ChromaLocation property. Set _ColorRange property to "full" if source is Alpha plane
- AviSynth interface additions: extend queryable internal environment properties.
Since Interface version 8 IScriptEnvironment::GetEnvProperty (Avisynth.h) and avs_get_env_property (avisynth_c.h)
interface functions can query some specific internal properties of AviSynth core. Thread count, etc..
These are mainly for internal use but some can be useful for plugins and external applications.
Each requested property has an identification number, they are found in avisynth.h and avisynth_c.h

This addition brought new properties to query: host system's endianness, interface version and bugfix subversion.
Relevant enum names start with AEP_ (cpp) or AVS_AEP_ (c) (AEP stands for Avisynth Environment Property)

Details:

AEP_HOST_SYSTEM_ENDIANNESS (c++) AVS_AEP_HOST_SYSTEM_ENDIANNESS (c)
Populated by 'little', 'big', or 'middle' based on what GCC and/or Clang report at compile time.

AEP_INTERFACE_VERSION (c++) AVS_AEP_INTERFACE_VERSION (c)
for requesting actual interface (main) version. An long awaited function.
So far the actual interface version could be queried only indirectly, with trial and error, by starting from e.g. 10 then
going back one by one until CheckVersion() did not report an exception/error code.

Even for V8 interface this was a bit tricky, the only way to detect was the infamous
has_at_least_v8 = true;
try { env->CheckVersion(8); } catch (const AvisynthError&) { has_at_least_v8 = false; }
method.

Now (starting from interface version 8.1) a direct version query is supported as well.
Of course this (one or two direct call only) is the future.
Programs or plugins which would like to identify older systems still must rely partially on the CheckVersion method.

CPP interface (through avisynth.h).

IScriptEnvironment *env = ...
int avisynth_if_ver = 6;
int avisynth_bugfix_ver = 0;
try {
avisynth_if_ver = env->GetEnvProperty(AEP_INTERFACE_VERSION);
avisynth_bugfix_ver = env->GetEnvProperty(AEP_INTERFACE_BUGFIX);
}
catch (const AvisynthError&) {
try { env->CheckVersion(8); avisynth_if_ver = 8; } catch (const AvisynthError&) { }
}
has_at_least_v8 = avisynth_if_ver >= 8; // frame properties, NewVideoFrameP, other V8 environment functions
has_at_least_v8_1 = avisynth_if_ver > 8 || (avisynth_if_ver == 8 && avisynth_bugfix_ver >= 1);
// 8.1: C interface frameprop access fixed, IsPropertyWritable/MakePropertyWritable support, extended GetEnvProperty queries
has_at_least_v9 = avisynth_if_ver >= 9; // future

C interface (through avisynth_c.h)

AVS_ScriptEnvironment *env = ...
int avisynth_if_ver = 6; // guessed minimum
int avisynth_bugfix_ver = 0;
int retval = avs_check_version(env, 8);
if (retval == 0) {
avisynth_if_ver = 8;
// V8 at least, we have avs_get_env_property but AVS_AEP_INTERFACE_VERSION query may not be supported
int retval = avs_get_env_property(env, AVS_AEP_INTERFACE_VERSION);
if(env->error == 0) {
avisynth_if_ver = retval;
retval = avs_get_env_property(env, AVS_AEP_INTERFACE_BUGFIX);
if(env->error == 0)
avisynth_bugfix_ver = retval;
}
}
has_at_least_v8 = avisynth_if_ver >= 8; // frame properties, NewVideoFrameP, other V8 environment functions
has_at_least_v8_1 = avisynth_if_ver > 8 || (avisynth_if_ver == 8 && avisynth_bugfix_ver >= 1);
// 8.1: C interface frameprop access fixed, IsPropertyWritable/MakePropertyWritable support, extended GetEnvProperty queries
has_at_least_v9 = avisynth_if_ver >= 9; // future


AEP_INTERFACE_BUGFIX (c++) AVS_AEP_INTERFACE_BUGFIX (c)
Denotes situations where there isn't a breaking change to the API,
but we need to identify when a particular change, fix or addition
to various API-adjacent bits might have occurred. Could also be
used when any new functions get added.

Since the number is modelled as 'changes since API bump' and
intended to be used in conjunction with checking the main
AVISYNTH_INTERFACE_VERSION, whenever the main INTERFACE_VERSION
gets raised, the value of INTERFACE_BUGFIX should be reset to zero.

The BUGFIX version is added here with already incremented once,
both because the addition of AVISYNTH_INTERFACE_BUGFIX_VERSION
itself would require it, but also because it's intended to signify
the fix to the C interface allowing frame properties to be read
back (which was the situation that spurred this define to exist
in the first place).

- CMake build environment:
While we do need the compiler to support C++17 features, we can
get by on older GCC using CMake 3.6 and -std=c++-1z with some other fixes.
CMAKE_CXX_STANDARD can be raised intelligently to 17 based on whether we detect CMake 3.8 or higher.
- Add AVISYNTHPLUS_INTERFACE_BUGFIX_VERSION

- Avisynth programming interface V8.1 or V9(?)):
Add 'MakePropertyWritable' to the IScriptEnvironment (CPP interface), avs_make_property_writable (C interface)
Add 'VideoFrame::IsPropertyWritable' (CPP interface), avs_is_property_writable (C interface)
(AviSynth interface version will be stepped to V9 in the release version?)

bool env->MakePropertyWritable(PVideoFrame *);
bool VideoFrame::IsPropertyWritable();

'MakePropertyWritable' is similar to 'MakeWritable' but it does not copy all bytes of the frame content in order to have a writable property set.

Reason: 'propSet' is a filter which does not alter frame content, but sets the given frame property in its each GetFrame.
So far it used MakeWritable to obtain a safely modifiable copy of frame properties, however - as a side-effect - full copy of frame content was performed.
(env->getFramePropsRW alone does not ensure a uniquely modifiable property set, it just obtains a pointer which can be used in the property setter functions)
(Note: frame properties of frames obtained by NewVideoFrame, MakeWritable and SubFrame are still safe to modify)
- Expr: when actual bit depth is too large for building LUT table, fallback to realtime mode.
lut_x 1D (realtime when 32 bit)
lut_xy 2D (realtime when 16 or 32 bits)
- Expr: allow 'f32' as internal autoscale target (was: i8, i10, i12, i14, i16 were accepted, only integers)
affects: 'scale_inputs' when "int", "intf", "all", "allf"
more on that (todo: refresh docs) http://avisynth.nl/index.php/Expr
- Expr: fix conversion factor (+correct chroma scaling) when integer-to-integer full-scale automatic range scaling was required
- New: Expr: new parameter integer 'lut'
integer 'lut' (default 0)
0: realtime expression
1: expression is converted to 1D lut (lut_x)
2: expression is converted to 2D lut (lut_xy)
Valid bit depths: lut=1 : 8-16 bits. lut=2 : 8-14 bits. Note: a 14 bit 2D lut needs (2^14)*(2^14)*2 bytes buffer in memory per plane (~1GByte)
In lut mode some keywords are forbidden in the expression: sx, sy, sxr, syr, frameno, time, relative pixel addressing

[Dogway]

:thanks:

Everything seems to be working fine now. A bit sad because I lost one month of notes and drafts (the AvsPmod session file got cleansed : (

In big scripts like ex_binarize(mode="otsu") I don't see speed improvements but probably this is expected.

While updating the functions I got into a situation where the yexpr needs "intf" scale_inputs, and uexpr/vexpr requires "none" (or for 32-bits "none" and "floatUV" respectively). I could work around this by for example adding i16 (for 16-bit inputs) to the chroma expr, but this was slow so I baked the expression when 32-bit is being fed in.

# normalize to full range and back for limited range sources
# works fine over int HBD types and 32-bit float.
rangePCc = tv ? "f32 x 255 240 / *" : "f32 x"
rangeTVc = tv ? "240 255 / * " : " "

Another issue I found is ExtractU() sometimes doesn't work for float inputs.

[gispos]

A bit sad because I lost one month of notes and drafts (the AvsPmod session file got cleansed : (

Due to an error in the program? or how did that happen?
If the files are so important (one month of work) then the best thing to do is to make backup copies of them.

[Dogway]

I think it was my fault since I didn't enable avspmod's backups, but I'm not sure it works as I think it does.
I had a session from 24th of October, that's something.
I simply was writing something somewhere while avspmod launched, some letter got typed into the script editor and the last session got reset.
Is there an option for the current session to be stored as a "ghost" file (like Office does) until avspmod is closed?

[gispos]

I think it was my fault since I didn't enable avspmod's backups, but I'm not sure it works as I think it does.
I had a session from 24th of October, that's something.
I simply was writing something somewhere while avspmod launched, some letter got typed into the script editor and the last session got reset.
Is there an option for the current session to be stored as a "ghost" file (like Office does) until avspmod is closed?
The _last_session_.ses is overwritten with the current session.
If the backup is active, It will be overwritten every time the script has changed and the clip is re-initialized.
Hence the name 'last_session' :)

So if a session is important to you, you have to save this session yourself 'manually' with your own name.
A session you have saved yourself is never changed.
A shadow copy is therefore unnecessary, but I will think about whether it makes sense to save the currently opened '_last_session_.ses'.

Sorry Ferenc, I've finished. :)

[Dogway]

Manual save manual load, too manual for a program.

@pinterf: I noticed pixel addressing isn't being scaled with 'scale_inputs', at least for f32. Is it possible to include scaling the clip before pixels are being fetched?
Concerning 'lut' I didn't test but maybe it would come handy if it works when expression is scaled to supported bitdepth.
Also noticed there isn't a bool frame property, not sure if there's a reason for this.

[pinterf]

scaling for pixels obtained from rel.addressing: good catch, done in my work copy.
bool frame properties do not exist (they did not exist in VapourSynth), one must use integers for that purpose.

[pinterf]

@gispos: when you'd like to recognize Avisynth versions which have a fixed C interface for getting frame properties, you can do that since latest 3.7.1 test32.
See changelog for an example how to identify plain v8, then 8.1 or future 9 interface versions.

[StainlessS]

one must use integers for that purpose.
One advantage that tactic might have over bool is,
if originally some property defined as bool and you decide it aint enough and you want change to int ...
no longer a prob.

[Dogway]

It can be applied to _Interlaced, _ColorRange (there's also "extended" but oh well), _SceneChange, etc. Numbers imply several options and also forces you check against a number if either true or false.

[StainlessS]

That is true, [life's a bitch - and then you die]

[pinterf]

Avisynth+ 3.7.1 test build 33 (20211205) (https://drive.google.com/uc?export=download&id=1VxaMZaCee9kddzGOJ_0DTTJLgd8b4pSr)
I think it's time to make a feature freeze.
20211205 WIP
------------
- New array modifier function: ArraySet

For memo here is the list of avaliable array manipulator functions
- ArrayAdd - append
- ArrayDel - delete at position
- ArrayIns - insert before position
- ArraySet - replace at position

ArrayIns
^^^^^^^^

ArrayIns(array_to_mod, value_to_insert, index1 [, index2, index3...])

Insert a value into an array or into its subarray.
Returns a new array with value_to_insert inserted into array_to_mod (1D array) or array_to_mod[index1 (, index2, index3...)] (multi-dimensional array)
The indexes point to the insertion point. Index 0 will insert at the beginning of the array.
Index (ArraySize) will insert after the last element (same as ArrayAdd - append)
Original array (as with the other functions) remains untouched.

ArrayAdd
^^^^^^^^

ArrayAdd(array_to_mod, value_to_append [, index1, index2, index3...])

Appends value to the end of an array or its subarray
Returns a new array with value_to_append appended to array_to_mod (1D array) or array_to_mod[index1 (, index2, index3...)] (multi-dimensional array).
Original array (as with the other functions) remains untouched.

ArrayDel
^^^^^^^^

ArrayDel(array_to_mod, index1 (, index2, index3...])

Returns a new array in which the requested position was deleted.
Original array (as with the other functions) remains untouched.

ArraySet
^^^^^^^^

ArraySet(array_to_mod, replacement_value, index1 [, index2, index3...])

Returns a new array with array_to_mod[index1 (, index2, index3...)] = replacement_value
Original array (as with the other functions) remains untouched.

- Array modifier functions to allow multidimensional subarray indexes

Example:

ColorbarsHD()
# array indexes are zero based
a = []
a=ArrayAdd(a,[1,2]) # [[1,2]]
a=ArrayIns(a,3,0) # [3,[1,2]]
a=ArrayAdd(a,"s1") # [3,[1,2],"s1"]
a=ArrayAdd(a,"s2") # [3,[1,2],"s1","s2"]
a=ArrayDel(a,2) # [3,[1,2],"s2"]
a=ArraySet(a,"g",1,0) # [3,["g",2],"s2"]
a=ArrayAdd(a,"h",1) # [3,["g",2,"h"],"s2"]
a=ArrayAdd(a,[10,11,12],1) # append to (1) -> [3,["g",2,"h",[10,11,12]],"s2"]
a=ArrayDel(a,1,3,0) # del from (1,3,0) -> [3,["g",2,"h",[11,12]],"s2"]
a=ArrayAdd(a,"added") # [3,["g",2,"h",[11,12]],"s2","added"]
a=ArrayAdd(a,["yet","another","sub"]) # [3,["g",2,"h",[11,12]],"s2","added",["yet","another","sub"]]
x=a[0] #3
x=a[1,0] #g
x=a[1,2] #h
x=a[1,3,1] #12
x=a[3] #"added"
x=a[4,1] #"another"
SubTitle("x = " + String(x) + " Size=" + String(a.ArraySize()))

- Expr: allow auto scaling effect on pixels obtained from relative addressing
- ConvertBits: ordered dither: possible to dither down with more than 8 bits difference like in
Clip16.ConvertBits(8, dither=0, dither_bits=4)
Such conversion is made in two phases. First the clip is converted to (dither_bits+8) bits; in the above example it is 12.
If the temporary bit depth would be odd (no 9 or 11 bit support in Avisynth+) then it is made even.
bit depth that differs in only 8 bits for the target. Then this intermediate clip is converted to the required end target.
- Quicker ClearProperties and CopyProperties filters (by using MakePropertyWritable instead of MakeWritable).

[FranceBB]

Thanks for the new version!


I think it's time to make a feature freeze.

Oh, so this is gonna be the extensive testing part and Test 33 is the release candidate before the stable release?! :D
And perhaps the stable will be released for Christmas?

https://c.tenor.com/6gyNP9vCJH4AAAAC/its-a-christmas-miracle-christmas-miracle.gif

[gispos]

Manual save manual load, too manual for a program.

:rolleyes: It is automatically backed up and restored, and if you want to back up something, no program can only do it with good talk.
But I can also save all sessions with the date... and nobody looks through it afterwards and knows what is in the files.

There will be a backup copy of the last session and a previous one.

@gispos: when you'd like to recognize Avisynth versions which have a fixed C interface for getting frame properties, you can do that since latest 3.7.1 test32.
See changelog for an example how to identify plain v8, then 8.1 or future 9 interface versions.
Thanks Ferenc, had already read that.
And since this is not present in older versions, I had already determined it this way:
From header version 8 only the matrix is read (if necessary with Eval), and from version 3.71 the properties are also read.

Edit: Where can I find your integer properties value conversion to string?

[Dogway]

pinterf, do you know if Clip32.Expr(a,b,"x y - abs 128 > 255 0 ?", lut=2, scale_inputs="all") is a compatible lut expression? I think this is not documented.

[pinterf]

pinterf, do you know if Clip32.Expr(a,b,"x y - abs 128 > 255 0 ?", lut=2, scale_inputs="all") is a compatible lut expression? I think this is not documented.
Valid, because when the given bit depth is not available for LUT mode it fallbacks to realtime (lut=0). You said you didn't like error messages

- Expr: when actual bit depth is too large for building LUT table, fallback to realtime mode.
lut_x 1D (realtime when 32 bit)
lut_xy 2D (realtime when 16 or 32 bits)

[Dogway]

Sure, it's not about error messages but if it is possible to use lut calculations on scaled_inputs. The above for example could use an 8-bit lut table.

[StainlessS]

Doggie, Probably slower than realtime. [I think]

[Dogway]

test33 crashes avspmod with a simple bilinearresize(), same in avsmeter, has this to do with the interface change in test32? Strange because I was using test32 without issues.

[qyot27]

[Some? Most?] of the filter sources that have x86 intrinsics in them are in the process of being de-duplicated. It would seem there's a regression introduced in there somewhere, since a build with the intrinsics disabled works as expected:
$ mpv test.avs
mpv: ../avs_core/filters/resample.cpp:424: FilteredResizeH::FilteredResizeH(PClip, double, double, int, ResamplingFunction*, IScriptEnvironment*): Assertion `0' failed.
Aborted (core dumped)
(gdb) bt
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737101604416)
at pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=140737101604416)
at pthread_kill.c:80
#2 __GI___pthread_kill (threadid=140737101604416, signo=signo@entry=6)
at pthread_kill.c:91
#3 0x00007ffff5e5f476 in __GI_raise (sig=sig@entry=6)
at ../sysdeps/posix/raise.c:26
#4 0x00007ffff5e457b7 in __GI_abort () at abort.c:79
#5 0x00007ffff5e456db in __assert_fail_base
(fmt=0x7ffff5ff9770 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7fffd29785a1 "0", file=0x7fffd2978580 "../avs_core/filters/resample.cpp", line=424, function=<optimized out>) at assert.c:92
#6 0x00007ffff5e56e26 in __GI___assert_fail
(assertion=0x7fffd29785a1 "0", file=0x7fffd2978580 "../avs_core/filters/resample.cpp", line=424, function=0x7fffd2978518 "FilteredResizeH::FilteredResizeH(PClip, double, double, int, ResamplingFunction*, IScriptEnvironment*)")
at assert.c:101
#7 0x00007fffd25366c3 in FilteredResizeH::FilteredResizeH(PClip, double, double, int, ResamplingFunction*, IScriptEnvironment*) ()
at /usr/local/lib/libavisynth.so
#8 0x00007fffd25397e7 in FilteredResize::CreateResizeH(PClip, double, double, int, ResamplingFunction*, IScriptEnvironment*) ()
--Type <RET> for more, q to quit, c to continue without paging--
at /usr/local/lib/libavisynth.so
#9 0x00007fffd2539d9f in FilteredResize::CreateResize(PClip, int, int, AVSValue const*, ResamplingFunction*, IScriptEnvironment*) ()
at /usr/local/lib/libavisynth.so
#10 0x00007fffd253a182 in FilteredResize::Create_BilinearResize(AVSValue, void*, IScriptEnvironment*) () at /usr/local/lib/libavisynth.so
#11 0x00007fffd223c19b in FilterConstructor::InstantiateFilter() const ()
at /usr/local/lib/libavisynth.so
#12 0x00007fffd2276b73 in ScriptEnvironment::Invoke_(AVSValue*, AVSValue const&, char const*, Function const*, AVSValue const&, char const* const*, InternalEnvironment*, bool) () at /usr/local/lib/libavisynth.so
#13 0x00007fffd22822a2 in ThreadScriptEnvironment::Invoke_(AVSValue*, AVSValue const&, char const*, Function const*, AVSValue const&, char const* const*) ()
at /usr/local/lib/libavisynth.so
#14 0x00007fffd22d88a0 in ExpFunctionCall::Evaluate(IScriptEnvironment*) ()
at /usr/local/lib/libavisynth.so
#15 0x00007fffd22d5037 in ExpExceptionTranslator::Evaluate(IScriptEnvironment*)
() at /usr/local/lib/libavisynth.so
#16 0x00007fffd22d53cf in ExpLine::Evaluate(IScriptEnvironment*) ()
at /usr/local/lib/libavisynth.so
#17 0x00007fffd22d4f8e in ExpSequence::Evaluate(IScriptEnvironment*) ()
at /usr/local/lib/libavisynth.so
#18 0x00007fffd22d4e25 in ExpRootBlock::Evaluate(IScriptEnvironment*) ()
--Type <RET> for more, q to quit, c to continue without paging--
at /usr/local/lib/libavisynth.so
#19 0x00007fffd22dbe15 in Eval(AVSValue, void*, IScriptEnvironment*) ()
at /usr/local/lib/libavisynth.so
#20 0x00007fffd223c19b in FilterConstructor::InstantiateFilter() const ()
at /usr/local/lib/libavisynth.so
#21 0x00007fffd2276ae5 in ScriptEnvironment::Invoke_(AVSValue*, AVSValue const&, char const*, Function const*, AVSValue const&, char const* const*, InternalEnvironment*, bool) () at /usr/local/lib/libavisynth.so
#22 0x00007fffd2281c03 in ThreadScriptEnvironment::Invoke(char const*, AVSValue, char const* const*) () at /usr/local/lib/libavisynth.so
#23 0x00007fffd22dcec9 in Import(AVSValue, void*, IScriptEnvironment*) ()
at /usr/local/lib/libavisynth.so
#24 0x00007fffd223c19b in FilterConstructor::InstantiateFilter() const ()
at /usr/local/lib/libavisynth.so
#25 0x00007fffd2276ae5 in ScriptEnvironment::Invoke_(AVSValue*, AVSValue const&, char const*, Function const*, AVSValue const&, char const* const*, InternalEnvironment*, bool) () at /usr/local/lib/libavisynth.so
#26 0x00007fffd2281c03 in ThreadScriptEnvironment::Invoke(char const*, AVSValue, char const* const*) () at /usr/local/lib/libavisynth.so
#27 0x00007fffd22aa949 in avs_invoke () at /usr/local/lib/libavisynth.so
#28 0x000055555574e4ba in ()
#29 0x0000555555c80201 in ()
#30 0x0000555555815984 in ()
--Type <RET> for more, q to quit, c to continue without paging--
#31 0x000055555580dc0d in ()
#32 0x000055555580e49a in ()
#33 0x000055555586f716 in ()
#34 0x00007ffff5eb1927 in start_thread (arg=<optimized out>)
at pthread_create.c:435
#35 0x00007ffff5f419e4 in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

[tormento]

I am having crashes too with some scripts and not others.

The following

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_58 Ieri oggi domani\ieri.dgi",ct=132,cb=132,cl=0,cr=0)
CompTest24(1)
ConvertBits(16)
SMDegrain (tr=3, thSAD=300, refinemotion=true, contrasharp=false, PreFilter=6, plane=4, chroma=true)
fmtc_bitdepth (bits=8,dmode=8)
Prefetch(6)


gives me

System exception - Access Violation
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.9d~Dogway.avsi, line 901)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.9d~Dogway.avsi, line 228)

[FranceBB]

test33 crashes avspmod with a simple bilinearresize(), same in avsmeter, has this to do with the interface change in test32? Strange because I was using test32 without issues.

Yep, I've just tried with a simple:

ColorBars(848, 480, pixel_type="YV12")

BilinearResize(1920, 1080)

crashes immediately.

Even worse, getting rid of BilinearResize() makes it crash just as well, for instance:

ColorBars(848, 480, pixel_type="YV12")

makes AVSPmod x64 crash.

Trying with:


SetMaxCPU("none")

ColorBars(848, 480, pixel_type="YV12")


works

https://i.imgur.com/wI6IW7z.png

same goes for:


SetMaxCPU("none")

ColorBars(848, 480, pixel_type="YV12")

Spline64Resize(1280, 720)


https://i.imgur.com/2xZHoP8.png

and even more complicated scripts like:


SetMaxCPU("none")
FFVideoSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\SIC_Preview_REC709_20211124_de.mov")

DeBilinearResizeMT(720, 480)
t = QTGMC( Preset="Slower", InputType=2, ProgSADMask=1.0, ShutterBlur=3)
b = QTGMC( Preset="Slower", InputType=3, PrevGlobals="Reuse" )
Repair( t, b, 1 )
mt_convolution("1","1 2 1",chroma="process")
Blur(0.0, 1.58).Blur(0.0, 1.58).Blur(0.0, 1.58).Blur(0.0, 1.58)
dfttest(sigma=64, tbsize=1, lsb_in=false, lsb=false, Y=true, U=true, V=true, opt=0, dither=0)
dfttest(sigma=64, tbsize=1, lsb_in=false, lsb=false, Y=true, U=true, V=true, opt=0, dither=0)
super = MSuper(pel=2, sharp=1)
bv1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
fv1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
bv2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
fv2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super,bv1,fv1,bv2,fv2,thSADC=800, thSAD=800)
Spline64ResizeMT(2048, 858)


work just fine as long as there's SetMaxCPU("none")

https://i.imgur.com/3Z16znb.png


Hence confirming Stephen's theory about intrinsics being the cause of the crashes.

I'm gonna stick with Test 32 in all my servers.

[Dogway]

Yes, no problem, I was just confirming this was known. I installed test33 only to update all the filters that used pixel addressing and could benefit also from scale_inputs.

[pinterf]

Found a typo, resizer related. New build is in progress.
EDIT: I see you've found it as well. Sorry for the inconvenience.

[Dogway]

Found a typo, resizer related. New build is in progress.
EDIT: I see you've found it as well. Sorry for the inconvenience.

Thank you.
BTW do you know if CombinePlanes can be optimized?
Currently the old YtoUV() is faster by some 15%, probably same with mergechroma() and mergeluma().

Also StainlessS stated (https://forum.doom9.org/showthread.php?p=1958765#post1958765)that using 'lut' with scaled_inputs (ie. lut over scaled down to 8-bit expression) would be slower than realtime.

[pinterf]

Rebuild.
Avisynth+ 3.7.1 test build 34 (20211208) (https://drive.google.com/uc?export=download&id=1nVYll8WKoHBYOjh53AxPWnZLXxkeIpzi)

[pinterf]

BTW do you know if CombinePlanes can be optimized?
Currently the old YtoUV() is faster by some 15%, probably same with mergechroma() and mergeluma().

Do yo mean the three input clips version, where Y is obtained from the 1st clip for example?
Maybe. The in-clip plane shuffles are optimized if I remember correctly. The other cases make a new empty frame and copy source planes into that. In YtoUV the original Y plane could be kept and only U and V is copied actually.

Also StainlessS stated (https://forum.doom9.org/showthread.php?p=1958765#post1958765)that using 'lut' with scaled_inputs (ie. lut over scaled down to 8-bit expression) would be slower than realtime.
I've seen it, yes, good question, which can only be proved or discarded if I actually implement it. Tempting :)

[Dogway]

Yes, actually I was testing with the following:
Y = ExtractY()
U = ExtractU()
V = ExtractV()
# some per plane filtering
YtoUV(U,V,Y)
# CombinePlanes(Y,U,V,planes="YUV",sample_clip=a)
But probably there are more cases for optimizations.

Thanks for looking into that.

[VoodooFX]

Can CombinePlanes combine directly to YV12 when U and V clips are half resolution of Y?

[StainlessS]

Originally Posted by Dogway View Post
Also StainlessS stated that using 'lut' with scaled_inputs (ie. lut over scaled down to 8-bit expression) would be slower than realtime.
That's just a 'gut feeling'.

[Dogway]

Can CombinePlanes combine directly to YV12 when U and V clips are half resolution of Y?

Yes, you just need to provide the "pixel_type" or "sample_clip".
Actually even Expr() can combine planes if you declare the "format" type, but it might use the slow CombinePlanes() code path, so maybe pinterf can also look into that.

That's just a 'gut feeling'.

Ah well, I have you in very high regard lol

[pinterf]

Yes, actually I was testing with the following:
Y = ExtractY()
U = ExtractU()
V = ExtractV()
# some per plane filtering
YtoUV(U,V,Y)
# CombinePlanes(Y,U,V,planes="YUV",sample_clip=a)
But probably there are more cases for optimizations.

Thanks for looking into that.
How did you measure? YtoUV is not any quicker with this script. Sometimes CombinePlanes is faster but mostly they are even, any difference is probably a measurement glitch. Both are running at ~48000 fps when processing a Colorbars(pixel_type="YV12"). That 48000 fps also means that this is too quick to be a bottleneck function, making it even 50% quicker has hardly any measurable effect on a real script.

EDIT:
the provided sample was not perfect: you don't need to specify a third Y parameter because it is then copied as well and the whole thing is perfectly identical to present CombinePlanes.
Colorbars(pixel_type="YV12")
Y = ExtractY()
U = ExtractU()
V = ExtractV()
# some per plane filtering
YtoUV(U,V) # instead of YtoUV(U,V,Y)


Now I see the speed difference (which probably does not affect a bigger script's speed). But anyway, let's focus on perfection.

EDIT2:
The speed difference was because I was stupid and omitted the Y parameter from the YtoUV example.

Finally. These are giving identical results. With identical speed. They are working the same way internally. They create a new empty frame then copy source planes bytes one by one.
Colorbars(pixel_type="YV12")
a=last
Y = ExtractY()
U = ExtractU()
V = ExtractV()
# some per plane filtering
#YtoUV(U,V,Y)
CombinePlanes(Y,U,V,planes="YUV", source_planes="YYY",sample_clip=a)

[Dogway]

Yes sorry for the delay. Indeed with synthetic or even staged tests I get almost same speed (with a slight edge on YtoUV), in any case my tests were with the rework of ex_gaussianblur() which uses by default mergeluma() (UV=2). Here (https://pastebin.com/h8ZLyChG)an almost finished version of the filter.
Testing with 1080
setmemorymax()
DGSource("1080psource.dgi")
ConvertBits(16)
ex_gaussianblur(6) # 400fps (340fps with CombinePlanes)
Prefetch(4) # This seems ideal value for scalers on a 4/8 CPU


I will try to come up with a simplified script.

EDIT: Ok, this is a simpler script but still doesn't show the 15% speed difference
Y = ExtractY().BicubicResize(round(width()/1.5),round(height()/1.5))
Y = Y.BicubicResize(width(),height())
mergeluma(y)

I thought the crop in my filter didn't go well with CombinePlanes, but testing with pad=false (no padding) showed the same if not more speed gap. So I can only think GaussResize() doesn't go well with CombinePlanes.

EDIT2: Ok, here's a stripped down test that starts to show the issue (still not quite 15% diff) probably cropping in-between makes things worse:
a=last
ExtractY().BilinearResize(344,204)
GaussResize(a.width(),a.height(),p=9)
#mergeluma(a,last)
CombinePlanes(last,a,planes="YUV",sample_clip=a)

[VoodooFX]

Tested and got +11% speed with CombinePlanes, instead of upsizing chroma then downsizing it when using MergeLuma.

[Dogway]

This shows a +6% speed for mergeluma():
setmemorymax()
DGSource("1080psource.dgi")
ConvertBits(16)
a=last
w0=width() h0=height() p=64

ExtractY().BilinearResize(344,204, src_left=-p, src_top=-p, src_width=w0+p+p, src_height=h0+p+p)
GaussResize(w0+p+p,h0+p+p,p=9)
crop (p, p, -p, -p)
mergeluma(a,last) # 384
#CombinePlanes(last,a,planes="YUV",sample_clip=a) # 363
Prefetch(4)
When used with ex_GaussianBlur() (for unknown reasons) it can reach over 10~15%, my linked script only needs ResizersPack for nmod()

[VoodooFX]

What if you remove decoding "bottleneck" with BlankClip(last) after DGSource? I didn't used Prefetch.

[Dogway]

With BlankClip(last) it's same speed using Prefetch(4) but I always test in context with real world material, maybe CombinePlanes() is superfast on its own, but has a harder time when frame is served in a specified manner.
To note I also experienced these slowdowns in TransformsPack which is a different monster, there I don't crop nor resize, simply do per-plane matrix operations with Expr() and then Combine.
YUV444 source # to don't add more YUV -> RGB overhead
m=RGB_to_XYZ("sRGB",true)
MatrixClip2(m)
Prefetch(4)

function MatrixClip2 ( clip clp, float_array mat, string "fmt_o") {

rgb = isRGB(clp)
fmt_o = Default(fmt_o, rgb ? "RGB" : "YUV")

CLPa = ExtractClip(clp)

# clip · 3x3
C = DotClipA(CLPa,[mat[0],mat[3],mat[6]])
L = DotClipA(CLPa,[mat[1],mat[4],mat[7]])
P = DotClipA(CLPa,[mat[2],mat[5],mat[8]])

# YtoUV(L, P, C) } # 141
CombinePlanes(C, L, P, planes=fmt_o) } # 139


With prefetch(6) this doesn't happen, they are same speed, but at the cost of a lower speed. The optimal Prefetch here again is 4, at least for my CPU which sees both methods increase speed albeit one more than the other.


On another note, in AddBorders() I was about to add "color" to the Alpha plane of RGB, there's no such an option right?

[pinterf]

This shows a +6% speed for mergeluma():
setmemorymax()
DGSource("1080psource.dgi")
ConvertBits(16)
a=last
w0=width() h0=height() p=64

ExtractY().BilinearResize(344,204, src_left=-p, src_top=-p, src_width=w0+p+p, src_height=h0+p+p)
GaussResize(w0+p+p,h0+p+p,p=9)
crop (p, p, -p, -p)
mergeluma(a,last) # 384
#CombinePlanes(last,a,planes="YUV",sample_clip=a) # 363
Prefetch(4)
When used with ex_GaussianBlur() (for unknown reasons) it can reach over 10~15%, my linked script only needs ResizersPack for nmod()
Thanks, I got it, checked how MergeLuma works.
When the input clip is not referenced by other clips in the filter chain (there is exactly one reference on it, technically "IsWritable") MergeLuma can obtain a write-permission directly, sparing the need of copying Y plane. I'm gonna check this on CombinePlane.

EDIT:
MergeLuma is called w/o passing weight, so weight is 1.0.
This means the luma of second clip (last) is kept 100%.
This also means that the smaller U and V planes (4:2:0) are needed to be copied, saving time.
But:
this is not the case here, since 'last' is a luma-only Y and cannot accept U+V copy.
In this MergeLuma example all three planes are copied to a brand new empty frame which is the worst case.

[Dogway]

Thanks for looking into it. Not sure what that means, that MergeLuma (CombinePlanes regardless) can work even faster? In any case good you could spot it because a 15% speed difference isn't normal with the example of ex_gaussianblur().

I think I found another bug, masktools2 related though while trying to match my ex_lutspa() version:
mt_lutspa(mode="relative", expr="x range_max *",U=128,V=128)

Outputs 65501 as YPlaneMax for 16-bit instead of 65535.

And some issues with internal filters with color arguments like BlankClip, Blackness, Letterbox, AddBorders and FadeXXX. If you specifiy white (color_yuv=$ffffff or $ff8080) output is 65280 for 16-bit, not suitable for masks. I think a solution would be to map automatically 0~15 and 236~255 values to full scale if a fulld argument is not desired.

[pinterf]

Thanks for looking into it. Not sure what that means, that MergeLuma (CombinePlanes regardless) can work even faster? In any case good you could spot it because a 15% speed difference isn't normal with the example of ex_gaussianblur().

I think I found another bug, masktools2 related though while trying to match my ex_lutspa() version:
mt_lutspa(mode="relative", expr="x range_max *",U=128,V=128)

Outputs 65501 as YPlaneMax for 16-bit instead of 65535.

And some issues with internal filters with color arguments like BlankClip, Blackness, Letterbox, AddBorders and FadeXXX. If you specifiy white (color_yuv=$ffffff or $ff8080) output is 65280 for 16-bit, not suitable for masks. I think a solution would be to map automatically 0~15 and 236~255 values to full scale if a fulld argument is not desired.
Aside from the fact that similar filters sometimes report different speeds I've specialized two cases for CombinePlanes, but I'm gonna test them more.

- when the 1st clip's Y plane can be kept - when selected stars has sysygy and Mars stops its retrograde motion
Condition: 1st clip has the same format as the output

- when the 2nd clip's UV planes can be kept - condition as at the first case :)
Condition: 2nd clip has the same format as the output

Yep, colors are simply treated as rec601 limited range ones.
For this reason BlankClip has an exact color array syntax for passing exact, unscaled color values.
For other filters where color is given by a single integer number my first statement holds.

Masktools: yes, there are things to backport from Expr, just for the sake of make consistency between them. The mt_lutspa you mentioned maybe not a such difference but a simple difference, but hey, the more they resemble each other the happier world.

[Dogway]

Yes, no problem, I was working from the other side, making Expr wrappers work more like masktools2, felt it was nice to report even if I don't use masktools much.
What array syntax do you mean? I only see int color, and int color_yuv the later has an extra option (http://avisynth.nl/index.php/Colors)for integer color definition instead of hexadecimal, but testing with max value 16777215 still gives me 65280.
Currently the only option is a post processing to scale values from bitshift to fullscale, but this makes things slower.

[pinterf]

There is a colors array parameter. (signature: [colors]f+)
Now that script arrays are part of Avisynth+, Wiki could be refreshed with it. It was my test parameter but seems it was missed from documentation, it appeared only in an early change log.

[pinterf]

Integer and hexadecimal covers the very same 32 bit integer in the background, former is in decimal radix, latter is written is hexadecimal notation.

[Dogway]

ooooooh wonderful! colors=[65535,65535,65535]

By the way I was refactoring ex_bs() to make it behave like avisynth does but found some inconsistencies or maybe I'm mistaken.

Taking as example 10-bit value 514, which is range_half in full scale and converting to narrow range bitshift scale
# HBD Scaling Range compression
# fs bs fs-1 bs bs bs
# ((((514*(65280/1023)) *56064)/65280)+4096) = 32265 (ConvertBits(16,fulls=true,fulld=false)) 32265.009/256 = 126.03519
# fs bs+1 fs bs bs bs
# ((((514*(65281/1024)) *56064)/65280)+4096) = 32237 (Alternative) Satisfies: (32237.93/256 = 125.929418 = ((128*219)/255)+16) = 125,92941176470588235294117647059

Another example. Simply full range scaling the same 10-bit value to 16-bit full scale
# fs fs fs-1
# 514*65536/1023 = 32928 (ConvertBits(16,fulls=true,fulld=true))
# fs fs fs
# 514*65536/1024 = 32896 (Alternative) Satisfies n*256+n = 128*256+128

For an exact roundtrip (saving rounding conventions) you have to repeat process in the opposite direction.
For example I found that doing range conversion first in higher bitdepth is preferable otherwise convert up in bitdepth before range conversion.

Converting 512 from TV levels bitshift scale to 16-bit PC levels full scale

# bs fs bs+1 fs fs fs
# (((512*(65536/1021)) -4112)/56283)*65535 = 33478,684609501831404
# fs bs fs bs bs+1 bs
# (((33478.68461*56283)/65535)+4112)*(1021/65536) = 512

[Boulder]

@pinterf, do you see anything Avisynth+ core related with this issue? Settings some points values manually works, but inputting a GIMP curve file fails with an access violation. It seems to pass inputting the luma points but crashes with the next set. I also tried installing the Avisynth+ build from AviSynthPlus_3.7.0_20210111.exe but it didn't help.

https://forum.doom9.org/showthread.php?p=1959135#post1959135

[DTL]

Finally found what was wrong with first attempt to use large pages buffers for MVtools processing - it is cache set overloading because of limited capacity of N-ways set-associative caches for some memory access patterns. It was especially critical to MVtools with lots (>8) equal sized buffers processing of single x,y coordinates data from many frames at once (in MDegrainN operation at least).

This hardware-limit issue also described in Intel Software Optimization manual:
3.6.7 Capacity Limits and Aliasing in Caches
There are cases in which addresses with a given stride will compete for some resource in the memory
hierarchy.
Typically, caches are implemented to have multiple ways of set associativity, with each way consisting of
multiple sets of cache lines (or sectors in some cases). Multiple memory references that compete for the
same set of each way in a cache can cause a capacity issue. There are aliasing conditions that apply to
specific microarchitectures. Note that first-level cache lines are 64 bytes. Thus, the least significant 6 bits
are not considered in alias comparisons.
3.6.7.1 Aliasing Cases in the Pentium® M, Intel® Core™ Solo, Intel® Core™ Duo and Intel® Core™
2 Duo Processors
Pentium M, Intel Core Solo, Intel Core Duo and Intel Core 2 Duo processors have the following aliasing
case:
• Store forwarding — If a store to an address is followed by a load from the same address, the load
will not proceed until the store data is available. If a store is followed by a load and their addresses
differ by a multiple of 4 KBytes, the load stalls until the store operation completes.
Assembly/Compiler Coding Rule 49. (H impact, M generality) Avoid having a store followed by a
non-dependent load with addresses that differ by a multiple of 4 KBytes. Also, lay out data or order
computation to avoid having cache lines that have linear addresses that are a multiple of 64 KBytes
apart in the same working set. Avoid having more than 4 cache lines that are some multiple of 2 KBytes
apart in the same first-level cache working set, and avoid having more than 8 cache lines that are some
multiple of 4 KBytes apart in the same first-level cache working set.
When declaring multiple arrays that are referenced with the same index and are each a multiple of 64
KBytes (as can happen with STRUCT_OF_ARRAY data layouts), pad them to avoid declaring them contiguously. Padding can be accomplished by either intervening declarations of other variables or by artificially
increasing the dimension.
User/Source Coding Rule 8. (H impact, ML generality) Consider using a special memory allocation
library with address offset capability to avoid aliasing. One way to implement a memory allocator to
avoid aliasing is to allocate more than enough space and pad. For example, allocate structures that are
68 KB instead of 64 KBytes to avoid the 64-KByte aliasing, or have the allocator pad and return random
offsets that are a multiple of 128 Bytes (the size of a cache line).
User/Source Coding Rule 9. (M impact, M generality) When padding variable declarations to
avoid aliasing, the greatest benefit comes from avoiding aliasing on second-level cache lines,
suggesting an offset of 128 bytes or more.
4-KByte memory aliasing occurs when the code accesses two different memory locations with a 4-KByte
offset between them. The 4-KByte aliasing situation can manifest in a memory copy routine where the
addresses of the source buffer and destination buffer maintain a constant offset and the constant offset
happens to be a multiple of the byte increment from one iteration to the next.

So corrected version of largepage allocation is: (in DeviceManager.cpp)

virtual BYTE* Allocate(size_t size, int margin)
{
#define RAND_OFFSET_MAX 256
/* overhead about 1.6% max over 2 MB largepage, support about 256 different allocations without addresses of same offset inside frame hit same cache set. */

#define L2L3_CACHE_LINE_SIZE 128

size += margin;
// return new BYTE[size + 16];
// large pages
// to prevent cache set overloading when accessing same frame regions - add random 128-bytes sized offset to different allocations
size_t random = rand();
random *= RAND_OFFSET_MAX;
random /= RAND_MAX;
random *= L2L3_CACHE_LINE_SIZE;

SIZE_T stLPGranularity = GetLargePageMinimum();
size_t iNumLPUnits = (size + random) / stLPGranularity;
SIZE_T stSizeToAlloc = (iNumLPUnits + 1) * stLPGranularity;

BYTE* data = (BYTE*)VirtualAlloc(0, stSizeToAlloc, MEM_LARGE_PAGES | MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE);
DWORD error = GetLastError();

if (error == ERROR_NO_SYSTEM_RESOURCES)
env->ThrowError("CPUDevice: LargePages alloc error. Insufficient system resources exist to complete the requested service.\n \
While allocating %d pages of %d size.\n", iNumLPUnits+1, stSizeToAlloc);

if (error != ERROR_SUCCESS)
env->ThrowError("CPUDevice: LargePages alloc error. GetLastError returned: %d\n", error);

data += random; // add random L2L3_CACHE_LINE_SIZE-byte granulated offset up to RAND_OFFSET_MAX*L2L3_CACHE_LINE_SIZE size to the pointer, \
need to be cleared at freeing
return data;
#endif
}

virtual void Free(BYTE* ptr)
{
if (ptr != nullptr) {
// large pages here

// clear random offset and feed 2M-granularity address
ptr = (BYTE*)((uint64_t)ptr >> 21);
ptr = (BYTE*)((uint64_t)ptr << 21);

VirtualFree(ptr, 0, MEM_RELEASE);
// delete[] ptr;
}
}


So it is ready for next round of testing. Make testbuild based on 3.7 sources - https://github.com/DTL2020/AviSynthPlus-LP-mod/releases/tag/3.7-01

This issue may also hits 'standard allocations' so it is good to track used allocated addresses and if found dangerous combinations - adjust it somehow to prevent competing for the same cache set at Avisynth filters working (though multi-frames accessing filters may be rare enough - like temporal denoisers ?).

The size of possible performance penalty - about 60% in fps.

The users of other AVS filters may suffer from this issue if using multi-line accessing (with lines > ways of cache) and data buffer pitch (one or multi lines) is close to 4096 (+-32 or +-64 ?). The solution is to pad frame width with some value to move pitch value outside critical values (around 4096 ?).
One example: MVtools frame width size 1920 and MSuper(hpad=64) - creates 1920+64+64=2048 pitch and each 2nd line vertical access hits same L1D cache set. With 8-ways cache after 8 hit (block height > 16) it become overloaded (no more free lines in the 8-lines cache set) and execution got visible performance penalty.

[gispos]

@Ferenc or he who knows, where can I find your integer properties value conversion to string?
I can't find the latest avisynth source on GitHub, is it available somewhere?

[pinterf]

@Ferenc or he who knows, where can I find your integer properties value conversion to string?
I can't find the latest avisynth source on GitHub, is it available somewhere?
You can find some (there is no Primaries and Transfer handling in avs+) in conditional_reader.cpp and for constants you can look into convert_helper.cpp. Maybe these constants would be moved into a separate header file which can be used by plugins developers.

[pinterf]

Taking as example 10-bit value 514, which is range_half in full scale
Why 514?

[Dogway]

Why 514?
ConvertBits(8)
Expr("range_half")
ConvertBits(10,fulls=true,fulld=true) # outputs 514

More:
128*2^(10-8) = 512
128*2^(10-8)+(2^(10-8))/2 = 514
257*2^(10-8)/2 = 514
128*2^(10-8) *256*2^(10-8)/255*2^(10-8) = 514
(255*2^(10-8))/2+(256*2^(10-8)-255*2^(10-8)) = 514
32896/2^(16-10) = 514

32896 is range_half in full scale (http://forum.doom9.org/showthread.php?p=1957551#post1957551):
"Special even quicker case: 8->16 bit fulls=true, fulld=true (simply *257)"
128*257 = 32896
128*2^(16-8)+(2^(16-8))/2 = 32896

Another issue is that the constants are not being mapped correctly when range property is full
ConvertBits(16,fulls=true)
Expr("range_half") # outputs 32768

[pinterf]

I see now, but the logic fails because your base value is 128 which is not true.
8 bit Y full scale range_half is not a precise thing since it must be 127.5, and rounded up to 128 so it cannot be the strict base of further calculations.
My note is that range_half in other than chroma planes has not that special meaning as in chroma, and - as beeing just a rounded value somewhere betwen the extremes - is not more special like e.g. 787.
In luma/rgb full range the only thing in you can be sure that the two extremes are 0 and (2^N )-1, all values in between are scaled proportionally.

[pinterf]

Another issue is that the constants are not being mapped correctly when range property is full
ConvertBits(16,fulls=true)
Expr("range_half") # outputs 32768
Expr does not use range information from frame property.
There are several Expr parameters which directly control that expression is full or limited when autoscaled feature is used. Not to mention the explicite scaleb and scalef usage.

[gispos]

I just found a problem with the newer avisynth's (don't know from which version) tested with 3.71 test 34

video = LWLibavVideoSource (SourceFile, cache = False)
audio = LWLibavAudioSource (SourceFile, cache = False)
audioDub(video, audio)
Spline36Resize(1920, 1036)
prefetch(2)

does not work with AvsPmod (C Interface ?), but works with e.g. Avisynth Version 3.62 test 6
If I add a 'last' or remove the prefetch it works.

Spline36Resize(1920, 1036)
last
prefetch(2)

Can anybody confirm this?

Edit:
If no external filter is used but only internal ones like resizer, crop, sharpen, Levels etc. avisynth gets stuck if prefetch used.
If I use an external filter or an older Avisynth, there are no problems with prefetch.

Edit2
3.71 test 12 is OK
3.71 test 25 does not work

Edit3
3.71 test 22 is the last version that works

[pinterf]

I just found a problem with the newer avisynth's (don't know from which version) tested with 3.71 test 34

video = LWLibavVideoSource (SourceFile, cache = False)
audio = LWLibavAudioSource (SourceFile, cache = False)
audioDub(video, audio)
Spline36Resize(1920, 1036)
prefetch(2)

does not work with AvsPmod (C Interface ?), but works with e.g. Avisynth Version 3.62 test 6
If I add a 'last' or remove the prefetch it works.

Spline36Resize(1920, 1036)
last
prefetch(2)

Can anybody confirm this?

Edit:
If no external filter is used but only internal ones like resizer, crop, sharpen, Levels etc. avisynth gets stuck if prefetch used.
If I use an external filter or an older Avisynth, there are no problems with prefetch.

Edit2
3.71 test 12 is OK
3.71 test 25 does not work

Edit3
3.71 test 22 is the last version that works
Works for me. I'd try installing latest vc++ redistributables?

[jpsdr]

@gispos: Not sure if it's significant, but what Windows version are you using ?

[gispos]

I think so, my last version is 14.30.30401.0 from 21.07.2021
Win10 x64

I just tried again, same result. There are no problems with 3.71 test 22.
I think the runtimes should be up to date with version 14.30.30401.0 Wouldn't like to install something over it again.
Unusual.

https://i.postimg.cc/4d6MTdSX/visual-c.jpg (https://postimg.cc/MMGt7WwL)

Edit:
It's probably not the latest, just found version 14.31.30818.0.
Will try it.

[gispos]

The latest runtimes are installed but unfortunately no changes.
3.71 test 22 is the last version that has no problems with prefetch for me.

Edit:
The newer Avisynth's behave very strangely with me.

No matter how many threads, it doesn't work
Spline36Resize(1280, 720)
prefetch(2)

It doesn't work with prefetch 2
NonlinUSM(z=3, pow=1.2, str=0.25, rad=6)
prefetch(2)

With prefetch 4 it works
NonlinUSM(z=3, pow=1.2, str=0.25, rad=6)
prefetch(4)

Edit2:
And it also seems to depend on the filter.
UnsharpMask (strength = 52, radius = 3, threshold = 4)
prefetch(2)

works, but not with prefetch(4)

Some things work with prefetch(8) but not with prefetch(6).

This is not the case with version 3.71 test 22 and older

[gispos]

I don't want to cause any stress now, maybe it's just my system, but it's strange how it behaves with the newer versions for me.
What kind of runtimes are needed exactly? As written, I installed the latest package.

[StainlessS]

GP, if you've got "avstp.dll" in plugins, remove it and re-test.

[cretindesalpes]

avstp.dll is not related to any of these functions.

[gispos]

I deleted the DLL years ago. And with earlier versions, earlier 3.71 test 23 it works.
No one else with the problem with AvsPmod?

I save the script and open it with AviSource then there are no problems, which is an indiez for a C Interface bug for me. At least it was so in the past.

[StainlessS]

Just 'clutching at straws' pussycat :)

[gispos]

All back. I'm so sorry
I tested an older AvsPmod version and thus no problems.:o
Now I'm the ass that gets stressed.

Sorry Ferenc :o

[gispos]

All back. I'm so sorry
I tested an older AvsPmod version and thus no problems.:o
Now I'm the ass that gets stressed.

Sorry Ferenc :o
I have to revise.
I had only once tested the older AvsPmod version and apparently a suitable combination of filters and prefetch caught that it seemed it to work.

But if I change filter and prefetch it does not work for most combinations.

[Dogway]

Pixel addressing seems to fluctuate a lot depending on prefetch value.
Using ex_expand(3) as an example which is a mix of pixel fetching and 'max' operator, but I'm assuming 'max' is not the issue here.

Source is 8-bit 1080p, loading with DGSource()
ex_expand(3) # 430
Prefetch(4)

ex_expand(3) # 190
Prefetch(8)

2 months ago I tested adding Prefetch() inside the functions, but it didn't work out, performance was much worse than a single call at end of the script.

[kedautinh12]

I think DGSource() affect to prefetch. You can change to L-SmashSource and speed will increase

[Boulder]

Pixel addressing seems to fluctuate a lot depending on prefetch value.
Using ex_expand(3) as an example which is a mix of pixel fetching and 'max' operator, but I'm assuming 'max' is not the issue here.

Source is 8-bit 1080p, loading with DGSource()
ex_expand(3) # 430
Prefetch(4)

ex_expand(3) # 190
Prefetch(8)

2 months ago I tested adding Prefetch() inside the functions, but it didn't work out, performance was much worse than a single call at end of the script.
I think you want to set the number of threads and frames separately in Prefetch. For example, I use threads=24, frames=12 on my 3900X. Too high value for frames will decrease performance (as will too low too, of course).

[Dogway]

@kedautinh12: L-SmashSource doesn't work for me in latest AVS+, using latest build (from August)

Setting frames improves it but still far from Prefetch(4)
ex_expand(3) # 340
Prefetch(8,4)

Anyway, I'm trying to describe a bigger issue. For example for big functions like QTGMC best performance is achieved with Prefetch(8), that traces back to any ex_expand() I have in the function and hits performance. Setting Prefetch(4) to just after ex_expand() won't fix things, but actually make them much worse, there's some discussion here (https://github.com/AviSynth/AviSynthPlus/issues/244).

The problem is on what basis do you set threads and frames? Trial and error?
My main concern is that while I'm doing a bunch of code optimizations in QTGMC that doesn't reflect on speed, compared to say using unoptimzed MaskTools2 calls.

That the code below is faster than my Expr optimization (even with Prefetch(4) ) is something to worry about, even if masktools2 is AVX2.
Merge( lossed1.mt_expand( mode="vertical", U=3,V=3 ), lossed1.mt_inpand( mode="vertical", U=3,V=3 ) )
Prefetch(8)

lossed1.Expr("x[0,-1] A@ x[0,0] B@ max x[0,1] C@ max A B min C min + 0.5 *")
Prefetch(8)

I tried with:
Merge( lossed1.mt_expand( mode="vertical", U=3,V=3,avx=true,avx2=false,sse4=false), lossed1.mt_inpand( mode="vertical", U=3,V=3,avx=true,avx2=false,sse4=false ) )
Prefetch(8)
and indeed it's a tad bit slower.
Looks like SSE4 makes the most change.

[tormento]

What is the status of VFR (variable framerate) support?

I went thru a VFR anime with 23.976/29.97 sequences and I really don't know if AVS can handle it in the proper way.

Now that we have frame properties, it could become an interest thing.

[Boulder]

The problem is on what basis do you set threads and frames? Trial and error?

I have a pretty stable script setup for my encodes, so I just took a long enough sample range and tested various values multiple times. Threads is always set to the maximum, but the amount of frames is something that might need tweaking. Increasing the maximum cache size is also a must with UHD sources and Prefetch with several frames.

[Dogway]

I don't think that might work, at least for my CPU (4C/8T) maxing threads to 8 performs worse than with 4, with whatever 'frames' I set it to.

For example (always from my CPU POV) internal resizers perform best with Prefetch(6), pixel addressing with Prefetch(4), mathematical expressions with Prefetch(6), and heavy functions like QTGMC with many expression calls Prefetch(8).

Now if you think you can take the best of both worlds and do:
ex_median()
Prefetch(4)
BicubicResize()
Prefetch(6)

It will actually perform much worse than with a single final Prefetch with 4 or 6, 6 being faster but as I said not optimal for the resizer.

I will run some benchmarks later.

[kedautinh12]

And if you use prefetch with L-SMASH source. I think it's faster than DGSource

[Dogway]

And if you use prefetch with L-SMASH source. I think it's faster than DGSource

What L-SMASH build are you using that works with test34? As I said the latest(?) build crashes. A few months back I benchmarked it and it wasn't very fast but you can't compare with GPU decoding.

[kedautinh12]

What L-SMASH build are you using that works with test34? As I said the latest(?) build crashes. A few months back I benchmarked it and it wasn't very fast but you can't compare with GPU decoding.

I did met crashed same you but in some benchmarked with TemporalDegrain2(postFFT=5) #cause if use bm3d. I seen L-SMASH source faster than DGSource when used with prefetch

[Boulder]

None of the source filters should do any multithreading regarding Prefetch. L-SMASH might do some internal multithreading when decoding.

[qyot27]

What L-SMASH build are you using that works with test34? As I said the latest(?) build crashes. A few months back I benchmarked it and it wasn't very fast but you can't compare with GPU decoding.
From git:
https://github.com/AkarinVS/L-SMASH-Works/commits/ffmpeg-4.5

[Dogway]

From git:
https://github.com/AkarinVS/L-SMASH-Works/commits/ffmpeg-4.5

Thanks! I got deceived with the "AviSynth+ users please use ???" message.

[qyot27]

Noting, of course, that that mainly arose from the part where you can't build HomeOfAviSynthPlusEvolution/L-SMASH-Works against current FFmpeg-git (https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/issues/11), thus creating problems for those Linux/mac/etc. users that prefer to install the git version of FFmpeg to their system, only to find out that they can't build LSMASHSource, because it was using stuff that FFmpeg made explicitly part of the private API as part of the transition to the next release. So eventually those fixes will either have to be re-upstreamed to HomeOfAviSynthPlusEvolution/L-SMASH-Works or land in the master branch of AkarinVS/L-SMASH-Works. The AviSynth fix occurred as a side-effect of getting it compiled for newer FFmpeg, as now that it could build, it was throwing assertion failures with AviSynth+-git. Again, something that'll have to be merged back upstream.

[Dogway]

@qyot27, thanks, so notes for developer only. EDIT: Didn't work either "There is no function named 'LSMASHVideoSource'"

I did some benchmarks, over 1080p@16-bit:
ex_median("median5") # 97 P(4) 111 P(6) 115 P(8) 116 P(8,12)
BicubicResize(1280,720)
BicubicResize(1920,1080) # 340 P(4) 300 P(6) 285 P(8) 280 P(8,8) 250 P(8,12) 305 P(8,4)
ex_edge("scharr") # 390 P(4) 380 P(6) 240 P(8) 200 P(8,12) 280 P(8,8) 200 P(8,16)

With this in mind one would want to run ex_median() with P(8,12) and the rest with P(4), well, it doesn't work like that.
ex_median("median5")
Prefetch(8,12)
BicubicResize(1280,720)
BicubicResize(1920,1080)
ex_edge("scharr")
Prefetch(4) # 90

You get better performance with a single last prefetch, which is not optimal for the resizers nor ex_edge():
ex_median("median5")
BicubicResize(1280,720)
BicubicResize(1920,1080)
ex_edge("scharr")
#Prefetch(4) # 74
#Prefetch(6) # 90
#Prefetch(8) # 93
#Prefetch(8,8) # 81
#Prefetch(8,12) # 92

[DTL]

Need some ideas about available Avisynth API about configurable multi-frame requesting between MDegrain and MAnalyse: https://forum.doom9.org/showthread.php?p=1960022#post1960022

With ability of MAnalyse to process several src-ref pairs with hardware accelerator and to balance load of CPU/HW_acc with motion search and degrain work it require to configure >1 src-ref pairs (motion search 'frames') requested from MAnalyse by MDegrain. What are the best solutions possible with current Avisynth inbetween filters API ? Can sink filter ask several frames from source at once or special hacks required to ask for > 1 frames in one call ?

May be possible solution is only to use several MAnalyse objects per one MDegrain so MDegrain can call GetFrame from >1 MAnalyse in parallel ?

[Nuihc88]

What is the status of VFR (variable framerate) support?

I went thru a VFR anime with 23.976/29.97 sequences and I really don't know if AVS can handle it in the proper way.

Now that we have frame properties, it could become an interest thing.

I'd like to know about this as well.

I've been trying make more advanced realtime frame interpolation scripts both by DeDupping+Interpolating (https://forum.doom9.org/showthread.php?p=1930740#post1930740) & Interpolating over Dupe-ranges (https://forum.doom9.org/showthread.php?p=1947800#post1947800), but keep hitting roadblocks with either functionality or speed depending on the approach.

Support for Variable Frame Rates would open up new ways of addressing both of my problems.

[pinterf]

I don't think that might work, at least for my CPU (4C/8T) maxing threads to 8 performs worse than with 4, with whatever 'frames' I set it to.

For example (always from my CPU POV) internal resizers perform best with Prefetch(6), pixel addressing with Prefetch(4), mathematical expressions with Prefetch(6), and heavy functions like QTGMC with many expression calls Prefetch(8).

Now if you think you can take the best of both worlds and do:
ex_median()
Prefetch(4)
BicubicResize()
Prefetch(6)

It will actually perform much worse than with a single final Prefetch with 4 or 6, 6 being faster but as I said not optimal for the resizer.

I will run some benchmarks later.
There were issues with multiple Prefetchers which were probably fixed in recent days, I built no test version since then.

[Dogway]

So the optimal performance for a given filter (after the last Prefetch refactor) is to set its MT mode (normally this is autoregistered) and pair it with an optimal Prefetch value, depending on CPU (cores, threads).

I would like to think that then the Prefetch value should also be autoregistered into the plugin for always an optimal performance (and script clearance) but I guess this dependes on many factors.

For example it would be a good experiment for someone with 8C/16C or more and test three types of filters each time with different configuration of Cores/Threads. I recommend to run the filter twice to also test memory transfer.

SourceFilter() # Test with several loaders, FFMS2, LSmash and DGDecNV

BicubicResize()
BicubicResize() # known to perform best in P(4) for 4C/8T

Expr() # some expression with trascendentals
Expr() # some expression with trascendentals # known to perform best in P(6) for 4C/8T

ex_median("median7")
ex_median("median7") # a simple filter known to perform best in P(8) for 4C/8T

[pinterf]

I don't think filter writers can guess the optimal multithreading level. You have to experiment it on your actual PC/script when you need further percents. Too many important factors: actual video resolution, processor cache size, is script using filters with internal MT or not. Then the output side: encoding settings. Manufacturer AMD or Intel, OS Windows or Linux, other parallel tasks your PC/laptop is doing in the background.

[Dogway]

I know but it looks to me that there are a few evidences that certain filters work better with a different Prefetch() configuration than others. Scalers are one of these (typical used in pairs for super or undersampling) and other ones yet to test. I'm not saying true or false but that a global benchmark with some core filters (scalers, algebra expressions, convolutions) would tell us something. I mean, stacking two BicubicResize() and needing to use Prefetch(4) looks suspicious. Other users can test as well.

Source 1080p@16-bit (DGSource)

BicubicResize(1280,720)
BicubicResize(1920,1080) # 340 P(4) 300 P(6) 285 P(8) 280 P(8,8) 250 P(8,12) 305 P(8,4)

[pinterf]

See my results, but with BlankClip.

It is not a real-life result but it shows better how the filters alone behave.

I guess the slowdown you are seeing (P8 is slower than P4) is not because of the resizers but how the source filter gets requests.

As I experienced earlier, out-of-sequential order requests from source filter affect hugely the performance. In MT environment, when the Prefetcher works in the background parallel with a the final consumer (this time an AvsMeter64) it can easily happen that at specific timing conditions consumer overtakes Prefetcher and the source filter is seeing fluctuating frame requests e.g. 1,2,3,4,5,7,8,6,9,10,11,13,14,12,...

BlankClip(100000, 1920, 1080, pixel_type="YUV420P16")
BicubicResize(1280,720)
BicubicResize(1920,1080) # 340 P(4) 300 P(6) 285 P(8) 280 P(8,8) 250 P(8,12) 305 P(8,4)
Prefetch(8,4)
/*
i7-7700 (Core 4 Thread 8)
685 (48%) P(4)
734 (71%) P(6)
740 (85%) P(8)
742 (84%) P(8,8)
743 (86%) P(8,12)
690 (44%) P(8,4)
*/

[pinterf]

Or in your configuration the memory bandwidth and cache usage are getting their penalty earlier than in my test.

[DTL]

Current CPUs with set-associative caches and current SDRAMs can not provide stable performance with not-equal addresses of data in RAM in different runs. Depending on addresses values the performance of caches and SDRAM will change. And typical 'new' universal object-oriented programming way of memory request from OS can not guarantee equal addresses relative to cache structure and SDRAM structure. Also fragmenting of virtual memory change placement of virtual pages on physical SDRAM even with equal virtual addresses. So it may require many test runs to found some 'stable' performance change.

Though SDRAM-page miss possibly rare and lower performance penalty in compare with cache set ways exhausting. It blocks cache activity IMHO close to completely even with still lots of empty lines available in different sets and drop performance to RAM-only.

[Dogway]

That made a difference. But since Prefetch() couldn't be used several times I used RequestLinear().

RequestLinear(50)
BicubicResize(1280,720)
BicubicResize(1920,1080) # 510 P(4) 480 P(8,4) 530 P(8,8)

Unfortunately cannot test latest LSmash, and FFMS2 is too slow to make a point. Still need to update to latest DGSource, in case there are changes.

[kedautinh12]

That made a difference. But since Prefetch() couldn't be used several times I used RequestLinear().

RequestLinear(50)
BicubicResize(1280,720)
BicubicResize(1920,1080) # 510 P(4) 480 P(8,4) 530 P(8,8)

Unfortunately cannot test latest LSmash, and FFMS2 is too slow to make a point. Still need to update to latest DGSource, in case there are changes.

I still use LSmash latest ver with latest avs+ normally

[Boulder]

Still need to update to latest DGSource, in case there are changes.
None of the source filters utilize Prefetch, so updating DGSource won't change anything.

Prefetch does have some problems compared to Vapoursynth's multithreading method. The Zopti thread contains some of my test results, I was unable to use the AVS version like I can use the VS one.

https://forum.doom9.org/showthread.php?t=175723&page=6

[videoh]

None of the source filters utilize Prefetch, so updating DGSource won't change anything. Also, nothing in DGSource() affecting performance has been changed for a very long time.

[real.finder]

None of the source filters utilize Prefetch, so updating DGSource won't change anything.

Prefetch does have some problems compared to Vapoursynth's multithreading method. The Zopti thread contains some of my test results, I was unable to use the AVS version like I can use the VS one.

https://forum.doom9.org/showthread.php?t=175723&page=6

not just source filters https://github.com/AviSynth/AviSynthPlus/issues/229

[qyot27]

AviSynth+ 3.7.1 has been released (https://github.com/AviSynth/AviSynthPlus/releases/tag/v3.7.1).

So yeah, getting this out by New Year's, at least. Windows installers and filesonly (including for Windows 10 on ARM) and macOS 10.13+ filesonly are available (10.15+ builds and installers will come later).

Additions:

- Linux: Show more information when dlopen fails
- Expr: allow auto scaling effect on pixels obtained from relative addressing
- New array manipulators: ArrayDel, ArrayAdd, ArrayIns, ArraySet with accepting multi dimensional indexes
- ExtractY/U/V/R/G/B/A, PlaneToY: delete _ChromaLocation property. Set _ColorRange property to "full" if source is Alpha plane
- Add new AEP (Avisynth Environtment Property) constants to directly query Avisynth interface main and bugfix version and system endianness:
AEP_HOST_SYSTEM_ENDIANNESS, AEP_INTERFACE_VERSION, AEP_INTERFACE_BUGFIX (c++)
AVS_AEP_HOST_SYSTEM_ENDIANNESS, AVS_AEP_INTERFACE_VERSION, AVS_AEP_INTERFACE_BUGFIX (c)
- Interface: introduce AVISYNTHPLUS_INTERFACE_BUGFIX_VERSION.
- New interface functions env->MakePropertyWritable/VideoFrame::IsPropertyWritable.
- Expr: allow 'f32' as internal autoscale target (was: i8, i10, i12, i14, i16 were accepted, only integers)
- Expr: LUT mode! 'lut'=1 or 2 for 1D (lut_x) and 2D (lux_xy) support
- xPlaneMin/Max/Median/MinMaxDifference to accept old packed formats (RGB24/32/48/64 and YUY2) by autoconverting them to Planar RGB or YV16
- New runtime function: PlaneMinMaxStats returns an array and/or set global variables.
- Language syntax: accept arrays in the place of "val" script function parameter type regardless of being named or unnamed.
- Histogram "Levels": more precise drawing when bit depth is different from histogram's resolution bit depth, plus using full/limited flag.
- Expr: no more banker's rounding when converting back float result to integer pixels. Using the usual truncate(x+0.5) rounding method
- ColorYUV: More consistent and accurate output across different color spaces, match with ConvertBits fulls-fulld conversions
- ColorYUV: set _ColorRange frame property
- ColorYUV: when no hint is given by parameter "levels" then it can use _ColorRange (limited/full) frame property for establishing source range for gamma
- ColorYUV "showyuv_fullrange"=true: fix shown U and V ranges. E.g. for bits=8: 128 +/- 127 (range 1..255 is shown) instead of 0..255
- propShow: display _Matrix, _ColorRange and _ChromaLocation constants with friendly names
- Expr: new function "sgn". Returns -1 when x is negative; 0 if zero; 1 when x is positive
- Expr: add "neg": negates stack top: a = -a
- ConvertBits: Support YUY2 (by autoconverting to and from YV16), support YV411
- ConvertBits: "bits" parameter is not compulsory, since bit depth can stay as it was before. Call like ConvertBits(fulld=true)
- ConvertBits: much nicer output for low bit depth targets such as dither_bits 1 to 7.
- ConvertBits: allow dither down from 8 bit sources by giving a lower dither_bits value
- ConvertBits: dither=1 (Floyd-S) to support dither_bits = 1 to 16 (similar to ordered dither)
- ConvertBits: dither=0 (ordered) to allow odd dither_bits values. Any dither_bits=1 to 16 (was: 2,4,6,8,..)
- ConvertBits: dither=0 (ordered) allow larger than 8 bit difference when dither_bits is less than 8.
- ConvertBits: Correct conversion of full-range chroma at 8-16 bits. Like 128+/-112 -> 128+/-127 in 8 bits
- ConvertBits: allow dither from 32 bits to 8-16 bits
- ConvertBits: allow different fulls fulld when converting between integer bit depths (was: they must have been the same)
- ConvertBits: allow 32 bit to 32 bit conversion
- frame property support: _ChromaLocation in various filters (e.g. ConvertToYUV422)
- Support additional chroma locations "top", "bottom_left", "bottom"
- New syntax for "matrix" parameters (e.g. in ConvertToYUV444 old:"rec601" new "170m:l") which separate matrix and full/limited marker.
Old syntax is still valid but does not support all new matrix values.
- frame propery support: _Matrix and _ColorRange in various filters. New "matrix" string constants
- RGB<->YUV (YUY2) conversions: frame property support _Matrix and _ColorRange (_Primaries and _Transfer is not used at all yet)
- ConvertBits: use input frame property _ColorRange to detect full/limited range of input clip
- ColorBars, ColorBarsHD, BlankClip: set frame properties _ColorRange and _Matrix
- New function: propCopy to copy or merge frame properties from one clip to another.
- xxxPlaneMin xxxPlaneMax, xxxPlaneMinMaxDifference for 32 bit float formats:
when threshold is 0 then return real values instead of 0..1 (chroma -0.5..0.5) clamped histogram-based result
- Allow propGetXXX property getter functions called as normal functions, outside runtime. Frame number offset can be used.
- YUY2 RGB conversions now allow matrix "PC.2020" and "Rec2020"
- 4:2:2 conversions: allow ChromaInPlacement and ChromaOutPlacement:
Valid values: left/mpeg2, center/mpeg1/jpeg
- 4:2:0 conversions: new ChromaInPlacement and ChromaOutPlacement values:
top_left, left (alias to mpeg2), center (alias to mpeg1), jpeg (alias to mpeg1) (see http://avisynth.nl/index.php/Convert)
- Expr: atan2 (SIMD acceleration as well)
- Expr: sin and cos SIMD acceleration (SSE2 and AVX2) port from VapourSynth (Akarin et al.)
- Expr: x.framePropName syntax for injecting actual frame property values into expression
- Script functions to supports arrays with _nz type suffix. (one or more)
- Expr: arbitrary variable names (instead of single letters A..Z), up to 128 different one.
- Expr: add 'round', 'floor', 'ceil', 'trunc' operators (nearest integer, round down, round up, round to zero)
Acceleration requires at least SSE4.1 capable processor or else the whole expression is running in C mode.
- Recognize \\' and \\b and \\v in escaped (e"somethg") string literals (see http://avisynth.nl/index.php/The_full_AviSynth_grammar#Literals)
- Expr: allow TAB, CR and LF characters as whitespace in expression strings
- Clip types for propSet, propGet, add propSetClip, propGetClip
- Clip content support for propGetAsArray, propSetArray and propGetAll
- RGBAdjust: analyse=true 32 bit float support


Build environment, Interface:

- Added stubs for compiling on RISC-V and SPARC
- Visual Studio 2022: Add /fp:contract to compilation parameters (addition to /fp:precise)
- Check Visual Studio 2022, add build examples to documentation. Recognized: it has still an option to use v141_xp toolkit
- CMake build environment: older GCC can be used which knows only -std=c++-1z instead of c++17
- AviSynth programming interface V8.1 / V9:
Add 'MakePropertyWritable' to the IScriptEnvironment (CPP interface), avs_make_property_writable (C interface)
Add 'VideoFrame::IsPropertyWritable' (CPP interface), avs_is_property_writable (C interface)
- Info on Windows XP compatibility (must revert to an older Visual C++ Redistributable)
- CMake/source: Intel C++ Compiler 2021 and Intel C++ Compiler 19.2 support
- experimental! Fix CUDA plugin support on specific builds, add CMake support for the option.
- Fixes for building the core as a static library


Fixes:

- Fix: "Text" filter would crash when y coord is odd and format has vertical subsampling
- Fix: MinMax runtime filter family: check plane existance (e.g. error when requesting RPlaneMinMaxDifference on YV12)
- Fix: prevent x64 debug AviSynth builds from crashing in VirtualDub2 (opened through CAVIStreamSynth)
- Expr: fix conversion factor (+correct chroma scaling) when integer-to-integer full-scale automatic range scaling was required
- ColorYUV: fix 32 bit float output
- ColorYUV: fix display when showyuv=true and bits=32
- ConvertBits: "dither" parameter: type changed to integer. Why was it float? :)
- ConvertBits: Fix: fulls=true -> fulld=true 16->8 bit missing rounding
- Fix: Planar RGB 32 bit -> YUV matrix="PC.709"/"PC.601"/"PC.2020" resulted in greyscale image
- SelectRangeEvery: experimental fix on getting audio part (TomArrow; https://github.com/AviSynth/AviSynthPlus/issues/232)
- Fix: Overlay "blend" 10+ bit clips and "opacity"<1 would leave rightmost non-mod8 (10-16 bit format) or non-mod4 (32 bit format) pixels unprocessed.
- Fix: Overlay "blend" with exactly 16 bit clips and "opacity"<1 would treat large mask values as zero (when proc>=SSE4.1)
- Parser: proper error message when a script array is passed to a non-array named function argument
(e.g. foo(sigma=[1.1,1.1]) to [foo]f parameter signature)
- Fix: Expr: wrong constant folding optimization when ternary operator and Store-Only (like M^) operator is used together.
- ColorBars: fixed studio RGB values for -I and +Q for rgb pixel types
- ColorBarsHD: use BT.709-2 for +I (Pattern 2), not BT.601.
Also fixed Pattern 1 Green.Y to conform to SMPTE RP 219-1:2014 (133, not 134).
- Overlay mode "multiply": proper rounding in internal calculations
- Fix: ConvertAudio integer 32-to-8 bits C code garbage (regression in 3.7)
- Fix: ConvertAudio: float to 32 bit integer conversion max value glitch (regression in 3.7)
- Fix: Crash in ColorBars very first frame when followed by ResampleAudio
- Fix: frame property access from C interface
- Fix: StackVertical and packed RGB formats: get audio and parity from the first and not the last clip


Optimizations:

- Quicker ClearProperties and CopyProperties filters (by using MakePropertyWritable instead of MakeWritable).
- ConvertBits: AVX2 support
- ConvertBits: Special case for: 8->16 bit fulls=true, fulld=true
- Expr: consume less bytes on stack. 48x Expr call in sequence caused stack overflow
- xxxPlaneMin xxxPlaneMax, xxxPlaneMinMaxDifference for threshold 0 became a bit quicker for 8-16 bit formats (~10% on i7-7700)
- Speedup: Overlay mode "multiply": overlay clip is not converted to 4:4:4 internally when 420 or 422 subsampled format
(since only Y is used from that clip)
- Speedup: Overlay mode "multiply": SSE4.1 and AVX2 code (was: C only)
SSE4.1: ~1.2-2.5X speed, AVX2: ~2-3.5X speed (i7700 x64 single thread, depending on opacity full/not, mask clip yes/no)
- ConvertAudio: Add direct Float from/to 8/16 conversions (C,SSE2,AVX2)

[LigH]

:) Happy New Year!

[StainlessS]

:) Happy New Year!
+1 on that :)

Nice goin' guys, guess I is gonna havta learn it all from scratch again, one helluva lotta changes / improvements.

[tormento]

AviSynth+ 3.7.1 has been released
Thanks!

Is the x64 build CUDA aware?

Just to know if I can delete previous pinterf build or keep it.

[qyot27]

I did not enable CUDA on the release builds. For one [big] reason, I don't have an Nvidia GPU to verify that such a build functions correctly.

[pinterf]

Thanks!

Is the x64 build CUDA aware?

Just to know if I can delete previous pinterf build or keep it.

When you don't use or have built yourself those experimental Avs+ CUDA interface filters then you don't need it. My CUDA builds are not making anything quicker by default, CUDA is not used internally. These builds are just containing an interface extension with which such plugins can use Avisynth+ 'device' interface additions.

[pinterf]

qyot27, thank you for the release
Happy New Year for the community!

[Dogway]

Thanks for the release!

A few questions.
-Did the CombinePlanes optimization make it for this version? (is that the IsPropertyWritable fix?)
-Did the multiple Prefetch optimization make it for this version?
-This "ConvertBits: allow dither from 32 bits to 8-16 bits", I think 32-bit to 16-bit is not dithering but rounding
-Maybe also include the BlankClip array type for color arg (ie. [0,32768,32768]), not sure if it was a change within last and this, didn't find in any changelog.

[tormento]

CUDA is not used internally
I do know, thank you.

But it's a "nice to have" :)

[DTL]

CUDA is not used internally. These builds are just containing an interface extension with which such plugins can use Avisynth+ 'device' interface additions.

It may be big project addition - to add DirectX resources management in HW accelerator memory (using Microsoft API). Microsoft API is less manufacturer-dependent.

It support many enough different resources formats:
https://docs.microsoft.com/en-us/windows/win32/api/dxgiformat/ne-dxgiformat-dxgi_format and conversion may be with byte-manipulation without any math and lost speed or quality.

So some memory-limited filters can use much faster accelerator board memory and processing on accelerator.

May be add some 'upload' and 'download' commands to switch between host and accelerator memory data placement.
It may looks like ConvertTo functions:
ConvertToDXGI_XXX to upload resource to accelerator memory and
ConvertTo standard Avisynth format to download back to host memory.

[pinterf]

Thanks for the release!

A few questions.
-Did the CombinePlanes optimization make it for this version? (is that the IsPropertyWritable fix?)
-Did the multiple Prefetch optimization make it for this version?
-This "ConvertBits: allow dither from 32 bits to 8-16 bits", I think 32-bit to 16-bit is not dithering but rounding
-Maybe also include the BlankClip array type for color arg (ie. [0,32768,32768]), not sure if it was a change within last and this, didn't find in any changelog.
- yes, multiple Prefetcher fix is included
- yes, CombinePlanes geet the optimization of that very special case. (IsPropertyWritable/MakeProperyWritable are different things, they are programming interface additions which are used at some places where frame content does not need to be changed, and only frame properties are altered)
- yes (from 32 bit float to 16 bit integer there is zero dithering)
- BlankClip has 'colors' array parameter since the beginning of script array concept, it was just undocumented.

[pinterf]

It may be big project addition - to add DirectX resources management in HW accelerator memory (using Microsoft API). Microsoft API is less manufacturer-dependent.

It support many enough different resources formats:
https://docs.microsoft.com/en-us/windows/win32/api/dxgiformat/ne-dxgiformat-dxgi_format and conversion may be with byte-manipulation without any math and lost speed or quality.

So some memory-limited filters can use much faster accelerator board memory and processing on accelerator.

May be add some 'upload' and 'download' commands to switch between host and accelerator memory data placement.
It may looks like ConvertTo functions:
ConvertToDXGI_XXX to upload resource to accelerator memory and
ConvertTo standard Avisynth format to download back to host memory.
This must be planned carefully and must keep the logic of existing 'device' interface, similar to CUDA frame and memory transfer which is already implemented. OnCPU, OnCUDA, device types, filters capability queries, cache, SetMemoryMax, etc...

[DTL]

"similar to CUDA frame and memory transfer which is already implemented. "

May it can be copy of CUDA with a bit different names of resources ? May be Microsoft do not design completely new methods for the same hardware device.

[FranceBB]

Oh, stable build, nice!
What a way to celebrate the end of 2021.
Happy New Year, everyone!! :D

[Reel.Deel]

Thanks for the new release pinterf and qyot27. And happy new years :)

I think I found a bug 2 days ago:

Blankclip(width=352, height=288, pixel_type="YV12") # bug not related to dimensions
Histogram("audiolevels"))

The 0dB is cropped and the numbers and dB are no longer in the center of the dashes like they used to be:
https://i.ibb.co/TvqMzD5/histogram-audiolevels-bug.png (https://imgbb.com/) http://avisynth.nl/images/Histogram_audiolevels.jpg

Not sure when it happened but it used to look fine.

[pinterf]

Thanks for the new release pinterf and qyot27. And happy new years :)

I think I found a bug 2 days ago:

Blankclip(width=352, height=288, pixel_type="YV12") # bug not related to dimensions
Histogram("audiolevels"))

The 0dB is cropped and the numbers and dB are no longer in the center of the dashes like they used to be:

Not sure when it happened but it used to look fine.
Since 3.6 (Linux support, "Text" filter). The vertical coordinates were shifted up by 10 but the alignment was changed from center to none internally. Character height is 20, this is why there is a half character shift.
EDIT: fixed on git

[Reel.Deel]

Since 3.6 (Linux support, "Text" filter). The vertical coordinates were shifted up by 10 but the alignment was changed from center to none internally. Character height is 20, this is why there is a half character shift.

No wonder I did not noticed it earlier, I have not used "AudioLevels" in a while. Funny that you mention the Text filter, that is my next bug report :D

I compared the Text filter to FreeSub (http://avisynth.nl/index.php/FreeSub)and it seems the Text filter does not honor the transparency value.

Text | FreeSub
https://i.ibb.co/JQwYNXw/text-bug.png (https://ibb.co/jRPCd7P)

For the halo color the Text filter only reacts when it's 0 or 255, at 255 it draws the box around the text. And text color does not do anything with any transparency value, unless both halo and text color are at 255, which at that point does not render anything. I know the Filter is mainly intended for debugging purposes but it would be nice if it honored the transparency values. I like the box it draws around the text, maybe it would not be a bad idea to have another color parameter for it, that way we can control the text, halo, and box color.

Here's the bdf font file (https://files.videohelp.com/u/223002/ter-u18n.bdf) I used for FreeSub and here's the script:
Blankclip(color=$FDA50F, pixel_type="RGB24", width=170, height=115)

Text("TESTING", font="Terminus", x=10, y=10, text_color=$00FFFFFF, halo_color=$FF000000)
Text("TESTING", font="Terminus", x=10, y=30, text_color=$00FFFFFF, halo_color=$00000000)
Text("TESTING", font="Terminus", x=10, y=50, text_color=$00FFFFFF, halo_color=$80000000)
Text("TESTING", font="Terminus", x=10, y=70, text_color=$FFFFFFFF, halo_color=$00000000)
Text("TESTING", font="Terminus", x=10, y=90, text_color=$80FFFFFF, halo_color=$80000000)

Freesub("TESTING", font="ter-u18n.bdf", x=125, y=19, text_color=$00FFFFFF, halo_color=$FF000000)
Freesub("TESTING", font="ter-u18n.bdf", x=125, y=39, text_color=$00FFFFFF, halo_color=$00000000)
Freesub("TESTING", font="ter-u18n.bdf", x=125, y=59, text_color=$00FFFFFF, halo_color=$80000000)
Freesub("TESTING", font="ter-u18n.bdf", x=125, y=79, text_color=$FFFFFFFF, halo_color=$00000000)
Freesub("TESTING", font="ter-u18n.bdf", x=125, y=99, text_color=$80FFFFFF, halo_color=$80000000)

PointResize(width*4, height*4)

[pinterf]

Yes, "Text" is just a dumb debug filter using fixed size fonts, ignores real transparency. It was created quickly as a poor man's Subtitle, because under Linux we cannot use Windows GDI which is the core of SubTitle.
EDIT: Or I do not remember well.
I found this comment
halocolor MSB
- FF: fadeIt
- 01-FE: no halo
- 00: use halocolor

[Reel.Deel]

Thanks for the explanation pinterf. I will document those limitations.

Edit: So if the Text filter can't do transparency, how does it overlay the almost transparent box around the text? To match the same color I overlaid a black square onto the clip at .12 opacity (1F).

[pinterf]

Thanks for the explanation pinterf. I will document those limitations.

Edit: So if the Text filter can't do transparency, how does it overlay the almost transparent box around the text? To match the same color I overlaid a black square onto the clip at .12 opacity (1F).
Hardcoded and quick ratio. This or that ratio is used in almost all plugins that use some version of "info.h" for their debug displays.

https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/core/info.cpp#L636

[Dogway]

Thanks for feedback pinterf! I will test it then, specially the Prefetch() changes, I'm very intrigued.

Aside from LUT calculations with scaled down bitdepth (with scale_inputs) I would like to spot an usage issue I'm having with scale_inputs.
I think the setting would be better implemented as a decorator if it's possible. I'm doing some kind of crazy checks for some of my latest filters which involves different luma and chroma expressions, so for example you can have scaling in luma plane but not on chroma, or reverse.
Some examples:

16-bit source
expr("f32 x 2 ^", "x cmin - range_max cmax cmin - / *", scale_inputs="int")
This converts chroma to 8-bit, unless I explicitly add a protective decorator:
expr("f32 x 2 ^", "i16 x cmin - range_max cmax cmin - / *", scale_inputs="int")
Maybe to something like this:
expr("f32int x 2 ^", "x cmin - range_max cmax cmin - / *")

By the way, despite internal calculations done in float, the output chroma don't match in both versions.


Or simply copy the luma plane:
16-bit source
expr("x", "f32 x blabla ", scale_inputs="int")

This would scale the luma plane to 8-bit and back despite just wanting to copy it.

Also the 'floatUV' option makes it impossible to scale anything while wanting to shift chroma at the same time, so one has to explicitly shift it in expression which is slower.

[FranceBB]

The vertical coordinates were shifted up by 10 but the alignment was changed from center to none internally. Character height is 20, this is why there is a half character shift.

Dang it, it affects my VideoTek() :(


EDIT: fixed on git


I love you! :D
(in a friendly, figurative, non homo way xD)

[ryrynz]

(in a friendly, figurative, non homo way xD)

I think we're past needing to define love. Take whatever you can get lol.
Appreciate ur work Pinterf *wink*

[qyot27]

The macOS installers and filesonly archives are up now. I had to re-do the tarball for 10.13/14 because of some things I'd overlooked about how tar works. The new ones should be okay.

[gispos]

Before avisynth plus there were no problems with Pascal code and avisynth. I could open any text with 'Eval' and create a clip.

Since avisynth plus only 'AviSource' worked, so I couldn't pass any text with 'Eval' to create a clip.
But I could still open a script with AviSource.

And now the 64bit avisynth works anymore as soon as MCTemporalDenoise is called in the script the error 'Division by Zero' comes.
What has been changed please? The 'AviSource' still works perfectly with all the 64bit 3.71 test versions.

The last version that works without problems is 3.71 test 22. Everything else after that causes problems with prefetch. With the stable 3.71 there are no problems with prefetch, but I can no longer use it at all. Only the 32bit version works.

AvsPThumb has stopped working. I have a number of programs that I've been using for decades that no longer work.
It's horrible, I'm crying.

[Boulder]

Before avisynth plus there were no problems with Pascal code and avisynth. I could open any text with 'Eval' and create a clip.

Since avisynth plus only 'AviSource' worked, so I couldn't pass any text with 'Eval' to create a clip.
But I could still open a script with AviSource.

And now the 64bit avisynth works anymore as soon as MCTemporalDenoise is called in the script the error 'Division by Zero' comes.
What has been changed please? The 'AviSource' still works perfectly with all the 64bit 3.71 test versions.

The last version that works without problems is 3.71 test 22. Everything else after that causes problems with prefetch. With the stable 3.71 there are no problems with prefetch, but I can no longer use it at all. Only the 32bit version works.

AvsPThumb has stopped working. I have a number of programs that I've been using for decades that no longer work.
It's horrible, I'm crying.

It would be much easier to investigate the issue if you posted the simplest example scripts of how to reproduce the issue, and also link to any functions that you use in them.

[gispos]

It would be much easier to investigate the issue if you posted the simplest example scripts of how to reproduce the issue, and also link to any functions that you use in them.
As I wrote it, an 'Eval' with Pascal code has not worked for a long time. But now an invoke with 'AviSource' is no longer possible.
But that had worked with all Avisynth versions since Avisynth was born.


LWLibavVideoSource(SourceFile)
MCTemporalDenoise(settings="low", sigma=4, strength=100, tovershoot=1, GPU=false)


https://i.postimg.cc/5ysRVYv4/avisynth-error.jpg

[Boulder]

Did you check those scripts and the corresponding lines? That's why I asked for the links..

[gispos]

Did you check those scripts and the corresponding lines? That's why I asked for the links..

I can't find anything extraordinary when I look at the relevant places.

So far with Pascal code there were only problems with 'Eval' and 'Import' if MCTemporalDenoise was in the script.
With invoke ('AviSource', "test.avs') I could always open a script and that with all existing Avisynth versions.

But with the latest version nothing works anymore. No 'Eval', no 'Import' and no 'AviSource'

See also AvsPmod thread or here (https://forum.doom9.org/showthread.php?p=1959543#post1959543), since version 3.71 test 23 something went wrong.

[Boulder]

I'm baffled why any Pascal code would be related Avisynth internals. Did you try the latest official version with the installer (published just days ago)? And still, it would be good to see where you got those functions from.

[cretindesalpes]

The quoted line from GradFun2DBmod v1.5 contains the following:
mt_lut(expr="255 x 1 "+string(range)+" / * 2 ^ /",u=1,v=1)
It generates a div by 0 when the pixel value is 0 (probably something normal, at least it is part of the LUT entries) or when the range parameter is 0 (value checked by the script as legal). The code is already wrong twice, and the weird thing is why did it run without error before?

Maybe some exception handling has changed in MaskTools2 or Avisynth?

BTW what is this “Pascal code”?

[gispos]

The quoted line from GradFun2DBmod v1.5 contains the following:
mt_lut(expr="255 x 1 "+string(range)+" / * 2 ^ /",u=1,v=1)
It generates a div by 0 when the pixel value is 0 (probably something normal, at least it is part of the LUT entries) or when the range parameter is 0 (value checked by the script as legal). The code is already wrong twice, and the weird thing is why did it run without error before?

Maybe some exception handling has changed in MaskTools2 or Avisynth?

BTW what is this “Pascal code”?

Pascal code are programs written in Delphi.
And I get this error message when I want to use avisynth with a Delphi program.
In the past the error only came when I used 'Eval'. Now the error also comes with 'Eval' and 'AviSource'.

You have probably uncovered this dubious problem with 'MCTemporalDenoise' and programs with Pascal code.:thanks:
Why doesn't this error come with other programs? Very strange, AvsPmod or VirtualDub etc. show no errors.

Can someone please fix the faulty code in GradFun2DBmod.

[StainlessS]

Can someone please fix the faulty code in GradFun2DBmod.

Gispos, does this work ok. [I've never used that function, dont even know what it does].
FIXED: bad attempt removed
RPN = "x 0 == 255 255 x " + string(range) + " / 2 ^ / ?"

Infix = "x == 0 ? 255 : (255/((x/range)^2))"

With range=128
https://i.postimg.cc/520RBmcR/Range128.jpg (https://postimages.org/)

With range = 64
https://i.postimg.cc/rwPYQtMF/range64.jpg (https://postimages.org/)

With range = 32
https://i.postimg.cc/tJprY3tD/Range32.jpg (https://postimages.org/)

EDIT: Plotted with my Brain Dead Folly grapher thingy [stepping by 2, ie x = 0,2,4,6,8 etc, else my func crashes - my func memory prob {stack exhaustion}]

EDIT: Where 42 represents range, [we used dummy 42 so we can use Mt_polish and mt_Infix so we know where to replace the "string(range)" stuff]
We changed original INFIX "255/((x*(1/42))^2)"

To "255/((x/42)^2)"


And then added the fix,
"x == 0 ? 255 : (255/((x/42)^2))"

and convert back to RPN, and replace 42 with "string(range)" stuff.

So final, [without the div by zero when x == 0 ]

mt_lut(expr="x 0 == 255 255 x " + string(range) + " / 2 ^ / ?",u=1,v=1)


EDIT: Graphs are not very visible with white background, I use FireFox dark reader and they look fine.
EDIT: Re-did images.

EDIT:
(value checked by the script as legal).

Damn, I guess I did skim read that, I thought cretindesalpes meant that range=0 could not occur, I'll fix that problem too.
EDIT: Double damn, trickier than I thought, no idea how to fix the other divide by zero.

[gispos]

Damn, I guess I did skim read that, I thought cretindesalpes meant that range=0 could not occur, I'll fix that problem too.
EDIT: Double damn, trickier than I thought, no idea how to fix the other divide by zero.

:D Give your best. :) Thanks

[StainlessS]

Think this does it but a bit verbose, [R = range]


Infix = "255 / ((((x == 0) | (R == 0) ? 1 : x) / ((x == 0) | (R == 0) ? 1 : R)) ^ 2)"

RPN = "255 x 0 == R 0 == | 1 x ? x 0 == R 0 == | 1 R ? / 2 ^ /"


Somebody check it please. [dont bother, I'm pretty sure its right, and works fine in grapher]

When either x or R is 0, result is 255 [I hope :) ].

EDIT:

And then added the fix,
"x == 0 ? 255 : (255/((x/42)^2))"

Also, can somebody confirm or deny that if above x ==0 sets 255, but the remainder in white is still processed,
not 'short cut' skipped over or stripped from stack. [if 42 (range) is 0, there would still be a divide by zero] even with that earlier fix.

[Dogway]

Does it work if you limit it to epsilon?
range = max(range,0.001)
"255 x 1 " + string(range) + " / * 2 ^ 0.001 max /"

[StainlessS]

Oh, I forgot to post the R -> String(range) expanded whotsit,

"255 x 0 == " + String(range) + " 0 == | 1 x ? x 0 == " + String(range) + " 0 == | 1 " + string(range) + " ? / 2 ^ /"

so,
mt_lut(expr="255 x 0 == " + String(range) + " 0 == | 1 x ? x 0 == " + String(range) + " 0 == | 1 " + string(range) + " ? / 2 ^ /",u=1,v=1)

limit it to epsilon?
I was not getting any divide by zero with range == 0 anyway [I dont think, even though I should have], I'll retry.

limit it to epsilon?
Well testing in the BrainDeadFolly thingy, [may have to zoom it a bit - 1 dot in top left corner, and rest along the bottom]
https://i.postimg.cc/MHtSNzmD/Brain-Dead-Folly-00.jpg (https://postimages.org/)
So looks correct. [only even x values plotted - avoid stack exhaustion]

EDIT: I did originally think of Dogway fix too, but avoided it for some reason (cant remember why, and chose the verbose one).

[gispos]

Thank you StainlesSS, I just tried it and it works with no errors.
Does the mt_lut in the first line also have to be changed? (line 309)

GFmask = radius==1 ? input.mt_edge(mode="min/max",thY1=0,thY2=255,u=1,v=1).mt_lut(expr="255 x 1 "+string(range)+" / * 2 ^ /",u=1,v=1).removegrain(19,-1)
\ : mt_luts(input,input,mode="range",pixels=mt_square(radius),expr="y",u=1,v=1).
\ mt_lut(expr="255 x 0 == " + String(range) + " 0 == | 1 x ? x 0 == " + String(range) + " 0 == | 1 " + string(range) + " ? / 2 ^ /",u=1,v=1).removegrain(19,-1)

[gispos]

I would like to draw the avisynth developers attention to the problem with the frame properties and prefetch.
If the frame properties are read immediately after a get_frame, avisynth gets stuck when prefetch is used.

Sequences when initializing a new clip, for illustration only:

clip = env.invoke('Eval', args)
frame = clip.get_frame(nr)
re = env.props_get_matrix(frame)

That still worked up to 3.71 test 22 version, the 3.71 final 3593 stuck.
With the final version, a waiting time must be set after get_frame, which is 0.1 to 1.0 seconds, depending on the filters used. (Only with prefetch)

clip = env.invoke('Eval', args)
frame = clip.get_frame(nr)
time.sleep(1.0)
re = env.props_get_matrix(frame)

Furthermore there is the problem that the frame properties of the old clip are apparently still in the memory when a new clip is created and the same frame number is called and no frame properties are available because a filter has not passed them on.

Then it can happen that you get the properties from the previous clip.
This is not the case with version 3.71 test 22 and all older ones

It would also be desirable if one would test newer versions with AvsPmod. I think AvsPmod is used by some users and can be used as an indicator for the C interface:
1.) The C interface is used.
2.) Reads and displays the frame properties.

AvsPmod test version 1 second waiting (https://www.mediafire.com/file/e8iz3bb9u39sw0c/AvsPmod_v2.7.0.2.test_(Windows_x86-64).zip/file)

[StainlessS]

Does the mt_lut in the first line also have to be changed? (line 309)

Sorry, GP, I'll try get to it today.

[Emulgator]

gispos: I would like to draw the avisynth developers attention to the problem with the frame properties and prefetch.
Maybe you can PM Ferenc (pinterf) about that.

[gispos]

Maybe you can PM Ferenc (pinterf) about that.
For what reason?
The version isn't even from him, and if everyone would send him their problems with Avisynth he would be rightly annoyed.

[real.finder]

I think pinterf already seen the problem by now, but maybe he busy to answer

anyway, maybe it better to post it https://github.com/AviSynth/AviSynthPlus/issues to not be forgotten

[Dogway]

I found that certain pixel addressing combinations have a hit on performance, for example the following:
Expr("x[-1,1] x[1,-1] - dup * x[-1,-1] x[1,1] - dup * + sqrt ","") # 447

is about 2% slower than this:
Expr("x[-1,-1] x[1,1] - dup * x[-1,1] x[1,-1] - dup * + sqrt ","") # 455

[tormento]

I found that certain pixel addressing combinations have a hit on performance
Yesterday I did an encode on x264 with SMDegrain (last update of yours) and for the first time I saw the CPU to "idle", i.e. not going to 100% and that was reflected by core temperatures too, that were 4-5 °C lower than usual.

I don't know if you found some not optimized working path and that is correlated.

[Dogway]

After some suggestions I tested overlap with blksize/4 when refinemotion is used, this is more performant but I had to assess quality. I found that it retains more details akin to truemotion=true but not as much probably, similarly grain isn't recovered as much so in a balance I thought it was worth it. Maybe that was it? I plan to run zopti in a few months after I take some rest to further tune SMDegrain and FrameRateConverter.

[DTL]

It is good to add to wiki about RGB and YUV formats (http://avisynth.nl/index.php/Color_spaces ): the digital YUV colorspace is more limited in precision in compare with digital RGB colorspace. So for comparable quality it is about RGB 8bit and YUV 10bit. One consequence mean using YUV8 (YUV 4xx 8bit, typical storage) stage for converting to RGB8 (RGB24, typical display) cause more digital distortions (banding).

YUV cover 'wider' colour gamut in compare with RGB but this cause precision lost with 'natural' colour gamut with same storage integer bits per value. So the conversions RGB<->YUV with same bitdepth are not lossless even with YUV444. The error is about 1 LSB but it visible on smooth colour gradients with 8bit. At first I think it is an issue in Convert() functions.
May be it is already somewhere covered in AVS wiki ?

[LigH]

Your conclusion is not completely correct. The precision reduction has two possible reasons: "TV range" (but full range YUV exists too) and float arithmetics in the YCbCr conversion formulas (avoidable in the YCgCo model). If you don't need to convert between RGB and YUV, then full range YUV has the same precision of 8 bits per component as RGB. Thus, YUV is not generally worse; it is only worse for RGB based devices.

[DTL]

""TV range" (but full range YUV exists too)"

Here is the test: taking RGB $808080 something around mid-grey colour (no-colour grey) and slowly increase saturation of green

BlankClip(color=$FF808080)

ConvertToYUV444(matrix="PC.601")
ConvertToRGB32(matrix="PC.601")

ScriptClip ("""Subtitle (String (AverageR()) + " " + String (AverageG()) +" "+ String (AverageB()))""")


Input $FF808080 - out 128 128 128 = OK
Input $FF808180 - out 129 129 129 < ??? - luma raised, no green colour = error in colour tone and colour saturatrion
Input $FF808280 - out 128 130 127 < ??? - green colour + some minus-blue colour = error in colour tone
Input $FF808380 - out 129 131 128 < ??? - green colour + some plus-red colour = error in colour tone

The only known way to fix - use >8 bit processing of RGB->YUV->RGB:

BlankClip(color=$FF808080)

ConvertBits(16)

ConvertToYUV444(matrix="PC.601")
ConvertToRGB32(matrix="PC.601")

ConvertBits(8)

ScriptClip ("""Subtitle (String (AverageR()) + " " + String (AverageG()) +" "+ String (AverageB()))""")

Input $FF808080 - out 128 128 128 = OK
Input $FF808180 - out 128 129 128 = OK
Input $FF808280 - out 128 130 128 = OK
Input $FF808380 - out 128 131 128 = OK

[poisondeathray]

It is good to add to wiki about RGB and YUV formats (http://avisynth.nl/index.php/Color_spaces ): the digital YUV colorspace is more limited in precision in compare with digital RGB colorspace. So for comparable quality it is about RGB 8bit and YUV 10bit. One consequence mean using YUV8 (YUV 4xx 8bit, typical storage) stage for converting to RGB8 (RGB24, typical display) cause more digital distortions (banding).

YUV cover 'wider' colour gamut in compare with RGB but this cause precision lost with 'natural' colour gamut with same storage integer bits per value. So the conversions RGB<->YUV with same bitdepth are not lossless even with YUV444. The error is about 1 LSB but it visible on smooth colour gradients with 8bit. At first I think it is an issue in Convert() functions.
May be it is already somewhere covered in AVS wiki ?




That is the expected result for 8bit RGB=>YUV=>RGB round trip. +/- 3 value errors

Not just precision and 8bit rounding errors, but multiple 8bit YUV values can map to the same RGB value, which produces more "banding" instead of smooth gradient

In general, 10bit YUV or greater is required for lossless round trip from 8bit RGB => YUV => 8bit RGB, if implemented properly.


But you can demonstrate cases where 10bit is not sufficient with internal Convert(), yet 10bit is sufficient with AvsResize/zimg. ( 10bit also sufficient with other programs like NLE's) .

In this example, 12bit with internal convert is required , but 10bit with AvsResize/zimg or other programs works ok
https://forum.doom9.org/showthread.php?p=1897686#post1897686

This is pinterf's explanation for the difference using internal convert
https://forum.doom9.org/showthread.php?p=1902783#post1902783

[LigH]

(matrix="PC.601") ... please read: Wikipedia: YUV (https://en.wikipedia.org/wiki/YUV), specifically chapters 2.1: SDTV with BT.470 (oh, look, fractional matrix coefficients with 3 decimal places) and 3: Numerical approximations (okay, small integers here, but with rounding errors, and they cause most of the banding, especially because they need to be applied forth and back).

A YCoCg (https://en.wikipedia.org/wiki/YCoCg) matrix has much simpler coefficients, halves and quarters, nice to handle in binary form.

[DTL]

"3: Numerical approximations (okay, small integers here, but with rounding errors, and they cause most of the banding, especially because they need to be applied forth and back)."

May be good to copy to Avisynth wiki (and may be Convert() functions note). May be possible to develop some 'pre-distortions' for 8bit format conversions to minimize colour tone shift (that is more visible ?) ? Currently the distribution of rounding errors is equal for colour tone, luma and saturation errors ? I see typical solutions is add dithering. Unfortunately 8bit still widely used and in end-users displays it is the only supported with h.264 encoding. So using of h.265 with 10bit decreases number of possible users of encoded content.

[pinterf]

Avisynth 3.7.2 test 1
Avisynth+ 3.7.2 test 1 (20220113) (https://drive.google.com/uc?export=download&id=1A2Jb2OSYzGI0tBFEmvlT0RxxLwVoCiP_)

20220113 3.7.2-WIP
------------------
- Fix: Attempt to resolve deadlock when an Eval'd (Prefetch inside) Clip result is
used in Invoke which calls a filter with GetFrame in its constructor.
(AvsPMod use case which Invokes frame prop read / ConvertToRGB32 after having the AVS script evaluated)
Remark: problem emerged in 3.7.1test22 which is trying to read frame properties of the 0th frame in its constructor.
A similar deadlock situation was already fixed earlier in Neo branch and had been backported but it did not cover this use case.
- Fix: Histogram AudioLevels half character upshift (regression since v3.6)
- Bump Copyright year to 2022

[FranceBB]

Nice one, Ferenc!
But... it doesn't work. :(


Remark: problem emerged in 3.7.1test22 which is trying to read frame properties of the 0th frame in its constructor.

the problem is still there I'm afraid... :(


With 3.7.1 Stable:

https://i.imgur.com/hbWuDkI.png

And I had to get rid of frame properties to make it work:

https://i.imgur.com/vWyahcA.png


With 3.7.2 Test 1 it's the same thing:

https://i.imgur.com/WRND0xO.png

and once I get rid of frame properties...

https://i.imgur.com/A4BfjG3.png

Source:


General
Complete name : \\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\UCN12982_GHOST_FRAME.mxf
Format : MXF
Commercial name : XDCAM HD422
Format version : 1.3
Format profile : OP-1a
Format settings : Closed / Complete
File size : 5.50 GiB
Duration : 14 min 58 s
Overall bit rate : 52.6 Mb/s
Package name : Source Package
Encoded date : 2022-01-12 10:16:12.076
Writing application : Avid Technology, Inc. Avid MediaProcessor Plug-In 1.0.54.10052.1
Writing library : MXF::SDK (4.7.8) on Win64 4.7.8.10137.1

Video
ID : 512
Format : MPEG Video
Commercial name : XDCAM HD422
Format version : Version 2
Format profile : 4:2:2@High
Format settings : CustomMatrix / BVOP
Format settings, BVOP : Yes
Format settings, Matrix : Custom
Format settings, GOP : Variable
Format settings, picture structure : Frame
Format settings, wrapping mode : Frame
Codec ID : 0D01030102046001-0401020201040300
Duration : 14 min 58 s
Bit rate mode : Constant
Bit rate : 50.0 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 25.000 FPS
Standard : PAL
Color space : YUV
Chroma subsampling : 4:2:2
Bit depth : 8 bits
Scan type : Interlaced
Scan order : Top Field First
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.965
Time code of first frame : 00:00:00:00
Time code source : Group of pictures header
Stream size : 5.23 GiB (95%)
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

Audio #1
ID : 768
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (AES)
Codec ID : 0D01030102060300-0402020101000000
Duration : 14 min 58 s
Bit rate mode : Constant
Bit rate : 1 152 kb/s
Channel(s) : 1 channel
Sampling rate : 48.0 kHz
Frame rate : 25.000 FPS (1920 SPF)
Bit depth : 24 bits
Stream size : 123 MiB (2%)
Locked : Yes

Audio #2
ID : 1024
Format : PCM
Format settings : Little
Format settings, wrapping mode : Frame (AES)
Codec ID : 0D01030102060300-0402020101000000
Duration : 14 min 58 s
Bit rate mode : Constant
Bit rate : 1 152 kb/s
Channel(s) : 1 channel
Sampling rate : 48.0 kHz
Frame rate : 25.000 FPS (1920 SPF)
Bit depth : 24 bits
Stream size : 123 MiB (2%)
Locked : Yes

Other #1
ID : 1-Material
Type : Time code
Format : MXF TC
Frame rate : 25.000 FPS
Time code of first frame : 10:00:00:00
Time code settings : Material Package
Time code, striped : Yes
Title : Timecode

Other #2
ID : 0-Source
Type : Time code
Format : MXF TC
Frame rate : 25.000 FPS
Time code of first frame : 10:00:00:00
Time code settings : Source Package
Time code, striped : Yes

Other #3
Type : Time code
Format : SMPTE TC
Muxing mode : SDTI
Frame rate : 25.000 FPS
Time code of first frame : 10:00:00:00




Disabling "Read matrix from source or script" in AVSPmod fixes it, but still...

Using VirtualDub, it works, regardless of frame properties:

https://i.imgur.com/4rr1DmN.png

[pinterf]

This issue must be different, there is no Prefetch here.
EDIT:
Can you put a ConvertToRGB32() at end of the script?
(to check if it fails; AvsPMod (as far as I saw in the source) does two additional things after evaluating our original AVS script: reads frame properties and converts the clip to rgb32 in order to display it.


My fix was intended to heal only the "hang" issue.

[gispos]

As soon as FranceBB has it in his hands, he breaks it. :D

Try this version, no problems for me.

32bit:
https://www.mediafire.com/file/ar8qpzeln1ul6so/AvsPmod_v2.7.0.2.3_(Windows_x86-32).zip/file
64bit:
https://www.mediafire.com/file/uvtp8xjutll09sj/AvsPmod_v2.7.0.2.3_(Windows_x86-64).zip/file

[FranceBB]

Try this version, no problems for me.

32bit:
https://www.mediafire.com/file/ar8qpzeln1ul6so/AvsPmod_v2.7.0.2.3_(Windows_x86-32).zip/file
64bit:
https://www.mediafire.com/file/uvtp8xjutll09sj/AvsPmod_v2.7.0.2.3_(Windows_x86-64).zip/file

Nope. :(
Same error as before.
Please note that this happens only with XDCAM files muxed in .mxf
I can share a sample if you want, 'cause I've just tried with a ProRes file muxed in .mov and it works just fine.
Same goes for an H.264 file in .mp4.

As soon as FranceBB has it in his hands, he breaks it. :D

eheheheh nah, it's more like: "literally no one except FranceBB uses weird FULL HD yv16 50 Mbit/s 25i TFF MPEG-2 files muxed in mxf anymore, so no one ever checked" xD


Can you put a ConvertToRGB32() at end of the script?



Interesting error, it fails with this:

https://i.imgur.com/nW1r5LT.png

however this is a bit weird given that it's a simple yv16 with the standard MPEG-2 chroma placement (after all it IS an MPEG-2 eheheheh).

Adding propclearall() fixes it and allows me to use ConverttoRGB32():

video=LWLibavVideoSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\UCN12982_GHOST_FRAME.mxf")
ch1=LWLibavAudioSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\UCN12982_GHOST_FRAME.mxf", stream_index=1)
ch2=LWLibavAudioSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\UCN12982_GHOST_FRAME.mxf", stream_index=2)
audio=MergeChannels(ch1, ch2, ch1, ch2, ch1, ch2, ch1, ch2)
AudioDub(video, audio)

propClearAll()

ConverttoRGB32()

speaking of which, let me see if I can just get rid of the chroma location property and try with this:

video=LWLibavVideoSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\UCN12982_GHOST_FRAME.mxf")
ch1=LWLibavAudioSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\UCN12982_GHOST_FRAME.mxf", stream_index=1)
ch2=LWLibavAudioSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\UCN12982_GHOST_FRAME.mxf", stream_index=2)
audio=MergeChannels(ch1, ch2, ch1, ch2, ch1, ch2, ch1, ch2)
AudioDub(video, audio)

propDelete("_ChromaLocation")


Success!
Getting rid of the Chroma Location property makes it work! :eek:

https://i.imgur.com/rlraTGU.png


So now we know what happened!
Basically ConverttoRGB32() fails due to the Chroma Location frame property, therefore when I try to use AVSPmod, it tries to convert to RGB32 invoking such a function, but since such a function fails, it won't display anything and fail instead!!

[pinterf]

Thanks! Can you put a propShow() before the property delete, to see what the original _ChromaLocation was set to?

[gispos]

(to check if it fails; AvsPMod (as far as I saw in the source) does two additional things after evaluating our original AVS script: reads frame properties and converts the clip to rgb32 in order to display it.
The clip itself is not converted to RGB32. A DisplayClip is created which is converted to RGB32.
But the frame properties (matrix) are read from the original source clip.

[FranceBB]

Thanks! Can you put a propShow() before the property delete, to see what the original _ChromaLocation was set to?

Sure thing, there you go:

https://i.imgur.com/f736Nld.png

Looks like the indexer is reporting another kind of Chroma Location, 'cause it should be _ChromaLocation = 0 left (mpeg2) and not _ChromaLocation = 2 top_left... interesting...

This is another XDCAM file and it has the very same problem:

https://i.imgur.com/igT1odL.png

and ffprobe seems to agree with the indexers:

[STREAM]
index=0
codec_name=mpeg2video
codec_long_name=MPEG-2 video
profile=4:2:2
codec_type=video
codec_tag_string=[0][0][0][0]
codec_tag=0x0000
width=1920
height=1080
coded_width=0
coded_height=0
closed_captions=0
film_grain=0
has_b_frames=1
sample_aspect_ratio=1:1
display_aspect_ratio=16:9
pix_fmt=yuv422p
level=2
color_range=tv
color_space=bt709
color_transfer=bt709
color_primaries=bt709
chroma_location=topleft

This is yet a different XDCAM file I received from A&E:

index=0
codec_name=mpeg2video
codec_long_name=MPEG-2 video
profile=4:2:2
codec_type=video
codec_tag_string=[0][0][0][0]
codec_tag=0x0000
width=1920
height=1080
coded_width=0
coded_height=0
closed_captions=0
film_grain=0
has_b_frames=1
sample_aspect_ratio=1:1
display_aspect_ratio=16:9
pix_fmt=yuv422p
level=2
color_range=tv
color_space=unknown
color_transfer=bt709
color_primaries=unknown
chroma_location=topleft

and this is a movie I've got from Notorious Pictures and it says top left too:

[STREAM]
index=0
codec_name=mpeg2video
codec_long_name=MPEG-2 video
profile=4:2:2
codec_type=video
codec_tag_string=[0][0][0][0]
codec_tag=0x0000
width=1920
height=1080
coded_width=0
coded_height=0
closed_captions=0
film_grain=0
has_b_frames=1
sample_aspect_ratio=1:1
display_aspect_ratio=16:9
pix_fmt=yuv422p
level=2
color_range=tv
color_space=unknown
color_transfer=bt709
color_primaries=unknown
chroma_location=topleft


Ok, either there's something fishy and totally broken here, or I've lost my certainties...

[FranceBB]

Ok, guys, for the glory and my mental health, what's the REAL chroma location of this file: https://we.tl/t-MhjJWTj0lQ

and is the metadata just wrong or is it FFMpeg and the indexers that are not getting it right?

(link available for 7 days)

[pinterf]

Topleft is treated as invalid for 422, it's accepted only for 420

[FranceBB]

Topleft is treated as invalid for 422, it's accepted only for 420

and it makes sense, it shouldn't exist for 4:2:2, like, I've never ever seen it.
I mean, let's think this through logically.

Top Left is Type 2, so the one generally found in H.265 UHD HDR PQ BD.

Now, those files are MPEG-2 50 Mbit/s BT709 SDR in FULL HD and 4:2:2 8bit planar.
Sony made the XDCAM standard in 2003 and the XDCAM-50 is from 2006 or something like that if I remember correctly.
Back then, Type 2 top left didn't even exist, did it?
Besides, I would find hard to believe that such a standard uses top left instead of left, given that it's an MPEG-2, right?

So we can assume that indexers and ffprobe are all wrong and that it's actually Type 0, left, MPEG-2, right?

If someone is willing to download the masterfile and check I would really appreciate it.
Also 'cause if it's actually Type 0 and indexers + FFMpeg say "Type 2", this is a BIG BIG PROBLEM and we need a fix soon.

[Dogway]

Same error trying to display the clip, I'm stable 3.7.1 and AvspMod 2.6.7.9 with "read from source script" disabled. propclearall() fixes it. I will try to check chroma placement, haven't seen an empirical method to find out so.

EDIT: Yes, looks MPEG2 to me

[takla]

and it makes sense, it shouldn't exist for 4:2:2, like, I've never ever seen it.
I mean, let's think this through logically.

Top Left is Type 2, so the one generally found in H.265 UHD HDR PQ BD.

Now, those files are MPEG-2 50 Mbit/s BT709 SDR in FULL HD and 4:2:2 8bit planar.
Sony made the XDCAM standard in 2003 and the XDCAM-50 is from 2006 or something like that if I remember correctly.
Back then, Type 2 top left didn't even exist, did it?
Besides, I would find hard to believe that such a standard uses top left instead of left, given that it's an MPEG-2, right?

So we can assume that indexers and ffprobe are all wrong and that it's actually Type 0, left, MPEG-2, right?

If someone is willing to download the masterfile and check I would really appreciate it.
Also 'cause if it's actually Type 0 and indexers + FFMpeg say "Type 2", this is a BIG BIG PROBLEM and we need a fix soon.

Maybe get in touch with ffmpeg devs?
https://ffmpeg.org/contact.html#IRCChannels
https://ffmpeg.org/contact.html#MailingLists

[FranceBB]

Maybe get in touch with ffmpeg devs?
https://ffmpeg.org/contact.html#IRCChannels
https://ffmpeg.org/contact.html#MailingLists

I opened a ticket: https://trac.ffmpeg.org/ticket/9598#ticket

[Balling]

Well, no, top-left is found in DV. See wikipedia. As for 4:2:2 is not there only 1 possible placement?

[Reel.Deel]

and it makes sense, it shouldn't exist for 4:2:2, like, I've never ever seen it.
I mean, let's think this through logically.


I was under the impression that 4:2:2 has always had a "left" (type 0) chroma location. My understanding is that the "left" chroma location was introduced in MPEG2, before then it was only centered since that was the spec on MPEG1. I've had some digital cameras that output 4:2:2 JPEGs and on all of them the "YCbCrPositioning property (https://freeimage.sourceforge.io/fnet/html/4A015DE9.htm)" is set to co-sited, aka left. Also, it would not make much sense for 4:2:2 to be top left since the chroma has full vertical resolution. If that were to be the case it would mean shifting the chroma half a pixel down when converting to RGB, for no good reason. I have not tested but would be curious if x265 even allows the chroma location to be set to top left for 4:2:2 sources. I've never seen any illustration/articles that show/mention a different chroma placement for 4:2:2.

Some things I've found:

MPEG-2 FAQ (https://web.archive.org/web/20010209092224/http://bmrc.berkeley.edu/research/mpeg/faq/mpeg2-v38/faq_v38.html)
How are the subsampled chroma samples cited ?
A. It is moderately important to properly co-site chroma samples, otherwise a sort of chroma shifting effect (exhibited as a "halo") may result when the reconstructed video is displayed. In MPEG-1 video, the chroma samples are exactly centered between the 4 luminance samples (Fig 1.) To maintain compatibility with the CCIR 601 horizontal chroma locations and simplify implementation (eliminate need for phase shift), MPEG-2 chroma samples are arranged as per Fig.2.

[Mjpeg-users] chroma sample alignment (https://mjpeg-users.narkive.com/O1iMu1ro/chroma-sample-alignment#post3)
As you say in your web page, 4:2:2 always has co-sited chroma samples...

Ok, guys, for the glory and my mental health, what's the REAL chroma location of this file: https://we.tl/t-MhjJWTj0lQ

and is the metadata just wrong or is it FFMpeg and the indexers that are not getting it right?


MediaInfo says it's Interlaced TFF :D.

Edit:

Well, no, top-left is found in DV. See wikipedia. As for 4:2:2 is not there only 1 possible placement?

It's a different top-left. Read the following posts: https://forum.doom9.org/showthread.php?p=1953360#post1953360

[FranceBB]

I was under the impression that 4:2:2 has always had a "left" (type 0) chroma location. My understanding is that the "left" chroma location was introduced in MPEG2, before then it was only centered since that was the spec on MPEG1.

I'm under this impression too, 4:2:2 has always had left, type 0, MPEG-2 chroma placement.


it would not make much sense for 4:2:2 to be top left since the chroma has full vertical resolution.


Exactly!


I've never seen any illustration/articles that show/mention a different chroma placement for 4:2:2.


Neither have I, in fact.

Yes, looks MPEG2 to me

Right! So an XDCAM-50 stream, namely an MPEG-2 stream at 50 Mbit/s 4:2:2 yv16 which is using the Type 0, left, MPEG2 chroma placement, as we all thought!
So FFProbe is getting it wrong, so are the indexers ffms2 and LSMASH.dll.

Also this
https://i.ibb.co/TKmRLfX/page431.png

seems to confirm the fact that for 4:2:2 the chroma location is always Type 0, left, MPEG-2, especially for MPEG-2 streams like XDCAM-50, right?

So we all agree that FFProbe, LWLibavVideoSource and FFVideoSource all report the WRONG chroma location as it shouldn't be top left at all and that's a bug and needs to be fixed, right?
This is important 'cause I opened a bug here: https://trac.ffmpeg.org/ticket/9598#comment:5 and if it's fixed in FFProbe / FFMpeg, it will be consequently fixed in LSMASH and ffms2.

[VoodooFX]

How VS guys didn't noticed this bug?

No idea what Balling implies on https://trac.ffmpeg.org/ticket/9598#comment:4, in "Figure D.2" I see only one chroma placement, and it's not topleft.

[DTL]

4:2:2 simply rare for home users and possible difference between left and top-left signalling may be never cause any (visible) errors.

[StvG]

How VS guys didn't noticed this bug?

It could be because zimg does treat all three locations top_left, left, bottom_left for 4:2:2 the same way (left).

[FranceBB]

How VS guys didn't noticed this bug?

That's odd...
And not just the VapourSynth guys, but also the ffmpeg guys...



https://trac.ffmpeg.org/ticket/9598#comment:4, in "Figure D.2" I see only one chroma placement, and it's not topleft.

Precisely my point.

4:2:2 simply rare for home users

Nah, just like Avisynth is used in professional settings (Crunchyroll, Viewster, Sky, RecordTV, NRK, CentrevilleTV, and plenty others across the world), I'm pretty sure VapourSynth is used somewhere too (although I can't name any 'cause I don't use it). Same goes for FFMpeg/FFProbe, which are both used in professional settings too by some companies. If we think about even just some of the user base here on Doom9, we have almost all the major streaming companies here eheheheheheheh (TL;DR Alex works at Hulu, Ben works at Amazon, Derek works at Disney, me and Livio work at Sky etc). Even you, DTL, I saw you discussing the resizing kernels topic, in particular SincPow2 and not only you had a deep insight about how thing worked and the math behind it and you could translate it into code that was able to run and be integrated into Jean Philippe's plugins, but you were testing the results and in particular aliasing on a professional waveform monitor by Tektronix, similar to the one I have sitting right next to me here at Sky, which is far too expensive for a home user, so... I'm pretty sure you work for some broadcasting company too ehehehehe
Now I'm curious, which one is it? :P

It could be because zimg does treat all three locations top_left, left, bottom_left for 4:2:2 the same way (left).

This is much more likely. :)

[DTL]

" in professional settings too by some companies. "

It is a sad feature of the end of dying civilization - even in the 'pro companies' almost no one understand how the digital visual technologies working. Also the quality control if even exist in some very poor form mostly can not dig in such thing like 'chroma placement'. I see currently only a few fans in the nowdays dying internet network at this planet still trying to keep things in the old software for digital moving pictures processing in some logic and quality. The general idea of the companies: If it digital - it is just perfect and no more any pro-s required to support it. Company simply buy pro-cameras and pro-NLEs and it work without any service but cleaning cooling from dust for decades and changed to next generation of digital cams and NLEs and air servers/switches. Even the worker 'engineer' sitting near the still used 'multi-functional measuring equipment' (Tek/Leader/other) rasterizer mostly can only check max/white level at iris setup and mostly tune picture by image in simple control monitor by its taste - not by numbers at the analyzer. It looks the quality is not required nowdays and I do not see/hear any complain from end-users for poor quality of broadcasting (really sent to the broadcasting company - not just posed somewhere in the internet). So no complains and 'digital equipment' mean no more any engineers really required at production. So if even some broadcasting company will many years produce fulltime air with not left but top-left chroma it mostly no one will see. The actual datastream for nowdays endusers of air broadcasting is significantly degraded by the very low MPEG 4:2:0 bitrate so have many more severe distortions in addition with possible slight chroma-shift from left/top-left chroma treatment in XDCAM sources.
The main pro companies activity is about making money - not about making perfect digital moving images content.

"which one is it? "

Currently most of organizations require to sign some form of NDA with restriction to make public any data that may harm the company commertial activity. So if some possible harmful content is typed in some internet forum - it is no good to make additional signs where from it can be. Unfortunately the degradation in digital moving industry hits not only the small broadcasting companies.

Also the more 'perfect' you make the digital content production tools (even freeware ffmpeg) - the less engineers reqiuired and in a few years they physically disappear. Someday the broadcasting owners find that no one left to help with any new issue.

At old analog days with servicing broadcast equipment about dayly with real understanding engineers - this helps to production and support some broadcasting engineers pool with ability to understand how things working.

[FranceBB]

"
It is a sad feature of the end of dying civilization - even in the 'pro companies' almost no one understand how the digital visual technologies working.

This is really sad, yet true... :(
I've witnessed this myself in some occasions...


Also the quality control if even exist in some very poor form mostly can not dig in such thing like 'chroma placement'.


I guess...
Most of the things that are checked are luma and chroma levels, subtitles alignment for TTX muxed as OP47 in mxf and few other things over here too by some of my colleagues... :( Unfortunately, algia, (so Livio Aloja), the only other colleague who was lurking here on Doom9 retired and he's not working here any longer... I still miss him 'cause he was one of the few people who actually really cared...
There's still Fabio Sonnati here on Doom9 from Sky, but he works in the technology department (i.e they're the ones who get the already encoded and conformed, loudness corrected mezzanines I encode and re-encode them in H.264 or H.265 for the end users), so we don't really work together...


Also the more 'perfect' you make the digital content production tools (even freeware ffmpeg) - the less engineers reqiuired and in a few years they physically disappear. Someday the broadcasting owners find that no one left to help with any new issue.


I've been talking about open source software with colleagues both here and from other TVs and we all seem to agree that the EBU should actually invest some of its budget in open source software, 'cause there are some features that otherwise will never be implemented. Take DolbyE indexing and decoding, it will never be properly addressed 'cause a tiny amount of user-base is interested in this. Of course, there are work-around and I've written lots of code to work around it and instruct FFMpeg into decoding it properly, but it needs Mediainfo (which is also at fault 'cause it reports like 2 tracks instead of 1 with 5.1+2.0) and some logic to instruct an ffmpeg decoder beforehand with what is going to receive, so there's nothing like an automatic detection and decoding, sadly. :(
If we had the EBU investing into things like indexers/decoders and also open source encoders, it would be a game changer.
Think about the XAVC intra class 300 and 480 inside x264. If it wasn't for ifb, Bug Master (Anton Mitrofanov), Jean Philippe and in a very little part me, it would have never been implemented, but it could have been implemented if the EBU funded it like it did for intra class 50, 100 and 200 (along with NRK). Speaking of which, if it wasn't for Steinar Apalnes, Kiearn Kunhya and NRK we wouldn't have had any implementation of the intra class at all in the first place. Another thing that is still not very reliable is the FFMpeg mxf muxer and I've been reporting issues which have later been fixed more than once. If it wasn't for the good heart of the BBC developers who made BMX Transwrap and raw2bmx open source and free, God knows what I would have done...
And yet, if the EBU funded it, we would have had a good compliant muxer in FFMpeg as well.





At old analog days with servicing broadcast equipment about dayly with real understanding engineers - this helps to production and support some broadcasting engineers pool with ability to understand how things working.


Yeah... It seems weird, but I've seen plenty more screw-ups in digital contents than in analog ones...
It's almost a paradox, but it's just how things are... :(
I'm glad to have found Doom9 in 2006, I'm glad to have been lurking for years and I'm glad to have learned everything from the right place and the right people, 'cause university alone wouldn't have even remotely been enough.

[DTL]

"it could have been implemented if the EBU funded it."

You can try to join some 'video' groups of EBU (at least VS-all) or may be VS-EBU if your company is EBU member and post the ideas/questions/suggestions to mailing lists or individual projects in 'workspaces'. May it will helps somehow. Link is https://tech.ebu.ch/groups/video . Send e-mail request to Frans de Jong about joining available workgroups.

[FranceBB]

Ok, the FFMpeg guys replied and this is something interesting for us all:


If the chroma plane is both vertically and horizontally aligned with the Luma plane, which MPEG-2 4:2:2 is, then "topleft" actually seems like the correct value - because it indicates the position of the first chroma sample, as per the ffmpeg documentation of AVChromaLocation.
You could even argue that for 4:2:2 both "left" and "topleft" are supposed to be handled identically, due to the absence of vertical subsampling, so in practice it would make no difference. The only reason h264 would result in a different value is that the interpretation of different developers differed on what to call it, but neither mpeg2 or h264 even encode a chroma location value for 422, and always contain the same one.
The entire "type 0" and "type 2" thing only applies to 4:2:0, so its best to just not bring it into the discussion at all. 4:2:2 in reality only ever has one chroma position, and if you want to call that left or topleft, which for both would exist arguments, is rather irrelevant, because there should be no difference between them.
TL;DR
You might as well ignore chroma location for 4:2:2, since it never differs as per the specification of both mpeg2 and h264. Or at the very least instruct your code to handle left and topleft identically for 4:2:2, because they are. mpeg2, h264, and h265 for that matter, do not even encode the chroma location for 422 - therefor you cannot associate any "type X" with it, since no type number is even specified. Its just 422 chroma, no alternates are possible.



So, in a nutshell, 4:2:2 doesn't have a chroma location, it's literally always the same. x262, x264, x265 don't encode any value for the chroma location and what ffprobe does (as well as the indexers) is reporting left for H.264 encoded streams and top left for MPEG-2 encoded streams, but even though the name is different, they're actually exactly the same.

In other words, in "Avisynth terms", it's always "left", so I think we have two choices:


1) We change this behavior in indexers so that they're gonna report "unknown" (or "left" eventually) every time we get a 4:2:2 stream

2) We make the internal resizers handle 4:2:2 anyway assuming the old MPEG-2 chroma location every time, no matter what the indexer says instead of throwing an error


Ferenc, you're the boss here, the decision is yours. ;)

[pinterf]

A boss knows when he is undereducated in a topic he can always rely on the opinion of quality consultants :) Verdict: let's treat top left the same way as left. (?)

[FranceBB]

Verdict: let's treat top left the same way as left. (?)

Sounds good to me. I'll wait others to confirm this too, though, then xD

[anton_foy]

Sounds good to me. I'll wait others to confirm this too, though, then xD

Mob rule? :eek:
But to be blunt what does it mean if chroma is located faulty in this way? Chroma bleeding?

[FranceBB]

what does it mean if chroma is located faulty in this way?


So if you handle let's say 4:2:0 Type 2 as Type 0?
Well, at high enough resolutions like UHD it will be hard to spot, but it will look blurrier and slightly out of the boundaries / shifted towards a side.


Chroma bleeding?


Sort of, but the higher the resolution, the lower the shift, the less you'll notice.

This is a chroma shift happening in an SD material, however I exasperated the shift a bit by 3 points rather than 0.5 to show you the effect:

https://i.imgur.com/6i38khc.png
https://i.imgur.com/bWeSVea.png

Let's now suppose to have a real life example in which we're wrongly assuming a chroma location and we're performing the 0.5 shift (when we shouldn't):

https://i.imgur.com/o46iMzc.png

Can you see it? No? Well, you're not alone, it takes a 2000% zoom in to get a proper grasp to what it does:

https://i.imgur.com/182F5of.png

you can see the color tone is different (wrong) on the left compared to the right, so as you can see a real life shift is so subtle that most people will never notice, however it's there and it's wrong.

[Reel.Deel]

The effect of correct and wrong chroma placement can be better seen with a synthetic example:

https://i.ibb.co/3kTYhvb/chromalocations.png

Here the RGB sample was converted to YUV420 using the "MPEG2" chroma placement. The it was converted back to RGB with the corresponding chroma placement, as shown in the image. Then PointResize 4X since it's easier to see.
As FranceBB said, it's hard to spot the wrong chroma placement, especially in live footage.

Here's the script I used to create the example:

Blankclip(width=48, height=16, pixel_type="RGB32", color=$FF0000, length=1) #red
AddBorders(0,0,0,16,$00FF00) # green
AddBorders(0,0,0,16,$0000FF) # blue
AddBorders(0,0,0,16,$00FFFF) # cyan
AddBorders(0,0,0,16,$FF00FF) # magenta
AddBorders(0,0,0,16,$FFFF00) # yellow
src = last

mask = src.ExtractR().mt_lutspa(mode="absolute", expr="x 32 % 16 < 0 1 ? y 32 % 16 < 0 1 ? + 1 == 0 255 ?")
mask2 = src.ExtractR().mt_lutspa(relative=false, expr="x 16 % 0 == y 16 % 0 == | 255 0 ?")

grid1 = Overlay(src, src.FlipVertical(), mask=mask)
grid2 = Overlay(src, src.FlipVertical(), mask=mask2)

StackHorizontal(grid1, grid2)
src2 = last.AddBorders(0,0,4,0)

ConvertToYV12(ChromaOutPlacement="mpeg2")

mpeg1 = last.ConvertToRGB32(ChromaInPlacement="mpeg1")
mpeg2 = last.ConvertToRGB32(ChromaInPlacement="mpeg2")
topleft = last.ConvertToRGB32(ChromaInPlacement="top_left")

Interleave(mpeg1, mpeg2, topleft)

StackHorizontal(src2, last)
PointResize(width*4, height*4)
AddBorders(0,16,0,0)

Text("RGB Source")
Text("MPEG1 CHROMA LOCATION (WRONG)", x=51*8, first_frame=0, last_frame=0)
Text("MPEG2 CHROMA LOCATION (CORRECT)", x=51*8, first_frame=1, last_frame=1)
Text("TOPLEFT CHROMA LOCATION (WRONG)", x=51*8, first_frame=2, last_frame=2)

AssumeFPS(1,2)

[anton_foy]

Thanks for the demonstrations. Probably this is happening to my UHD-footage when I import it into avs+ because it has got a slight color shift compared to the original file. Any suggestions on how to correct this? Maybe I need to upload a sample?

Edit: it is XAVC-S 8-bit 4:2:0 I think it is some mpeg4 format.

[DTL]

"how to correct this?"

Typically it is applying resample to YUV format with sub-sample UV (or Y is it give better result) shift to the required direction.

[FranceBB]

it is XAVC-S 8-bit 4:2:0 I think it is some mpeg4 format.

XAVC should be interpreted as left type 0, but it's always nice to double check. Feel free to upload a sample. A landscape will do, although someone holding a colour palette would be better.

[anton_foy]

XAVC should be interpreted as left type 0, but it's always nice to double check. Feel free to upload a sample. A landscape will do, although someone holding a colour palette would be better.

Thanks DTL and FranceBb
Yes I have somewhere a clip with an IT8 chart. Will locate and upload it asap.

[wonkey_monkey]

On the page for Invoke, it says:

env->Invoke does not automatically insert a cache after the filter you invoke. That means that each time you request a frame (using GetFrame) from the invoked filter the filter will need to generate the frame.

Can I rely on this remaining true in future versions? Or can it force it to be so (guaranteed no caching) with SetCacheHints?

Edit: actually the above quote doesn't seem to be true now. I tried Invoking a debugging filter and I can see that it stops being called if I keep requesting the same frames. And SetCacheHints now seems to do nothing?

int __stdcall SetCacheHints(int cachehints, int frame_range) { AVS_UNUSED(cachehints); AVS_UNUSED(frame_range); return 0; };

So I guess my question should just be... can I completely disable caching on an Invoke-generated clip?

[Reel.Deel]

Why does ConvertBits() shift the U/V planes from 0 to 1 in the script below? I thought setting fulls/fulld to true when working with full range would keep the pixel values unchanged. When I set fulls/fulld to false, the shift does not happen :confused:.

Blankclip(width=7, height=12, pixel_type="RGB32", colors=[0,0,0,0], length=1)
Text("T", size=12, text_color=$ffffff, halo_color=$000000, x=1)
ExtractR()
YToUV(last, last, last)
fulls = true
ConvertBits(last, 16, fulls=fulls, fulld=fulls)
ConvertBits(last, 8, fulls=fulls, fulld=fulls)
ExtractU()
Pixelscope()

https://i.ibb.co/ZYh9Ld6/convertbug-maybe.png

[wonkey_monkey]

Someone uses Pixelscope! Yay! :cool:

I must get around to updating for all bit depths one day.

[Reel.Deel]

Someone uses Pixelscope! Yay! :cool:

I must get around to updating for all bit depths one day.

I actually wanted to ask you about that. I wanted to use it after converting to 16-bit to see the values but forgot is limited to 8-bit.

I would also use your Polygon filter if it were available in 64-bit :sly:.

[StainlessS]

I on occasion use PixelScope.

[Emulgator]

Pixelscope HBD: yes, please !

[gispos]

My mistake?

ColorBarsHD(width=1280, height=720, pixel_type="YV24")
ConvertBits(16)
ConvertToRGB64()

Returns other pixel values than

ColorBarsHD(width=1280, height=720, pixel_type="YV24")
ConvertToRGB64()

Had tried to compare pixel values of Planar RGB 16bit with Packet RGB 16bit and I'm almost desperate that I always got different values.

This then brought success (both the same pixel values)

ColorBarsHD(width=1280, height=720, pixel_type="YV24")
ConvertBits(bits=16)
ConvertToPlanarRGB()

ColorBarsHD(width=1280, height=720, pixel_type="YV24")
ConvertBits(16)
ConvertToRGB64()

Or would I have had to set parameters with ConvertBits?

[DTL]

YUV 8bit to RGB 16bit is 2 conversions: YUV->RGB and 8bit->16bit.

8bit->16bit integer is lossless adding 8 zeroes to 8bit data. YUV->RGB is not lossless math. So with 8bit YUV input ConvertToRGB64() can firstly convert to RGB in 8bit and secondly add 8 zeroes to RGB. With ConvertBits(16) before RGB it explicitly command first add 8 zeroes to YUV and convert 16bit YUV->RGB.

So it mostly probably shows as 8bit and 16bit YUV->RGB gives different results.

Yes - for better results it recommended to convert YUV->RGB in 16bits. It gives less errors (less banding on gradients). So if ConvertToRGB64() with 8bit YUV input makes first conversion to RGB in 8bit it may be an issue to fix. As current workaround - add ConvertBits(16) before ConvertToRGB().

[gispos]

So if ConvertToRGB64() with 8bit YUV input makes first conversion to RGB in 8bit it may be an issue to fix. As current workaround - add ConvertBits(16) before ConvertToRGB().
But still, one should think that there should be no difference 16bit is 16bit. It made me despair for 2 evenings.

You never stop learning.;)

[DTL]

The Convert* functions have 'hidden first argument' for easy of use so it is really 2 different processings:
ConvertYUV8toRGB64() and ConvertYUV16toRGB64().

Wiki says for ColorBarsHD - http://avisynth.nl/index.php/ColorBars

AVS+ "YV24" or other 4:4:4 format.

May be try YUV444P16 ? May be it can output RGB48/RGB64 directly ?

[pinterf]

Why does ConvertBits() shift the U/V planes from 0 to 1 in the script below? I thought setting fulls/fulld to true when working with full range would keep the pixel values unchanged. When I set fulls/fulld to false, the shift does not happen :confused:.

Chroma full scale has valid values from 1 to 255.

U and V chroma are signed values and one would treat them as such in operations, even when stuffed into a 8 bit container.

16-240 means +/-112 with a bias of 128.

During calculations one must subtract the bias to have zero centered values, do the operation and convert it back by adding the bias.

For consistent full scale chroma operations we chose the available maximum symmetrical boundaries; +/-112 is scaled to +/-127.

With bias 128 it turnes into 128+/-127, that is 1-255. 0 is an invalid value there.

Same applies to other integer bit depths as well: for 16 bit 32768 +/- 32767 is the valid range.

[pinterf]

On the page for Invoke, it says:



Can I rely on this remaining true in future versions? Or can it force it to be so (guaranteed no caching) with SetCacheHints?

Edit: actually the above quote doesn't seem to be true now. I tried Invoking a debugging filter and I can see that it stops being called if I keep requesting the same frames. And SetCacheHints now seems to do nothing?

int __stdcall SetCacheHints(int cachehints, int frame_range) { AVS_UNUSED(cachehints); AVS_UNUSED(frame_range); return 0; };

So I guess my question should just be... can I completely disable caching on an Invoke-generated clip?
I remember a commit from recent years.
EDIT: yes, it appeared in commit e79f82be (https://github.com/AviSynth/AviSynthPlus/commit/e79f82be5bad10f6b5974480d177ef18ae8ae8aa) the modification is here (https://github.com/AviSynth/AviSynthPlus/commit/e79f82be5bad10f6b5974480d177ef18ae8ae8aa#diff-f8057bae068be1f19fd7f1e2349931996b7989b45426a049ace0883a429bd32aR3285) after cherrypicking from Neo branch, here it is still commented out, then I made it live.

EDIT2:

Or you handle and return 1 on CACHE_DONT_CACHE_ME request, as in the Avisynth code below appears.

int __stdcall NonCachedGenericVideoFilter::SetCacheHints(int cachehints, int frame_range)
{
switch(cachehints)
{
case CACHE_DONT_CACHE_ME:
return 1;
case CACHE_GET_MTMODE:
return MT_NICE_FILTER;

case CACHE_GET_DEV_TYPE:
return (child->GetVersion() >= 5) ? child->SetCacheHints(CACHE_GET_DEV_TYPE, 0) : 0;

default:
return GenericVideoFilter::SetCacheHints(cachehints, frame_range);
}
}

[anton_foy]

XAVC should be interpreted as left type 0, but it's always nice to double check. Feel free to upload a sample. A landscape will do, although someone holding a colour palette would be better.

Here is a clip (30mb) of an IT8 color chart on the floor, it is jittery and shaky though but I did not find any other.
IT8 color chart clip (https://we.tl/t-83WtkIjgMZ)

[Wilbert]

I made some AviSynth+ threads here and in the usage forum sticky.

[StvG]

Chroma full scale has valid values from 1 to 255.

But zimg and fmtconv returns 0 in this example (meaning they have valid values from 0 to 255 for chroma full scale).

[Reel.Deel]

Chroma full scale has valid values from 1 to 255.

U and V chroma are signed values and one would treat them as such in operations, even when stuffed into a 8 bit container.

16-240 means +/-112 with a bias of 128.

During calculations one must subtract the bias to have zero centered values, do the operation and convert it back by adding the bias.

For consistent full scale chroma operations we chose the available maximum symmetrical boundaries; +/-112 is scaled to +/-127.

With bias 128 it turnes into 128+/-127, that is 1-255. 0 is an invalid value there.

Same applies to other integer bit depths as well: for 16 bit 32768 +/- 32767 is the valid range.

Thanks for the explanation pinterf. So how would one go about treating YUV masks that are expected to be [0,255]? For single plane masks there is no problem, but for YUV masks it does create a problem. I still don't understand why fulls/fulld = false does not cause the 0 to 1 shift.

But zimg and fmtconv returns 0 in this example (meaning they have valid values from 0 to 255 for chroma full scale).

Thanks for pointing that out. I had not checked their behavior.

[cretindesalpes]

For consistent full scale chroma operations we chose the available maximum symmetrical boundaries; +/-112 is scaled to +/-127.
ITU Rec H.273 specifies it slightly differently (for full range, eq. 30 p. 10):
Cb = Clip1_C ( Round( ( ( 1 << BitDepth_C ) − 1 ) * E′_PB ) + ( 1 << ( BitDepth_C − 1 ) ) )
Same thing with Cr. In 8 bits, chroma is scaled with a factor 255, meaning that -0.5 and +0.5 map to 0.5 and 255.5 respectively, before rounding and clipping. Of course this slightly truncates the chroma values which are very close to +0.5, but is this really a problem for real-world video signals? Full range already truncates ringing that naturally occurs with resizing or with other filtering operations.

[pinterf]

ITU Rec H.273 specifies it slightly differently (for full range, eq. 30 p. 10):
Cb = Clip1_C ( Round( ( ( 1 << BitDepth_C ) − 1 ) * E′_PB ) + ( 1 << ( BitDepth_C − 1 ) ) )
Same thing with Cr. In 8 bits, chroma is scaled with a factor 255, meaning that -0.5 and +0.5 map to 0.5 and 255.5 respectively, before rounding and clipping. Of course this slightly truncates the chroma values which are very close to +0.5, but is this really a problem for real-world video signals? Full range already truncates ringing that naturally occurs with resizing or with other filtering operations.
That means that instead of +/-127 (or +/-32767), it must be handled as 128+/-127.5 (or 32768 +/-32767.5) before it gets truncated back to 8 (16) bit range. Then the conversion must use factor of 112 / 127.5 (that is 224.0/255.0) instead of my present 112/127.

The formula which calculates the ratio in https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/convert/convert_helper.h#L180 and
https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/convert/convert_helper.h#L183
must be corrected accordingly.
I don't know (did not calculate) whether it survives a cumulative full-limited conversion sequence throughout the range or on the extremes.

[qyot27]

The documentation in HTML-generated form is now live on Read The Docs:
https://avisynthplus.readthedocs.io/en/latest/

(the other branches are still generating)

[Balling]

That means that instead of +/-127 (or +/-32767), it must be handled as 128+/-127.5 (or 32768 +/-32767.5) before it gets truncated back to 8 (16) bit range. Then the conversion must use factor of 112 / 127.5 (that is 224.0/255.0) instead of my present 112/127.

The formula which calculates the ratio in https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/convert/convert_helper.h#L180 and
https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/convert/convert_helper.h#L183
must be corrected accordingly.
I don't know (did not calculate) whether it survives a cumulative full-limited conversion sequence throughout the range or on the extremes.

Indeed we came with this insanity in chromium/chrome. But I think it only matters for 10 bit and more. Also see my comments https://bugs.chromium.org/p/chromium/issues/detail?id=1174638

and https://chromium-review.googlesource.com/c/chromium/src/+/2658149/3/ui/gfx/color_space.cc#1098

There is this merge request but it breaks too much stuff. https://chromium-review.googlesource.com/c/chromium/src/+/3077005/

[Balling]

Someone uses Pixelscope! Yay! :cool:

I must get around to updating for all bit depths one day.

Already works in YUView hex view. Sumsampled planes too and what not, up to 16 bits. It is frankly speacking nuts! The only way I know to show hex values on 10 bit! 10 bit YCbCr to 64 bit PNG is broken BTW in swscale, for example.

Big endian was recently added for rgb formats due to my request! Just chroma sample location and beter FIR chroma upsample is needed.

[Balling]

But zimg and fmtconv returns 0 in this example (meaning they have valid values from 0 to 255 for chroma full scale).

This is YCbCr. Black in full YCbCr is 0, 128, 128, okay? So Y can be 0, that is for sure. But chroma planes cannot MAYBE, because that is out of range for all values. Just like all 0, x, y where x=y= !128 is out-of-range. Seriously? See this comment in ffmpeg. https://github.com/FFmpeg/FFmpeg/blob/7247a6fed8de9c2162ed408682e095f0b7f19350/libavutil/pixfmt.h#L595

Frankly speaking I do not understood this either. You can still write it, only in limited range these values reserved for sync (0, 255). Even in limited range you can still write it in avc or hevc. Just change full range flag with -bsf hevc_metadata.

[cretindesalpes]

These conversion functions generally don’t check the range of pixel values, this is not their job. They process the planes more or less independently. Overshoots caused by various operations should not be clipped until the final filterchain output, if possible. Also dithering may make a constant signal oscillate between two values, or even more.

[Balling]

XAVC should be interpreted as left type 0, but it's always nice to double check. Feel free to upload a sample. A landscape will do, although someone holding a colour palette would be better.

x264 defaults to left if it is ommited. While x265 does not have a default, you should always tag it in VUI in bitstream.

[FranceBB]

x264 defaults to left if it is ommited. While x265 does not have a default, you should always tag it in VUI in bitstream.

We were talking about an hardware camera encoder, so not x264, but it defaulted to left (same goes for x264 indeed).
That being said, this case is closed as we realized that there's no difference between left and top left for 4:2:2 and that the ffmpeg/ffprobe guys wrote two different values that actually meant the same thing.
This went into "public vote" with Ferenc and the others and we all agreed that within Avisynth we won't care about what the indexer (LWLibav, LSMASH, ffms2 etc) report for 4:2:2 in terms of chroma placement, and we'll always assume it to be left to play safe. This way, the issue I highlighted is gonna disappear.

Ferenc will probably release 3.7.2 Test 2 soon with the fix.

Case closed. ;)

[Balling]

BTW, this should be really fixed. https://trac.ffmpeg.org/ticket/9167#comment:40

[Balling]

YUV 8bit to RGB 16bit is 2 conversions: YUV->RGB and 8bit->16bit.

8bit->16bit integer is lossless adding 8 zeroes to 8bit data. YUV->RGB is not lossless math. So with 8bit YUV input ConvertToRGB64() can firstly convert to RGB in 8bit and secondly add 8 zeroes to RGB. With ConvertBits(16) before RGB it explicitly command first add 8 zeroes to YUV and convert 16bit YUV->RGB.

So it mostly probably shows as 8bit and 16bit YUV->RGB gives different results.

Yes - for better results it recommended to convert YUV->RGB in 16bits. It gives less errors (less banding on gradients). So if ConvertToRGB64() with 8bit YUV input makes first conversion to RGB in 8bit it may be an issue to fix. As current workaround - add ConvertBits(16) before ConvertToRGB().

YUV -> RGB is less a problem than vice versa, since YUV is so much bigger and most of that is out-of-gamut for RGB and so is usually not present, be it 100 bits RGB cannot be preserved unless you will color manage it to bigger primaries. But yeah, you first add zeroes to original YCbCr values and then convert to RGB, that is what is called internal precision.

See my [accepted] answer for this here: https://stackoverflow.com/a/66926260/11173412

[Balling]

BTW, a lot of stuff from my pull request is still not addressed https://github.com/AviSynth/AviSynthPlus/pull/215 like this. https://github.com/AviSynth/AviSynthPlus/blob/1cb31ca82a1503b525809fff1ecc0816222555d0/avs_core/filters/source.cpp#L1065

[pinterf]

Avisynth+ 3.7.2 test 2 (20220201) (https://drive.google.com/uc?export=download&id=1QpctWEKU1Mg9O5AEyK41iQ4fIuJAbktc)

[Dogway]

Thanks pinterf! Could you take a look at this (https://forum.doom9.org/showthread.php?p=1960946#post1960946)? Not sure if that's to be expected but I run a few tests and got consistent performance readings.
Here's another example, again 2% with prefetch(4), with prefetch(6) performance is worse and inconsistent.
expr("x[-1,-1] x[0,-1] x[1,-1] x[-1,0] x[0,0] x[1,0] x[-1,1] x[0,1] x[1,1] + + + + + + + + 0.111111111 *") # P(6) ~360 P(4) 389
#expr("x[1,1] x[0,1] x[-1,1] x[1,0] x[0,0] x[-1,0] x[1,-1] x[0,-1] x[-1,-1] + + + + + + + + 0.111111111 *") # P(6) ~360 P(4) 381

[FranceBB]

Avisynth+ 3.7.2 test 2 (20220201) (https://drive.google.com/uc?export=download&id=1QpctWEKU1Mg9O5AEyK41iQ4fIuJAbktc)

Thanks Ferenc!
This is with the chroma location fix for 4:2:2 right?

[StainlessS]

Thanks P,

FaBB, in blue [think thats what U R talkin' bout.] In the zip, "readme_history.txt"

Avisynth Plus change log
------------------------
Source: https://github.com/AviSynth/AviSynthPlus

This file contains all change log, with detailed examples and explanations.
The "rst" version of the documentation just lists changes in brief.

20220201 3.7.2-WIP
------------------
- Allow top_left (2) and bottom_left (4) chroma placements for 422 in colorspace conversions, they act as "left" (0, "mpeg2")
in order not to give error with video sources which have _ChromaLocation set to other than "mpeg2"
See https://trac.ffmpeg.org/ticket/9598#comment:5
- Fix: Expr LUT operation Access Violation on x86 + AVX2 due to an unaligned internal buffer (<32 bytes)
- Fix: Chroma full scale as ITU Rec H.273 (e.g +/-127.5 and not +/-127) in internal converters, ColorYUV and Histogram
- Fix #257: regression in 3.7.1: GreyScale to not convert to limited range when input is RGB. Regression in 3.7.1
Accepts only matrix names of limited range as it is put in the documentation.
- Fix #256: ColorYUV(analyse=true) to not set _ColorRange property to "full" if input has no such
property and range cannot be 100% sure established. In general: when no _ColorRange for input and
no parameter which would rely on a supposed default (such as full range for gamma), then an
output frame property is not added.
When no _ColorRange for input and no other parameters to hint color range then
- gamma<>0 sets full range
- opt="coring" sets limited range
- otherwise no _ColorRange for output would be set
- Overlay (#255): "blend": using accurate formula using float calculation. 8 bit basic case is slower now when opacity=1.0.
Higher bit depths and opacity<1.0 cases are quicker.
Mask processing suffered from inaccuracy. For speed reasons mask value 0 to 255 were handled
as mask/256 instead of mask/255. Since with such calculation maximum value was not the expected 1.0 but rather 255/256 (0.996)
this case was specially treated as 1.0 to give Overlay proper results at least the the two extremes.
But for example applying mask=129 to pixel=255 resulted in result_pixel=128 instead of 129. This was valid on higher bit depths as well.
- Fix: Attempt to resolve deadlock when an Eval'd (Prefetch inside) Clip result is
used in Invoke which calls a filter with GetFrame in its constructor.
(AvsPMod use case which Invokes frame prop read / ConvertToRGB32 after having the AVS script evaluated)
Remark: problem emerged in 3.7.1test22 which is trying to read frame properties of the 0th frame in its constructor.
A similar deadlock situation was already fixed earlier in Neo branch and had been backported but it did not cover this use case.

[FranceBB]

Thanks P,

FaBB, in blue [think thats what U R talkin' bout.]

Allow top_left (2) and bottom_left (4) chroma placements for 422 in colorspace conversions, they act as "left" (0, "mpeg2")
in order not to give error with video sources which have _ChromaLocation set to other than "mpeg2"

Nice! I was on my mobile, I couldn't check.
Testing straight away now that I'm on my computer! :D

EDIT: It works. Well done, Ferenc! :D

https://i.imgur.com/YG43AIR.png

https://i.imgur.com/kvjIY0b.png
https://i.imgur.com/fqTju1Y.png

[wonkey_monkey]

When I try to env->Invoke "import" on an .avs file that returns a clip with a depth other than 8-bit, it throws a ReturnExprException. I've got no idea what one of those is and can only find a couple of mentions of it in the AviSynth source code. I tried to include expression.h so ReturnExprException is defined but then that throws up other missing definitions...

I think I'm probably missing something here. Anyone got any guesses as to why an exception is being thrown - the file loads fine in VirtualDub2 - or how I'm supposed to catch the exception?

PS While I'm asking questions, is there a nice way to get the canonical colourspace name string for a clip?

[pinterf]

When I try to env->Invoke "import" on an .avs file that returns a clip with a depth other than 8-bit, it throws a ReturnExprException. I've got no idea what one of those is and can only find a couple of mentions of it in the AviSynth source code. I tried to include expression.h so ReturnExprException is defined but then that throws up other missing definitions...

I think I'm probably missing something here. Anyone got any guesses as to why an exception is being thrown - the file loads fine in VirtualDub2 - or how I'm supposed to catch the exception?

Check this part:
https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/core/main.cpp#L693

(I don't know though why you get ReturnExprException)


PS While I'm asking questions, is there a nice way to get the canonical colourspace name string for a clip?
function PixelType, parameter is a clip, returns a string.

[wonkey_monkey]

Thanks. I found my mistake - it seems that ReturnExprException is normal and handled properly during Invoke. My actual error was elsewhere, but I saw the ReturnExprExceptions in the debug output and assumed they were a problem.

[jpsdr]

If i run this kind of script :

Filter
Prefetch(2)


Filter is MT_NICE.
GetFrame can be called twice in parallel, but that's all ?
If in GetFrame i request "something" on the begining and release it at the end, i only need to have 2 of the "something" to never have a GetFrame waiting the "something" being avaible ?
Is there any reason that could result in the need of having 4 of the "something" instead of the 2 expected ?

[pinterf]

If i run this kind of script :

Filter
Prefetch(2)


Filter is MT_NICE.
GetFrame can be called twice in parallel, but that's all ?
If in GetFrame i request "something" on the begining and release it at the end, i only need to have 2 of the "something" to never have a GetFrame waiting the "something" being avaible ?
Is there any reason that could result in the need of having 4 of the "something" instead of the 2 expected ?
It can be called more than twice, the two occasion is only from Prefetch, but further GetFrames can occur at the same time, e.g. directly from the main consumer of the final clip. That's already 3. The the consumer can request directly another GetFrame through an additional Invoke (like AvsPMod does when obtaining frame properties on the evaluated clip with an additonal Invoke of ConvertToRgb - which in turn calls a GetFrame(0) in its constructor).
NICE filters must be fully reentrant and can theoretically be called arbitrary times in their specific invoke position.

[poisondeathray]

Thanks for fixing overlay mask accuracy issue
https://forum.doom9.org/showthread.php?t=183710


Some new issue with overlay opacity , 3.7.2 test 2 (r3620)

The value remains RGB 0,0,0 below opacity 0.5, and RGB 1,1,1 above opacity 0.5 . Opacity=0 and Opacity=1 gives correct result



b=blankclip(pixel_type="RGB24", colors=[0,0,0])
w=blankclip(pixel_type="RGB24", colors=[255,255,255])

overlay(b,w, opacity=0.5)



Reverting to 3.7.2 test 1 (r3600) fixes the overlay opacity parameter behaviour

I suspect it's this commit, but I don't know enough programming to narrow it down farther
https://github.com/AviSynth/AviSynthPlus/commit/ceae03c17b4095a7612dbbc7be864ca9ff0f872f

[pinterf]

Thanks for fixing overlay mask accuracy issue
https://forum.doom9.org/showthread.php?t=183710


Some new issue with overlay opacity , 3.7.2 test 2 (r3620)

The value remains RGB 0,0,0 below opacity 0.5, and RGB 1,1,1 above opacity 0.5 . Opacity=0 and Opacity=1 gives correct result



b=blankclip(pixel_type="RGB24", colors=[0,0,0])
w=blankclip(pixel_type="RGB24", colors=[255,255,255])

overlay(b,w, opacity=0.5)



Reverting to 3.7.2 test 1 (r3600) fixes the overlay opacity parameter behaviour

I suspect it's this commit, but I don't know enough programming to narrow it down farther
https://github.com/AviSynth/AviSynthPlus/commit/ceae03c17b4095a7612dbbc7be864ca9ff0f872f
Thanks for the report, fixed.

[pinterf]

Avisynth+ 3.7.2 test 3 (20220208) (https://drive.google.com/uc?export=download&id=1jyqttklu67ehTLnWlsHYHexMdbs2lfCq)
Fix Prefetch(1) + AvsPMod crash
Fix Overlay + blend + no mask (regression in 3.7.2 test 2)

[gispos]

Avisynth+ 3.7.2 test 3 (20220208) (https://drive.google.com/uc?export=download&id=1jyqttklu67ehTLnWlsHYHexMdbs2lfCq)
Fix Prefetch(1) + AvsPMod crash
Fix Overlay + blend + no mask (regression in 3.7.2 test 2)
Thank you!

Edit: Would it be easier (less problematic) if I waited 100 or 200 milliseconds between get_frame and get_properties?

[FranceBB]

Thanks Ferenc.
Every new release is always appreciated. :)

[StainlessS]

Yeah thanx P, Mucho Grassy Ass.

[tormento]

We f***ing need a Thank you button...

[pinterf]

Would it be easier (less problematic) if I waited 100 or 200 milliseconds between get_frame and get_properties?
No. It is expected to work without any timing magic. This was a hidden bug or issue on which nobody from earlier authors thought of and sooner or later there would be a similar use case by another plugin/application that would shed light on this problem.

[Raizo]

Yo Guys,

How to properly install the test version? Which files must be replaced? and where exactly?

https://i.postimg.cc/zv4SvmhV/avisynth-folder.png

Will replacing these two files be enough?

• x86-> C:\WINDOWS\SysWOW64\avisynth.dll

• x64-> C:\WINDOWS\SYSTEM32\avisynth.dll

And what about this one: "system\DevIL.dll" ?

Can I put the plugins here?
C:\Program Files (x86)\AviSynth+\plugins

[Emulgator]

Some .reg entries have to be set/reset with that.
Groucho2004's Universal Avisynth Installer will be your friend:
https://forum.doom9.org/showthread.php?t=172124
You can have multiple Avisynth versions ready to be swapped in with a batch script
and have 2 of them (x86+x64) installed side-by-side.
You swap in your desired latest test version where the Installer's 3.7.0 version sits
and maybe adapt the .bat file to reflect the name change.

[StainlessS]

Raizo,
If you want both x86 and x64 Avs+, suggest get this and set it up [reading docs]
https://forum.doom9.org/showthread.php?t=172124

Maybe copy to eg C:\VideoTools\AvisynthRepository\
Copy/replace the test version Avisynth x86 Avisynth.dll and system/devil.dll into the contained AVSPLUS370_x86,
and for x64 into AVSPLUS370_x64.
Edit the SetAvs cmd as per docs [if required, I dont bother],
and execute setavs.cmd as administrator, and select the Avs 3.7.0 one for each of x86 and x64.
The named versions will be wrong, but i guess you could change both folder names and edit setavs.cmd ,
but I personally dont bother.
Copy your plugins into each of the plugins directories in AVSPLUS370_x86/x64 folders.

Alernatively, get latest stable setup, and install, then copy from test version,
x86 avisnyth.dll and devil.dll into Windows\SysWOW64 and x64 version to Windows\system32.

sorted.

[Raizo]

Many Thanks Emulgator and StainlessS

yes, I use both version x86+x64 3.7.1 both already installed and working. I was just trying to figure out how to upgrade to the newest test version which does not come with the installer.

I'll try with this Groucho installer then. Tnx 1k again :)


EDIT:



Alernatively, get latest stable setup, and install, then copy from test version,
x86 avisnyth.dll and devil.dll into Windows\SysWOW64 and x64 version to Windows\system32.


just this seems working properly 👍🏻

Ty Guys!

[Emulgator]

Fix Prefetch(1) + AvsPMod crash
Yippie, thanks, Ferenc ! Köszönom szepen !
Was undoing all my Prefetch calls last week, now it works fine again.

[Emulgator]

And here is an updated .bat file for Groucho2004's Universal Avisynth Installer to accommodate AVS+372

https://forum.doom9.org/showthread.php?p=1963566#post1963566

[Dogway]

I've been wondering for some time. Is it possible with CombinePlanes to pick the chroma of the first clip and the luma of the second? Main reason, keep frameprops from the first clip, but also for other uses:
CombinePlanes(YUVa,YUVb,source_planes="UVY",planes="YUV") # pseudo-call

Also mix several clips into one, format is the same (bitdepth, size, subsampling):
CombinePlanes(YUVa,Ub,YUVc, source_planes="YUVUYUV",planes="YUV") # pseudo-call

By the way there's also this (https://forum.doom9.org/showthread.php?p=1960946#post1960946)old optimization issue on pixel fetching.

[real.finder]

I've been wondering for some time. Is it possible with CombinePlanes to pick the chroma of the first clip and the luma of the second? Main reason, keep frameprops from the first clip, but also for other uses:
CombinePlanes(YUVa,YUVb,source_planes="UVY",planes="YUV") # pseudo-call

like this https://github.com/realfinder/AVS-Stuff/blob/de638db2e833b71eb652d0949a3500eeb12f3d05/avs%202.5%20and%20up/F3KDB_s.avsi#L175 and https://github.com/realfinder/AVS-Stuff/blob/70e3f1dd630c59265aee23b83f108df5ff81fc56/avs%202.6%20and%20up/scaled444tonative.avsi#L51 ?

edit: don't know why but CombinePlanes(YUVa,YUVb,planes="UVY",source_planes="UVY") seems did what you want

[Dogway]

Replicating your call I get an error with:
CombinePlanes(YUVa,YUVa,YUVb,planes="UVY",sample_clip=YUVa)
"CombinePlanes: source and target plane dimensions are different"
YUVa and YUVb are YUV420 clips though.

[wonkey_monkey]

I'm having another of my wacky ideas, but I'm struggling to find the place in the AviSynth source where I'd need to put it. Whereabouts in the source would I find the code that actually opens an.avs file and gets ready to parse it? Is that always done via an invoke of "Import"?

[real.finder]

Replicating your call I get an error with:
CombinePlanes(YUVa,YUVa,YUVb,planes="UVY",sample_clip=YUVa)
"CombinePlanes: source and target plane dimensions are different"
YUVa and YUVb are YUV420 clips though.

which call? in the links? they need to be monochrome (Y/Y8) IIRC

anyway as I said try
CombinePlanes(YUVa,YUVb,planes="UVY",source_planes="UVY")

[Dogway]

Thanks real.finder, getting somewhere although very counterintuitive. But it was only picking U from YUVa, I need the next to pick chroma from YUVa and luma from YUVb.
CombinePlanes(YUVa,YUVa,YUVb,planes="UVY",source_planes="UVY")
I'm sure this is not an optimal way to merge planes. I don't know how ShufflePlanes work in VS but working in other programs you can mix everything (plane/channel independent) with everything like Shuffle in Nuke, Swizzle in Substance, etc

[StainlessS]

Wonkey,
clarify, you wanna hack the source ?
what are you tryin' to do. [your wacky ideas are often entertaining :) ]
Import is sorta load script into ram, do a SaveString() on it, and then an Eval() on the saved string.
.

[wonkey_monkey]

Well, it's borne from my frustration with AviSynth's multiline syntax, the requirement to put a \ on the end of every line (which is a pain to write and edit, and makes it look very cluttered) and the inability to comment out lines in a multiline block.

I thought about suggesting a change to the parser - it could just ignore newlines within function call brackets (except those in strings).

But then I thought, no, maybe it's too risky to make those kind of changes, even if they seem like they should be harmless.

So then I thought, what if AviSynth offered a way for a regular plugin, which takes a string as input and gives a string as output, to pre-parse a script? It could remove newlines and deal with comments, or do other things that the AviSynth grammar can't do. That way the AviSynth parser is blameless if someone wants to use such a plugin and runs into a problem.

So what I thought was this: if a script begins with a shebang and a filter name (e.g. "#!filter_name"), AviSynth will strip the shebang line and pass the script to the named filter as a string to be transformed. It will then parse what it receives back as normal.

So for example, you could give it this script:

#!monkify_my_script

AviSource("test_file.avi")

{FlipVertical|FlipHorizontal}

ConvertTorRGB32(
matrix = "rec709",
# interlaced = true,
chromainplacement = "MPEG2",
chromaresample = "sinc",
# chromaoutplacement = "DV"
)

and it would, before doing anything else, send the script (minus the first line) to the function monkify_my_script, which is just a regular AviSynth plugin filter. The filter would do its thing, and in this case it might return:

AviSource("test_file.avi")

FlipVertical # the filter chose this at random from the strings inside {...|...}

ConvertTorRGB32(matrix = "rec709", chromainplacement = "MPEG2", chromaresample = "sinc") # commented parameters, newlines, and trailing comma removed (and ths comment added, how meta is that?)

You could specify multiple shebang lines and AviSynth will only start parsing once the script has passed through all of the filters specified.

[StainlessS]

I see.
[EDIT: Due to avisynth syntax] There is some kind of problem with multiple expressions not being able to tell if whole lot is a single expression or
is multiple, where you need a newline to explicitly prevent wrong interpretation, however I cannot remember an expression where this occurs.
(It has come up before, but my brain aint feeling very chippa at the moment, I just wanna go to sleep).
So, there may be a dragon lurking in your idea, ready and waiting to gobble you up.

[wonkey_monkey]

a) That case doesn't apply to function parameters, which are strictly separated by commas and can't contain multiple statements anyway.

b) In any case, that's why I propose the filtering method rather than a change to the parser (even though the limited case wouldn't be very harmful), because then it's the user's choice to pass the script through the filter.

[StainlessS]

I meant dragons in the "monkify_my_script()" filter whotsit.

[wonkey_monkey]

Oh, quite possibly, but they would be my dragons to slay.

[real.finder]

Thanks real.finder, getting somewhere although very counterintuitive. But it was only picking U from YUVa, I need the next to pick chroma from YUVa and luma from YUVb.
CombinePlanes(YUVa,YUVa,YUVb,planes="UVY",source_planes="UVY")
I'm sure this is not an optimal way to merge planes. I don't know how ShufflePlanes work in VS but working in other programs you can mix everything (plane/channel independent) with everything like Shuffle in Nuke, Swizzle in Substance, etc

in http://avisynth.nl/index.php/CombinePlanes#Examples you can copy luma from first clip, U and V from the second

but you need the opposite, so maybe CombinePlanes need an update to accept arrays so it can be used as

CombinePlanes(YUVa,YUVb,planes="UVY",source_planes=[1, 2, 3]) # plane 0 is the 1st Y so we start from 1, plane 3 is the 2nd Y

or maybe update the present string cases so we can set it as source_planes="U1V1Y2"

or better both :) it's up to pinterf

[Dogway]

It's just an example. I might also want to merge UV from YUV and Luma from Y (depending on the prerequisites of the function needs), I know there's ExtractU() and ExtractV() but isn't that defeating the purpose of a combine planes function? Doing everything in one place might optimize some of these tasks.

[Mitra]

I compiled and installed AVS+Cuda and CUDAFilters without any problems, but when run this script:
FFMS2("s.ts")
onCUDA()
KTGMC()

gives this error:
This Avisynth does not support memory type 2 (CUDA)
(m.avs, line 2)
m.avs: Unknown error occurred

and with this:
FFMS2("s.ts")
KTGMC()
onCUDA()

gives this error:
Evaluate: Unhandled C++ exception!
(C:/Program Files (x86)/AviSynth+/plugins64+/KTGMC.avsi, line 451)
(m.avs, line 2)
m.avs: Unknown error occurred

and these are line 449~451 of KTGMC.avs :
449- bobbed = (InputType == 0) ? useEdiExt ? isyuy2(EdiExt) ? EdiExt.nonyuy2clipin(true) : EdiExt : planarClip.KTGMC_Bob( 0,0.5 ) : \
450- (InputType == 1) ? planarClip : \
451- planarClip.Blur( 0,1 )


i don't know how to solve the errors. :(

my machine : i7 10700k, GTX 1660, Nvidia driver 472.84 standard , Win 10

Please help me.

[Boulder]

There seems to be a problem with setting frame properties to a different clip than 'last'.

DGSource("whatever.dgi")
dbl=SmoothD2(quant=5, zw=1, ncpu=1).SmoothD2c(quant=5, zw=1, ncpu=1)
PropSet(dbl, "_Matrix", 1)
PropSet(dbl, "_Transfer", 1)
PropSet(dbl, "_Primaries", 1)
PropSet(dbl, "_ChromaLocation", 0)
PropSet(dbl, "_ColorRange", 1)
PropSet(dbl, "_FieldBased", 0)
propshow(last)

This script shows only the _FieldBased property even though the original clip has 11 keys.


EDIT: In fact, there's an issue without that external clip. If the 'last' clip has existing properties and you use PropSet to set some property, it deletes all the others. The possible subsequent PropSet calls don't do that, they just add keys.

[StainlessS]

Boulder,
I know nowt bout that there propset stuff, but check your intent that you show Propshow(last) and not Propshow(dbl) [maybe should be this]

[I dont see how setting properties of a filtered clip, can alter those of the source to the filtered clip, nor why that might be desirable]

EDIT: Indeed, if dbl plays no part in the output, does the propset stuff even do anything. [I've no idea ???]

EDIT: To below, OK B, was just checking that posted was as intended.

[Boulder]

That's just to replicate the issue. As SmoothD2 does not support passing the props through but deletes all, I have to put them back in to use the clip then in a different function. I was just debugging the problem and found out that something strange happens to the original clip props, which should be untouched in this case. Then later I notices that if I only change one property for the original clip, the rest get deleted.

[Boulder]

There might be other issues there regarding prop handling as well. I've tried testing Dogway's SmoothD2c function which uses frame props, and when refreshing the output in the script editor in VDub2 with F5, I often get errors complaining that the color range property cannot be found. It can be fixed only by actually reloading the script, and even then it sometimes fails. I'm not using Prefetch at this point to make sure it won't interfere.

[wonkey_monkey]

What's the proper way to get frame properties in C++? Information seems to be a bit scant, with most search results pointing back to this thread which has a few examples but some seem to be out of date. There's PVideoFrame's getProperties and getConstProperties, but I can't seem to assign them to local variables. Then there's env's getFramePropsRW and getFramePropsRO - I only want to read, but can't seem to use RO because I can't set it to a local variable (which will change on each frame) without a cast because the return value is a const...

Also... how long does an AVSMap(*) remain valid for? Should it be assumed to be invalid as soon as the PVideoFrame is destroyed? Is there a safe way to copy it elsewhere and store it more permanently, separate from its PVideoFrame?

[StvG]

There seems to be a problem with setting frame properties to a different clip than 'last'.

DGSource("whatever.dgi")
dbl=SmoothD2(quant=5, zw=1, ncpu=1).SmoothD2c(quant=5, zw=1, ncpu=1)
PropSet(dbl, "_Matrix", 1)
PropSet(dbl, "_Transfer", 1)
PropSet(dbl, "_Primaries", 1)
PropSet(dbl, "_ChromaLocation", 0)
PropSet(dbl, "_ColorRange", 1)
PropSet(dbl, "_FieldBased", 0)
propshow(last)

This script shows only the _FieldBased property even though the original clip has 11 keys.


EDIT: In fact, there's an issue without that external clip. If the 'last' clip has existing properties and you use PropSet to set some property, it deletes all the others. The possible subsequent PropSet calls don't do that, they just add keys.

You probably want DGSource("whatever.dgi")
dbl=SmoothD2(quant=5, zw=1, ncpu=1).SmoothD2c(quant=5, zw=1, ncpu=1)
PropSet(dbl, "_Matrix", 1) # this one becomes last
PropSet( "_Transfer", 1)
PropSet( "_Primaries", 1)
PropSet( "_ChromaLocation", 0)
PropSet( "_ColorRange", 1)
PropSet( "_FieldBased", 0)
propshow(last)

What's the proper way to get frame properties in C++? Information seems to be a bit scant, with most search results pointing back to this thread which has a few examples but some seem to be out of date. There's PVideoFrame's getProperties and getConstProperties, but I can't seem to assign them to local variables. Then there's env's getFramePropsRW and getFramePropsRO - I only want to read, but can't seem to use RO because I can't set it to a local variable (which will change on each frame) without a cast because the return value is a const...

Also... how long does an AVSMap(*) remain valid for? Should it be assumed to be invalid as soon as the PVideoFrame is destroyed? Is there a safe way to copy it elsewhere and store it more permanently, separate from its PVideoFrame?

Check here (https://github.com/AviSynth/AviSynthPlus/blob/master/distrib/Readme/readme_history.txt#L1217). Search for frameprop in this doc because there are some new additions.

[StainlessS]

PropSet(dbl, "_Matrix", 1) # this one becomes last
shit, why did I not see that !
Nice one StvG,
you are the biz :)

[Boulder]

Nice catch! I considered propSet like a variable setting, which would not affect the selection of active clip :)

[Boulder]

There might be other issues there regarding prop handling as well. I've tried testing Dogway's SmoothD2c function which uses frame props, and when refreshing the output in the script editor in VDub2 with F5, I often get errors complaining that the color range property cannot be found. It can be fixed only by actually reloading the script, and even then it sometimes fails. I'm not using Prefetch at this point to make sure it won't interfere.

This one still remains.

DGSource("whatever.dgi")
dbl=SmoothD2(quant=5, zw=1, ncpu=1).SmoothD2c(quant=5, zw=1, ncpu=1)
dbl=propCopy(dbl, last)
return last

Refreshing a couple of times will bring up the error that _ColorRange is not set. It definitely should be since the source filter attaches them to the clip.

At least these are needed to test:
https://github.com/Dogway/Avisynth-Scripts/blob/master/ExTools.avsi
https://github.com/Dogway/Avisynth-Scripts/blob/master/EX%20mods/DeblockPack.avsi
https://www.dropbox.com/s/ui8chlbzopuqs5a/SmoothD2-a3_x64.zip?dl=1

[StvG]

There might be other issues there regarding prop handling as well. I've tried testing Dogway's SmoothD2c function which uses frame props, and when refreshing the output in the script editor in VDub2 with F5, I often get errors complaining that the color range property cannot be found. It can be fixed only by actually reloading the script, and even then it sometimes fails. I'm not using Prefetch at this point to make sure it won't interfere.

This one still remains.

DGSource("whatever.dgi")
dbl=SmoothD2(quant=5, zw=1, ncpu=1).SmoothD2c(quant=5, zw=1, ncpu=1)
dbl=propCopy(dbl, last)
return last

Refreshing a couple of times will bring up the error that _ColorRange is not set. It definitely should be since the source filter attaches them to the clip.

At least these are needed to test:
https://github.com/Dogway/Avisynth-Scripts/blob/master/ExTools.avsi
https://github.com/Dogway/Avisynth-Scripts/blob/master/EX%20mods/DeblockPack.avsi
https://www.dropbox.com/s/ui8chlbzopuqs5a/SmoothD2-a3_x64.zip?dl=1

ResizersPack.avsi is also needed.

I don't have issues with this script (only replaced DGSource with ffms2) when refreshing in AvsPmod.

Edit: The source didn't have _ColorRange prop. I can reproduce the issue after I set this prop.

[Shinkiro]

Video encoding very often hangs in a random place, it just stops and that's it. it looks like this https://disk.yandex.com/i/ab0R1-txImXnVw and when you restart the encoding, it may end successfully or hang in another place.
Now I have Avisynth 3.7.2_test3 codec x265-3.5-Mod-by-Patman-x64-gcc11.2.0
But I had it happen on older versions of Avisynth and x265

x265 "script - 01.avs" -o "video [BDRip 1080p].hevc" ^
--profile main10 --level-idc 4.1 --output-depth 10 ^
--crf 16.0 --bframes 16 --bframe-bias 0 --ref 4 --b-pyramid ^
--aq-mode 3 --aq-strength 0.85 --qcomp 0.70 --pbratio 1.20 --qp-adaptation-range 2 --qg-size 16 --qpmin 0 --qpmax 51 --qpstep 4 ^
--limit-modes --limit-refs 3 --open-gop --cbqpoffs -2 --crqpoffs -2 ^
--ctu 32 --max-tu-size 16 --tu-intra-depth 2 --tu-inter-depth 2 --limit-tu 4 ^
--vbv-maxrate 10000 --vbv-bufsize 10000 --vbv-init 1.0 ^
--rd 4 --dynamic-rd 2 --rdoq-level 2 --psy-rd 2.00 --psy-rdoq 5.00 ^
--b-adapt 2 ^
--min-keyint 24 --keyint 240 --subme 5 --rc-lookahead 80 --merange 44 --max-merge 4 --me star --wpp ^
--no-sao --no-sao-non-deblock --no-rect --no-amp --no-tskip --rskip 0 ^
--no-strong-intra-smoothing ^
--no-early-skip ^
--colorprim 1 --transfer 1 --colormatrix 1 --sar 1:1 --frame-threads 14 ^
--deblock 1:-1


setmemorymax(6144)
tr=2
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
DGSource("01-02.dgi")
Trim(42630,0)

F=last
A=last.TAAmbk(aatype=-1, preaa=1, postaa=false, sharp=0, mtype=0, cycle=2, dark=0.0)
mthr=34
mask=flatmask(2, scale=5, lo=4, MSR=40, invert=false).mt_lut("x "+string(mthr)+" <= x 1 >> x 1 << ?", U=1, V=1).RemoveGrain((980>960) ? 20 : 11, -1)
AA=TAAmbk(aatype=-1, preaa=1, postaa=false, sharp=0, mtype=5, cycle=1, dark=0.0)
ed=ex_merge(F, AA,mask, luma=true, Y=3, UV=3)

FilterMapB="[37889 39964]"
FilterLineB=ed
ReplaceFramesSimple(FilterLineB, mappings=FilterMapB)

F=last
mthr=44
mask1b=ex_edge("kirsch",6,10).ConvertToY()
mask2b=ex_edge("kirsch",4,7).ConvertToY()
mask=last.ConditionalFilter(mask1b, mask2b, "AverageLuma()",">","50").mt_lut("x "+string(mthr)+" <= x 1 >> x 1 << ?", U=1, V=1).RemoveGrain((980>960) ? 20 : 11, -1)

deg1 = last.SMDegrain(tr=2,thSAD=121, thSADC=50, thSCD1=156,thSCD2=96, contrasharp=false, refinemotion=true, chroma=true, plane=4)
deg2 = last.SMDegrain(tr=3,thSAD=321, thSADC=150, thSCD1=256,thSCD2=96, contrasharp=false, refinemotion=true, chroma=true, plane=4)
deg=last.ConditionalFilter(deg2, deg1, "AverageLuma()",">","50")
ex_merge(deg ,F ,mask, luma=true, Y=3, UV=3)

ConvertBits(bits=10)
neo_f3kdb(sample_mode=2, Y=68, Cb=68, Cr=68, grainy=54, grainC=40, range=15, dynamic_grain=true)
Prefetch(tr)

But this happens on different settings and scripts
I have found repeated complaints on the Internet about the same or very similar situations, but I have not found a single solution to this problem or at least a hint of which element is responsible for this problem. The codec, avisyth or some plugin is to blame for this, or perhaps something in the system?

[FranceBB]

But this happens on different settings and scripts
I have found repeated complaints on the Internet about the same or very similar situations, but I have not found a single solution to this problem or at least a hint of which element is responsible for this problem. The codec, avisyth or some plugin is to blame for this, or perhaps something in the system?

My bet is on avstp.dll
Delete it from your plugins folder and try again.

https://github.com/pinterf/mvtools/issues/46

[Shinkiro]

My bet is on avstp.dll
Delete it from your plugins folder and try again.

https://github.com/pinterf/mvtools/issues/46
Thank you! I'll try without it.

[wonkey_monkey]

I compiled AviSynth+ from source using Visual Studio 2019 which all went swimmingly well. But with my AviSynth.dll copied over the previously installed one, .avs files have lost their filetype icon.

How to fix?

[gispos]

Hello Ferenc,
if I initialize a clip in AvsPmod and display e.g. frame number 100, frame 0 is called 3 times and only then frame 100.

ScriptClip("""
WriteFile("E:\Temp\get_frame_test.txt", string(current_frame), append=true, flush=True)
return last
""")

result
0
0
0
100

How can this be? I have gone through all the relevant AvsPmod code and cannot find a call to frame 0 anywhere.

Does avisynth itself call frame number 0 on various queries such as get_video_info or get_pixel_info or get_color_type or the like?
Matrix and properties reading I have deactivated.

[gispos]

I compiled AviSynth+ from source using Visual Studio 2019 which all went swimmingly well. But with my AviSynth.dll copied over the previously installed one, .avs files have lost their filetype icon.

How to fix?
Nirsoft has very useful freeware.
With FileTypesMan you can also set the file icons. In FileTypesMan left file menu A double click on the avs file opens a context menu with the options. You can also create new explorer entries with which program you want to open the avs.

https://www.nirsoft.net/utils/file_types_manager.html

https://i.postimg.cc/rRVBmNYP/Spnap-Shot.jpg (https://postimg.cc/rRVBmNYP)

[wonkey_monkey]

Sorry, I meant how to fix as in how to add the icon to the DLL, so if anyone wanted to try my DLL they wouldn't lose their icons.

With regard to your frame 0 problem, I had a look through the source code but couldn't find many places where AviSynth gets frame 0, but ConvertToRGB is one of them. Are you invoking that for display?

[gispos]

Sorry, I meant how to fix as in how to add the icon to the DLL, so if anyone wanted to try my DLL they wouldn't lose their icons.

Ok, but it is still worth taking a look at the tools offered there.

...but ConvertToRGB is one of them. Are you invoking that for display?
Yes.

real.finder makes trouble :)
https://forum.doom9.org/showthread.php?p=1964292#post1964292

I compared an older avisynth version with the latest version. There is different behavior when initializing a clip or using the preview filter.

Initialize clip or turn preview filter on or preview filter change a parameter in Levels() on frame 240
Avisynth 2772 MT

240
240
240

AvsPmod makes 1 call on frame 240 and avisynth makes 2 more calls on frame 240

Avisynth 3.72 test 3

0
0
0
240

I see AvsPmod makes 1 call on frame 240 but avisynth makes 3 more calls on frame 0
But I don't know what triggers these 3 additional calls to frame 0.
get_video_info, eval a clip, converttorgb32, set or read global variable ?

I think when avisynth makes internal calls to a frame, that should not be forwarded. WriteFile should not know about it.

With the old avisynth the 3 calls on frame 240 are much easier to clean up than with the 3 calls on frame 0

global last_frame = -1
/**avsp_filter
Levels(0, 1.00, 172, 0, 255, coring=true)
ScriptClip("""
if (last_frame != current_frame){
WriteFile("E:\Temp\get_frame_test.txt", string(current_frame), append=true, flush=True)
}
global last_frame = current_frame
return last
""")
**/

[Nuihc88]

0
0
0
240
I suspect that i have been occasionally running into this same behavior while using newer AviSynth+ test builds for realtime playback with CrendKing's DirectShow AviSynth Filter.
Older frames sometimes flicker on screen while seeking as the Script-environment is reinitializing. EDIT: I used to see same sort of behavior a lot years ago with SEt's AviSynth MT & ffdshow combo, except it was way worse then.
Possibly triggered in CrendKing's filter by this prefetching hang workaround (https://github.com/CrendKing/avisynth_filter/commit/07be726624633a85b2e58c79eb6d6d2c94810ae7), but i'm not sure as the issue only surfaced some time after i started using 3.7.2 test builds.

EDIT: This is my guess as to what is going on here:
1. during initialization current_frame = 0 in AviSynth+ cache?
2. prefetchers don't get cleared when closing and opening script clips?
3. current_frame doesn't get separate instances (unique identifiers) while calling for frame or clip info?
4. current_frame doesn't get cleared from cache when closing and opening Script-environments?

[gispos]

real.finder makes trouble :)
https://forum.doom9.org/showthread.php?p=1964292#post1964292

I compared an older avisynth version with the latest version. There is different behavior when initializing a clip or using the preview filter.

Initialize clip or turn preview filter on or preview filter change a parameter in Levels() on frame 240
Avisynth 2772 MT

240
240
240

AvsPmod makes 1 call on frame 240 and avisynth makes 2 more calls on frame 240

Avisynth 3.72 test 3

0
0
0
240


To be honest, I don't care, I don't need this, should those who need it continue to take care of it...
I only showed the incorrect behavior because I was forced to do so.

[StvG]

Hello Ferenc,
if I initialize a clip in AvsPmod and display e.g. frame number 100, frame 0 is called 3 times and only then frame 100.

ScriptClip("""
WriteFile("E:\Temp\get_frame_test.txt", string(current_frame), append=true, flush=True)
return last
""")

result
0
0
0
100

How can this be? I have gone through all the relevant AvsPmod code and cannot find a call to frame 0 anywhere.

Does avisynth itself call frame number 0 on various queries such as get_video_info or get_pixel_info or get_color_type or the like?
Matrix and properties reading I have deactivated.

Btw there is similar issue - https://github.com/AviSynth/AviSynthPlus/issues/210

[Kogarou]

One big advantage Vapoursynth has had for the last few years is nnedi3cl, which is one of the few old plugins re-written for vs without a port-back, and NNEDI3 is one of the backbones of filtering.
Is there anyone capable of porting nnedi3cl back to AVS+? I have tried but I'm not skilled enough with either SDKs to do it :(

[kedautinh12]

One big advantage Vapoursynth has had for the last few years is nnedi3cl, which is one of the few old plugins re-written for vs without a port-back, and NNEDI3 is one of the backbones of filtering.
Is there anyone capable of porting nnedi3cl back to AVS+? I have tried but I'm not skilled enough with either SDKs to do it :(

You can ask him a question with create new issue :D
https://github.com/Asd-g/AviSynthPlus-Scripts/issues

[qyot27]

Just musing, but for filters like nnedi and its offspring, I wonder how much of a difference it would make in performance being run on something with dedicated AI cores - the M1, for example. Granted, that would be an even heavier porting effort since you'd probably have to completely redesign the dispatcher to make sure it's using the Neural Engine rather than the CPU or the OpenCL features of the GPU, but still.

[wonkey_monkey]

Purely out of curiousity: why does ColorBarsHD generate a 1288x720 clip instead of 1280x720?

[FranceBB]

Purely out of curiousity: why does ColorBarsHD generate a 1288x720 clip instead of 1280x720?

Uhm if it was 1920x1088 I would have said that it was because of the inactive lines that are generally there in terrestrial signals etc anyway, but since it 1288x720 I have no idea.

[LigH]

May there be a mistake in the "pitch" interpretation?

[pinterf]

Hello Ferenc,
if I initialize a clip in AvsPmod and display e.g. frame number 100, frame 0 is called 3 times and only then frame 100.

ScriptClip("""
WriteFile("E:\Temp\get_frame_test.txt", string(current_frame), append=true, flush=True)
return last
""")

result
0
0
0
100

How can this be? I have gone through all the relevant AvsPmod code and cannot find a call to frame 0 anywhere.

Does avisynth itself call frame number 0 on various queries such as get_video_info or get_pixel_info or get_color_type or the like?
Matrix and properties reading I have deactivated.
When clip properties are needed then yes, there is a GetFrame(0) in such filters' constructor, e.g. in resizers, 444<->420 or <->rgb converters.

[FranceBB]

Is it gonna be the same if I use propclearall() after indexing? Is it still gonna call frame 0, find out that it holds nothing and move on or is it gonna behave like the old Avisynth and just call frame X?

[pinterf]

Purely out of curiousity: why does ColorBarsHD generate a 1288x720 clip instead of 1280x720?
I guess because it is divisible by 7 and allows of drawing perfectly spaced bar sections.

[Dogway]

I think green bar luma value is wrong in the wiki of ColorBarsHD. It should read 133 (currently 134), I had to double check with the original paper. Its full range value also has to be fixed.

I also had the chance to benchmark Expr() lut. I get slower performance (~7%) when using it instead of real time calculations.
ConvertBits(14)
expr(last,last,"f32 x y atan2 0.5 * x +", lut=2, scale_inputs="int")
expr(last,last,"f32 x y atan2 0.5 * x +", lut=2, scale_inputs="int")
expr(last,last,"f32 x y atan2 0.5 * x +", lut=2, scale_inputs="int")
Prefetch(6)

Unrelated, but as I understand it scale_inputs has no effect with LUT mode right? It's always dependent on input bitdepth (and input variables).

[pinterf]

Thanks pinterf! Could you take a look at this (https://forum.doom9.org/showthread.php?p=1960946#post1960946)? Not sure if that's to be expected but I run a few tests and got consistent performance readings.
Here's another example, again 2% with prefetch(4), with prefetch(6) performance is worse and inconsistent.
expr("x[-1,-1] x[0,-1] x[1,-1] x[-1,0] x[0,0] x[1,0] x[-1,1] x[0,1] x[1,1] + + + + + + + + 0.111111111 *") # P(6) ~360 P(4) 389
#expr("x[1,1] x[0,1] x[-1,1] x[1,0] x[0,0] x[-1,0] x[1,-1] x[0,-1] x[-1,-1] + + + + + + + + 0.111111111 *") # P(6) ~360 P(4) 381
I think it depends on how processor's memory data line prefetch works, L1/L2/L3 cache memory utilization and timing issues affect the performance seriously. The more thread you are using the larger likelihood that a processed part will be kicked out from the cache, thus the bottleneck of parallel processing will be not the processor itself but the memory speed.

Forum member DTL is using an interesting Intel tool with which one can analyze such metrics, cache hits, misses, unaligned access, etc... Art of fine tuning.

https://www.intel.com/content/www/us/en/develop/documentation/vtune-help/top/analyze-performance/microarchitecture-analysis-group/memory-access-analysis/memory-usage-view.html

But I'm sure that what you'd find optimal for one processor arhitecture, an older or a much newer one will not necessarily benefit from that.

[pinterf]

I think green bar luma value is wrong in the wiki of ColorBarsHD. It should read 133 (currently 134), I had to double check with the original paper. Its full range value also has to be fixed.

I also had the chance to benchmark Expr() lut. I get slower performance (~7%) when using it instead of real time calculations.
ConvertBits(14)
expr(last,last,"f32 x y atan2 0.5 * x +", lut=2, scale_inputs="int")
expr(last,last,"f32 x y atan2 0.5 * x +", lut=2, scale_inputs="int")
expr(last,last,"f32 x y atan2 0.5 * x +", lut=2, scale_inputs="int")
Prefetch(6)

Unrelated, but as I understand it scale_inputs has no effect with LUT mode right? It's always dependent on input bitdepth (and input variables).

A 14 bit lutxy needs a 0.5 GBytes LUT table. Probably this is too much to keep all of it in a fast cache memory.
When you decrease the bit depth to 12 and 10 bits, it would act quicker.
But LUT it is still going pixel-by-pixel, while Expr can translate 32 pixels at a time. We wish that the optimum - which to use, lut or Expr - would be guessed before use.

[pinterf]

Unrelated, but as I understand it scale_inputs has no effect with LUT mode right? It's always dependent on input bitdepth (and input variables).
To tell the truth I don't know, there are so many conversion options in Expr that I always have to look into the topic deeper, again and again.

[Dogway]

A 14 bit lutxy needs a 0.5 GBytes LUT table. Probably this is too much to keep all of it in a fast cache memory.
When you decrease the bit depth to 12 and 10 bits, it would act quicker.
But LUT it is still going pixel-by-pixel, while Expr can translate 32 pixels at a time. We wish that the optimum - which to use, lut or Expr - would be guessed before use.

That's right, 12-bit is faster at least for my CPU (8Mb L3 cache). I didn't know it depended on cache rather than RAM. Newer CPUs have bigger caches but in any case I think it's safer to limit lutxy to 12-bit for LUT.

To tell the truth I don't know, there are so many conversion options in Expr that I always have to look into the topic deeper, again and again.

I did some benchmarks and reached to that conclusion, but I will rerun them to be sure. These kind of things show up with big filters like QTGMC.

[Reel.Deel]

I think green bar luma value is wrong in the wiki of ColorBarsHD. It should read 133 (currently 134), I had to double check with the original paper. Its full range value also has to be fixed.


What should full range be? I'll go ahead and update the wiki.

I guess because it is divisible by 7 and allows of drawing perfectly spaced bar sections.

I think you're correct, from the original script (http://avisynth.nl/index.php/HDColorBars) that ColorBarsHD is based on there is this comment:

#Bugs/Limitations:
#Width of bars may not be totally accurate, especially e.g. section1 where width divisor of 7 doesn't add up.

[Dogway]

What should full range be? I'll go ahead and update the wiki.

Full range gives me 136 .0 with:
ConvertBits(8,fulls=false,fulld=true)

[Reel.Deel]

Full range gives me 136 .0 with:
ConvertBits(8,fulls=false,fulld=true)

Thanks. I also noticed that the Levels wiki page (http://avisynth.nl/index.php/Levels) has some questionable numbers listed for full range:

https://i.ibb.co/HnqpxKN/avslevels.png

Unless this is specific to levels, only 8bit 255 is correct and I'm not sure about 255/256 or any of the other 32-bit numbers.

[Dogway]

I always use n/255 for 32-bit float. You can check my list in ex_dlut() (https://github.com/Dogway/Avisynth-Scripts/blob/78444f2e558781ee63866757a4c4a59d7704b4f1/ExTools.avsi#L7077).
As you see it depends on clip range. This chart assumes legal range for the values so I guess for Levels() this would have something to do with 'coring'.
In my opinion you should set coring to false and treat the clip as full range.
The last column seems correct, it's the maximum permitted video signal value for either limited or full range.

[qyot27]

Side announcement/notifications:
FFmpeg has merged the patches which allow it to read certain selected frame properties.
x264 has now switched to AviSynth+ when on non-Windows platforms.

x264 had been using AvxSynth on Linux and macOS for a little over nine years (2013-Feb-12). Unlike FFmpeg, x264 ships a local copy of the avisynth_c.h header, so it's much more likely for the average distro's x264 package to just start working with a user's self-built AviSynth+ install, once they catch up to where x264-git is (and assuming they aren't disabling it because users don't understand how the input module selector works and complained about it failing even though it was working as expected).

The FFmpeg patches mean that the relevant frame properties (_FieldBased, _Matrix, _ChromaLocation, _Primaries, _Transfer, and _ColorRange) can now be passed through to the output, if the output format supports setting those properties for itself from what FFmpeg reports (HEVC encoding through libx265 does), and libavformat-based decoding has the ability to see those properties and programs using that information can then take appropriate action. mpv, for instance, can now use its autotonemapping for HDR content served through an AviSynth script. Requires a new enough build of FFMS2 or LSMASHSource to populate those properties when opening the source video, as well as AviSynth+ 3.7.1 for libavformat to try reading the frameprops.

[qyot27]

What I mean by the ffmpeg stuff:
The avs script:
FFmpegSource2("LG New York HDR UHD 4K Demo.ts",atrack=-1)
Spline64Resize(1280,720)

Prior to the patches (and still true if you try using 3.7.0 or below with FFmpeg):
$ ffmpeg -i test_hdr.avs
ffmpeg version N-105655-ge4ad38d0f7-6ubuntu5 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 11 (Ubuntu 11.2.0-7ubuntu2)
configuration: --prefix=/usr --extra-version=6ubuntu5 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --enable-debug --disable-stripping --enable-version3 --enable-mbedtls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libplacebo --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-avisynth --enable-vapoursynth --cpu=skylake --extra-cflags='-march=skylake' --extra-ldflags=-pthread --pkg-config-flags=--static
libavutil 57. 21.100 / 57. 21.100
libavcodec 59. 21.100 / 59. 21.100
libavformat 59. 17.102 / 59. 17.102
libavdevice 59. 5.100 / 59. 5.100
libavfilter 8. 27.100 / 8. 27.100
libswscale 6. 5.100 / 6. 5.100
libswresample 4. 4.100 / 4. 4.100
libpostproc 56. 4.100 / 56. 4.100
Guessed Channel Layout for Input Stream #0.1 : stereo
Input #0, avisynth, from 'test_hdr.avs':
Duration: 00:01:12.24, start: 0.000000, bitrate: 0 kb/s
Stream #0:0: Video: rawvideo (Y3[11][10] / 0xA0B3359), yuv420p10le(bottom first), 1280x720, 25 fps, 25 tbr, 25 tbn
Stream #0:1: Audio: pcm_f32le, 48000 Hz, stereo, flt, 3072 kb/s
At least one output file must be specified


$ ffmpeg -i test_hdr.avs
ffmpeg version N-105655-ge4ad38d0f7-6ubuntu5 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 11 (Ubuntu 11.2.0-7ubuntu2)
configuration: --prefix=/usr --extra-version=6ubuntu5 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --enable-debug --disable-stripping --enable-version3 --enable-mbedtls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libplacebo --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-avisynth --enable-vapoursynth --cpu=skylake --extra-cflags='-march=skylake' --extra-ldflags=-pthread --pkg-config-flags=--static
libavutil 57. 21.100 / 57. 21.100
libavcodec 59. 21.100 / 59. 21.100
libavformat 59. 17.102 / 59. 17.102
libavdevice 59. 5.100 / 59. 5.100
libavfilter 8. 27.100 / 8. 27.100
libswscale 6. 5.100 / 6. 5.100
libswresample 4. 4.100 / 4. 4.100
libpostproc 56. 4.100 / 56. 4.100
Guessed Channel Layout for Input Stream #0.1 : stereo
Input #0, avisynth, from 'test_hdr.avs':
Duration: 00:01:12.24, start: 0.000000, bitrate: 0 kb/s
Stream #0:0: Video: rawvideo (Y3[11][10] / 0xA0B3359), yuv420p10le(tv, bt2020nc/bt2020/smpte2084, progressive), 1280x720, 25 fps, 25 tbr, 25 tbn
Stream #0:1: Audio: pcm_f32le, 48000 Hz, stereo, flt, 3072 kb/s
At least one output file must be specified


The key lines there are,
before:
Stream #0:0: Video: rawvideo (Y3[11][10] / 0xA0B3359), yuv420p10le(bottom first), 1280x720, 25 fps, 25 tbr, 25 tbn
and after:
Stream #0:0: Video: rawvideo (Y3[11][10] / 0xA0B3359), yuv420p10le(tv, bt2020nc/bt2020/smpte2084, progressive), 1280x720, 25 fps, 25 tbr, 25 tbn.

[FranceBB]

Side announcement/notifications:
[list] FFmpeg has merged the patches which allow it to read certain selected frame properties.


Ah... Let the nightmare begin... :(


The FFmpeg patches mean that the relevant frame properties can now be passed through to the output

Is there a way to DISABLE it on the FFMpeg side or is it on by default and will always stay on?

'cause I noticed that when I use: propclearall() in Avisynth, FFMpeg gets bogus properties, but it's not like before, like it used to be.
I expected propclearall() to clear all frame properties and therefore make FFMpeg behave like it used be, but it doesn't, that only works from the Avisynth point of view, but now instead of not passing any properties it's passing the WRONG properties, which is even worse.
On the other hand, I can't make use of frame properties in any way in Avisynth and make sure that everything is always correct 'cause many filters just don't modify them, so I end up with the wrong properties anyway, hence the propclearall().

As I stated in the other topic, I very much believe that there should be a way, like a command, in FFMpeg to disable frame properties and go back to the way it was before.
This way, users can decide whether to use them or not.
On the Avisynth side we have propclearall() which has been my most used function and I've spread it everywhere in every script I made, but now we need something on the FFMpeg side to instruct it not to read any of what Avisynth is passing.

The reason why I've been so keen on insisting on this is that I've got lots of issues with some workflows that have been working for years and then stopped due to frame properties (like the top left chroma location thingy for XDCAM-50, remember?).
In this very moment in time it's much much better not to have frame properties than to have them for me.

[qyot27]

Ah... Let the nightmare begin... :(



Is there a way to DISABLE it on the FFMpeg side or is it on by default and will always stay on?

'cause I noticed that when I use: propclearall() in Avisynth, FFMpeg gets bogus properties, but it's not like before, like it used to be.
I expected propclearall() to clear all frame properties and therefore make FFMpeg behave like it used be, but it doesn't, that only works from the Avisynth point of view, but now instead of not passing any properties it's passing the WRONG properties, which is even worse.
On the other hand, I can't make use of frame properties in any way in Avisynth and make sure that everything is always correct 'cause many filters just don't modify them, so I end up with the wrong properties anyway, hence the propclearall().

As I stated in the other topic, I very much believe that there should be a way, like a command, in FFMpeg to disable frame properties and go back to the way it was before.
This way, users can decide whether to use them or not.
On the Avisynth side we have propclearall() which has been my most used function and I've spread it everywhere in every script I made, but now we need something on the FFMpeg side to instruct it not to read any of what Avisynth is passing.

The reason why I've been so keen on insisting on this is that I've got lots of issues with some workflows that have been working for years and then stopped due to frame properties (like the top left chroma location thingy for XDCAM-50, remember?).
In this very moment in time it's much much better not to have frame properties than to have them for me.

These are the default cases for those six properties in FFmpeg:
st->codecpar->field_order = AV_FIELD_UNKNOWN
st->codecpar->color_range = AVCOL_RANGE_UNSPECIFIED
st->codecpar->color_primaries = AVCOL_PRI_UNSPECIFIED
st->codecpar->color_trc = AVCOL_TRC_UNSPECIFIED
st->codecpar->color_space = AVCOL_SPC_UNSPECIFIED
st->codecpar->chroma_location = AVCHROMA_LOC_UNSPECIFIED
Which is correct, because there's no one true default for all rawvideo formats. If they aren't set, UNSPECIFIED or UNKNOWN is the correct value.

What it seems is happening, is that propDelete and propClearAll either remove the value, or set the value to 0 and treat that as implicitly having removed it. The problem is that the enums that set those properties consider 0 a valid number or somehow interpret the missing value to be equivalent to the value being 0. Which means it selects `case 0` instead of the default UNSPECIFIED case. These are the values for `case 0`:
st->codecpar->field_order = AV_FIELD_PROGRESSIVE
st->codecpar->color_range = AVCOL_RANGE_JPEG
st->codecpar->color_space = AVCOL_SPC_RGB
st->codecpar->chroma_location = AVCHROMA_LOC_LEFT
Primaries and Transfer have no case 0, so they actually work 'correctly' in the sense it reports back 'unknown'.

All the charts have a specific number of options corresponding to the cases in the switcher:
_FieldBased has three (0-2)
_ColorRange has two (0, 1)
_Primaries has twelve (1, 2, 4-12, 22)
_Transfer has seventeen (1, 2, 4-18)
_Matrix has fourteen (0-2, 4-14)
_ChromaLocation has six (0-5)

What am I getting at here? Well, if you happen to set a value for one of those properties to a number which does not coincide with one of the cases, it falls back to UNSPECIFIED, which would be the same as pre-frameprop treatment behavior.

So a safe value for all of them could be, say, -1. Or one way outside the boundaries, like 100. Or for _Primaries, _Transfer, and _Matrix, 3.

Does propClearAll actually remove the value entirely (propShow would imply that it does), or just set it to 0? Does FFmpeg (or more precisely, the compiler or C language itself) interpret the absense of a value to be equivalent to 0 when dealing with the switch-case? I have no clue. Those are areas that need to be explored when trying to figure it out, but what this does uncover is simple: if you want it to go back to how it was before, just use a function like this:
function propUnsetAll(clip c)
{
c.propClearAll()
last.propSet("_FieldBased",-1)
last.propSet("_ColorRange",-1)
last.propSet("_Primaries",-1)
last.propSet("_Transfer",-1)
last.propSet("_Matrix",-1)
last.propSet("_ChromaLocation",-1)
return last
}
and the result of
Import("../propunset.avsi")
FFmpegSource2("LG New York HDR UHD 4K Demo.ts",atrack=-1)
Spline64Resize(1280,720)
propUnsetAll()
is this:
ffmpeg version N-105655-ge4ad38d0f7-6ubuntu5 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 11 (Ubuntu 11.2.0-7ubuntu2)
configuration: --prefix=/usr --extra-version=6ubuntu5 --toolchain=hardened
--libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu
--arch=amd64 --enable-gpl --enable-debug --disable-stripping --enable-version3
--enable-mbedtls --enable-ladspa --enable-libaom --enable-libass
--enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio
--enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig
--enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm
--enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg
--enable-libopenmpt --enable-libopus --enable-libplacebo --enable-libpulse
--enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy
--enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora
--enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx
--enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg
--enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl
--enable-opengl --enable-sdl2 --enable-librsvg --enable-libmfx --enable-libdc1394
--enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint
--enable-frei0r --enable-libx264 --enable-avisynth --enable-vapoursynth
--cpu=skylake --extra-cflags='-march=skylake' --extra-ldflags=-pthread --pkg-config-flags=--static
libavutil 57. 21.100 / 57. 21.100
libavcodec 59. 21.100 / 59. 21.100
libavformat 59. 17.102 / 59. 17.102
libavdevice 59. 5.100 / 59. 5.100
libavfilter 8. 27.100 / 8. 27.100
libswscale 6. 5.100 / 6. 5.100
libswresample 4. 4.100 / 4. 4.100
libpostproc 56. 4.100 / 56. 4.100
Guessed Channel Layout for Input Stream #0.1 : stereo
Input #0, avisynth, from '/home/qyot27/Videos/test_hdr.avs':
Duration: 00:01:12.24, start: 0.000000, bitrate: 0 kb/s
Stream #0:0: Video: rawvideo (Y3[11][10] / 0xA0B3359), yuv420p10le, 1280x720, 25 fps, 25 tbr, 25 tbn
Stream #0:1: Audio: pcm_f32le, 48000 Hz, stereo, flt, 3072 kb/s
At least one output file must be specified

Stream #0:0: Video: rawvideo (Y3[11][10] / 0xA0B3359), yuv420p10le, 1280x720, 25 fps, 25 tbr, 25 tbn

[FranceBB]

Ahhhh, I get it now!
So that's why it didn't go back to unspecified when I was trying! Setting them all to -1 is indeed a nice trick. I'm gonna try to play with it on Monday as I get back to work, but I'm sure it's gonna work 'cause you already tried eheheheh.

It's incredible that I've never thought about setting them to impossible values! Thanks for the tip, Stephen!! ;)

[qyot27]

It certainly needs to be documented in a table, considering that it required taking a dive into FFmpeg's source code to even figure out what the entries in, to quote, "[X] as specified in ITU-T H.265 Table E.[Y]" were, and then compare it to the VapourSynth documentation on Resize (www.vapoursynth.com/doc/functions/video/resize.html) to make sure they were using the same values.

Just so it actually is somewhere, here you go, in table form:
+---------------------------------------+
| _FieldBased |
+------------------+--------------------+
| Value | Meaning |
+------------------+--------------------+
| 0 | Progressive |
+------------------+--------------------+
| 1 | Bottom field first |
+------------------+--------------------+
| 2 | Top field first |
+------------------+--------------------+
| All other values | Unknown / unset |
+------------------+--------------------+

+-----------------------------------------------+
| _ColorRange |
+------------------+----------------------------+
| Value | Meaning |
+------------------+----------------------------+
| 0 | Full range, a.k.a. JPEG |
+------------------+----------------------------+
| 1 | Limited range, a.k.a. MPEG |
+------------------+----------------------------+
| All other values | Unknown / unset |
+------------------+----------------------------+

+------------------------------------+
| _Primaries |
+------------------+-----------------+
| Value | Meaning |
+------------------+-----------------+
| 1 | BT709 |
+------------------+-----------------+
| 2 | Unspecified |
+------------------+-----------------+
| 4 | BT470M |
+------------------+-----------------+
| 5 | BT470BG |
+------------------+-----------------+
| 6 | SMPTE170M |
+------------------+-----------------+
| 7 | SMPTE240M |
+------------------+-----------------+
| 8 | FILM |
+------------------+-----------------+
| 9 | BT2020 |
+------------------+-----------------+
| 10 | SMPTE428 |
+------------------+-----------------+
| 11 | SMPTE431 |
+------------------+-----------------+
| 12 | SMPTE432 |
+------------------+-----------------+
| 22 | EBU3213 |
+------------------+-----------------+
| All other values | Unknown / unset |
+------------------+-----------------+

+------------------------------------+
| _Transfer |
+------------------+-----------------+
| Value | Meaning |
+------------------+-----------------+
| 1 | BT709 |
+------------------+-----------------+
| 2 | Unspecified |
+------------------+-----------------+
| 4 | GAMMA22 |
+------------------+-----------------+
| 5 | GAMMA28 |
+------------------+-----------------+
| 6 | SMPTE170M |
+------------------+-----------------+
| 7 | SMPTE240M |
+------------------+-----------------+
| 8 | LINEAR |
+------------------+-----------------+
| 9 | LOG |
+------------------+-----------------+
| 10 | LOG_SQRT |
+------------------+-----------------+
| 11 | IEC61966_2_4 |
+------------------+-----------------+
| 12 | BT1361_ECG |
+------------------+-----------------+
| 13 | IEC61966_2_1 |
+------------------+-----------------+
| 14 | BT2020_10 |
+------------------+-----------------+
| 15 | BT2020_12 |
+------------------+-----------------+
| 16 | SMPTE2084 |
+------------------+-----------------+
| 17 | SMPTE428 |
+------------------+-----------------+
| 18 | ARIB_STD_B67 |
+------------------+-----------------+
| All other values | Unknown / unset |
+------------------+-----------------+

+---------------------------------------+
| _Matrix |
+------------------+--------------------+
| Value | Meaning |
+------------------+--------------------+
| 0 | RGB |
+------------------+--------------------+
| 1 | BT709 |
+------------------+--------------------+
| 2 | Unspecified |
+------------------+--------------------+
| 4 | FCC |
+------------------+--------------------+
| 5 | BT470BG |
+------------------+--------------------+
| 6 | SMPTE170M |
+------------------+--------------------+
| 7 | SMPTE240M |
+------------------+--------------------+
| 8 | YCGCO |
+------------------+--------------------+
| 9 | BT2020_NCL |
+------------------+--------------------+
| 10 | BT2020_CL |
+------------------+--------------------+
| 11 | SMPTE2085 |
+------------------+--------------------+
| 12 | CHROMA_DERIVED_NCL |
+------------------+--------------------+
| 13 | CHROMA_DERIVED_CL |
+------------------+--------------------+
| 14 | ICTCP |
+------------------+--------------------+
| All other values | Unknown / unset |
+------------------+--------------------+

+------------------------------------+
| _ChromaLocation |
+------------------+-----------------+
| Value | Meaning |
+------------------+-----------------+
| 0 | LEFT |
+------------------+-----------------+
| 1 | CENTER |
+------------------+-----------------+
| 2 | TOPLEFT |
+------------------+-----------------+
| 3 | TOP |
+------------------+-----------------+
| 4 | BOTTOMLEFT |
+------------------+-----------------+
| 5 | BOTTOM |
+------------------+-----------------+
| All other values | Unknown / unset |
+------------------+-----------------+

[gispos]

Does propClearAll actually remove the value entirely (propShow would imply that it does), or just set it to 0? Does FFmpeg (or more precisely, the compiler or C language itself) interpret the absense of a value to be equivalent to 0 when dealing with the switch-case? I have no clue. Those are areas that need to be explored when trying to figure it out, but what this does uncover is simple: if you want it to go back to how it was before, just use a function like this:
function propUnsetAll(clip c)
{
c.propClearAll()
last.propSet("_FieldBased",-1)
last.propSet("_ColorRange",-1)
last.propSet("_Primaries",-1)
last.propSet("_Transfer",-1)
last.propSet("_Matrix",-1)
last.propSet("_ChromaLocation",-1)
return last
}

Then we are back to the ConvertToRGB32 error.
Avisynth and other plugin developers who use frame properties should agree on a consistent standard. Without the user having to assign a value to the properties themselves if they don't want to use them or delete them.

https://i.postimg.cc/qvGWh8K1/Error.jpg (https://postimages.org/)

If everyone agrees that -1 is unspecified, then it would probably work.

[pinterf]

Then we are back to the ConvertToRGB32 error.
Avisynth and other plugin developers who use frame properties should agree on a consistent standard. Without the user having to assign a value to the properties themselves if they don't want to use them or delete them.

https://i.postimg.cc/qvGWh8K1/Error.jpg (https://postimages.org/)

If everyone agrees that -1 is unspecified, then it would probably work.
Unspecified values are not equal to -1.
Some properties have no "Unspecified" value, such as _ColorRange: 0, 1, or does not exist at all.

[pinterf]

It can be a decision though how "invalid" values are treated by plugins or Avisynth's internal filters. ITU tables specify "unspecified" where applicable.

[Reel.Deel]

function propUnsetAll(clip c)
{
c.propClearAll()
last.propSet("_FieldBased",-1)
last.propSet("_ColorRange",-1)
last.propSet("_Primaries",-1)
last.propSet("_Transfer",-1)
last.propSet("_Matrix",-1)
last.propSet("_ChromaLocation",-1)
return last
}


For a property nuke function like this one, would it not be better to set all of the properties that have an actual unspecified value to that? For example, _Primaries, _Transfer and _Matrix can all be set to 2. Unless I missed something from the previous posts. Maybe even setting _ChromaLocation to 0 since left is the most common chroma placement.

edit: left the post in edit for about an hour and did not notice pinterf's reply.

It certainly needs to be documented in a table, considering that it required taking a dive into FFmpeg's source code to even figure out what the entries in, to quote, "[X] as specified in ITU-T H.265 Table E.[Y]" were, and then compare it to the VapourSynth documentation on Resize (www.vapoursynth.com/doc/functions/video/resize.html) to make sure they were using the same values.


Most of them are documented in the wiki but in various places:


Matrix: http://avisynth.nl/index.php/Convert#Parameter_Details
Chroma loc: http://avisynth.nl/index.php/Convert#Frame_properties_AVS.2B
and here: http://avisynth.nl/index.php/Internal_functions#Reserved_frame_property_names

But yes, I agree they need to be documented and formatted nicely. Once I get done with the internal filters rst docs I'll move onto the syntax portion and will include this info there. Hopefully by summer time :D

[wonkey_monkey]

Isn't PropDelete the thing to use?

[Boulder]

Isn't PropDelete the thing to use?

I agree with that idea, just a little cumbersome as of now. I always thought propClearAll deletes properties, but apparently a propDeleteAll would be very useful for backward compatibility cases.

[gispos]

Unspecified values are not equal to -1.
Some properties have no "Unspecified" value, such as _ColorRange: 0, 1, or does not exist at all.
I did not claim that either. I only wrote
"All (developers) should agree on a standard" and there the value -1 would be useful.

If now -1 would be considered as not existing, there would be no problems here.

[wonkey_monkey]

I tried PropClearAll and sure enough, no properties. All deleted. PropShow shows nothing and my own program, assuming I've written it correctly which I think I have, shows nothing.

If ffmpeg is getting spurious values from somewhere, is that not an ffmpeg problem?

[qyot27]

This is what the switch-cases for the frame properties look like in FFmpeg's AviSynth demuxer:
/* Field order */
switch (avs_library.avs_prop_get_int(avs->env, avsmap, "_FieldBased", 0, &error)) {
case 0:
st->codecpar->field_order = AV_FIELD_PROGRESSIVE;
break;
case 1:
st->codecpar->field_order = AV_FIELD_BB;
break;
case 2:
st->codecpar->field_order = AV_FIELD_TT;
break;
default:
st->codecpar->field_order = AV_FIELD_UNKNOWN;
}

All six of the frame properties in question are of type int (http://avisynth.nl/index.php/Internal_functions#Reserved_frame_property_names), so presumably, the appropriate API call to fetch that value is avs_prop_get_int.

If avs_prop_get_int is reporting the absence of a frame property as NULL, then it's equal to zero as guaranteed by C99. Zero is a valid value for four of those frame properties, which does not match it being absent (for _Primaries and _Transfer, no case 0 exists, so those do fall back to the default UNSPECIFIED value like they're supposed to).

[FranceBB]

I tried PropClearAll and sure enough, no properties. All deleted. PropShow shows nothing and my own program, assuming I've written it correctly which I think I have, shows nothing.

Of course, PropClearAll() does indeed remove all frame properties, so as long as you're gonna use filters inside Avisynth you're gonna be fine. The "problem" is when FFMpeg tries to access them.


If ffmpeg is getting spurious values from somewhere, is that not an ffmpeg problem?

It is... sort of.
I've looked at Stephen's code in FFMpeg and indeed each value has a default, namely:

/* Field order */
switch (avs_library.avs_prop_get_int(avs->env, avsmap, "_FieldBased", 0, &error)) {
case 0:
st->codecpar->field_order = AV_FIELD_PROGRESSIVE;
break;
case 1:
st->codecpar->field_order = AV_FIELD_BB;
break;
case 2:
st->codecpar->field_order = AV_FIELD_TT;
break;
default:
st->codecpar->field_order = AV_FIELD_UNKNOWN;
}

/* Color Range */
switch (avs_library.avs_prop_get_int(avs->env, avsmap, "_ColorRange", 0, &error)) {
case 0:
st->codecpar->color_range = AVCOL_RANGE_JPEG;
break;
case 1:
st->codecpar->color_range = AVCOL_RANGE_MPEG;
break;
default:
st->codecpar->color_range = AVCOL_RANGE_UNSPECIFIED;
}

/* Color Primaries */
switch (avs_library.avs_prop_get_int(avs->env, avsmap, "_Primaries", 0, &error)) {
case 1:
st->codecpar->color_primaries = AVCOL_PRI_BT709;
break;
case 2:
st->codecpar->color_primaries = AVCOL_PRI_UNSPECIFIED;
break;
case 4:
st->codecpar->color_primaries = AVCOL_PRI_BT470M;
break;
case 5:
st->codecpar->color_primaries = AVCOL_PRI_BT470BG;
break;
case 6:
st->codecpar->color_primaries = AVCOL_PRI_SMPTE170M;
break;
case 7:
st->codecpar->color_primaries = AVCOL_PRI_SMPTE240M;
break;
case 8:
st->codecpar->color_primaries = AVCOL_PRI_FILM;
break;
case 9:
st->codecpar->color_primaries = AVCOL_PRI_BT2020;
break;
case 10:
st->codecpar->color_primaries = AVCOL_PRI_SMPTE428;
break;
case 11:
st->codecpar->color_primaries = AVCOL_PRI_SMPTE431;
break;
case 12:
st->codecpar->color_primaries = AVCOL_PRI_SMPTE432;
break;
case 22:
st->codecpar->color_primaries = AVCOL_PRI_EBU3213;
break;
default:
st->codecpar->color_primaries = AVCOL_PRI_UNSPECIFIED;
}

/* Color Transfer Characteristics */
switch (avs_library.avs_prop_get_int(avs->env, avsmap, "_Transfer", 0, &error)) {
case 1:
st->codecpar->color_trc = AVCOL_TRC_BT709;
break;
case 2:
st->codecpar->color_trc = AVCOL_TRC_UNSPECIFIED;
break;
case 4:
st->codecpar->color_trc = AVCOL_TRC_GAMMA22;
break;
case 5:
st->codecpar->color_trc = AVCOL_TRC_GAMMA28;
break;
case 6:
st->codecpar->color_trc = AVCOL_TRC_SMPTE170M;
break;
case 7:
st->codecpar->color_trc = AVCOL_TRC_SMPTE240M;
break;
case 8:
st->codecpar->color_trc = AVCOL_TRC_LINEAR;
break;
case 9:
st->codecpar->color_trc = AVCOL_TRC_LOG;
break;
case 10:
st->codecpar->color_trc = AVCOL_TRC_LOG_SQRT;
break;
case 11:
st->codecpar->color_trc = AVCOL_TRC_IEC61966_2_4;
break;
case 12:
st->codecpar->color_trc = AVCOL_TRC_BT1361_ECG;
break;
case 13:
st->codecpar->color_trc = AVCOL_TRC_IEC61966_2_1;
break;
case 14:
st->codecpar->color_trc = AVCOL_TRC_BT2020_10;
break;
case 15:
st->codecpar->color_trc = AVCOL_TRC_BT2020_12;
break;
case 16:
st->codecpar->color_trc = AVCOL_TRC_SMPTE2084;
break;
case 17:
st->codecpar->color_trc = AVCOL_TRC_SMPTE428;
break;
case 18:
st->codecpar->color_trc = AVCOL_TRC_ARIB_STD_B67;
break;
default:
st->codecpar->color_trc = AVCOL_TRC_UNSPECIFIED;
}

/* Matrix coefficients */
switch (avs_library.avs_prop_get_int(avs->env, avsmap, "_Matrix", 0, &error)) {
case 0:
st->codecpar->color_space = AVCOL_SPC_RGB;
break;
case 1:
st->codecpar->color_space = AVCOL_SPC_BT709;
break;
case 2:
st->codecpar->color_space = AVCOL_SPC_UNSPECIFIED;
break;
case 4:
st->codecpar->color_space = AVCOL_SPC_FCC;
break;
case 5:
st->codecpar->color_space = AVCOL_SPC_BT470BG;
break;
case 6:
st->codecpar->color_space = AVCOL_SPC_SMPTE170M;
break;
case 7:
st->codecpar->color_space = AVCOL_SPC_SMPTE240M;
break;
case 8:
st->codecpar->color_space = AVCOL_SPC_YCGCO;
break;
case 9:
st->codecpar->color_space = AVCOL_SPC_BT2020_NCL;
break;
case 10:
st->codecpar->color_space = AVCOL_SPC_BT2020_CL;
break;
case 11:
st->codecpar->color_space = AVCOL_SPC_SMPTE2085;
break;
case 12:
st->codecpar->color_space = AVCOL_SPC_CHROMA_DERIVED_NCL;
break;
case 13:
st->codecpar->color_space = AVCOL_SPC_CHROMA_DERIVED_CL;
break;
case 14:
st->codecpar->color_space = AVCOL_SPC_ICTCP;
break;
default:
st->codecpar->color_space = AVCOL_SPC_UNSPECIFIED;
}

/* Chroma Location */
switch (avs_library.avs_prop_get_int(avs->env, avsmap, "_ChromaLocation", 0, &error)) {
case 0:
st->codecpar->chroma_location = AVCHROMA_LOC_LEFT;
break;
case 1:
st->codecpar->chroma_location = AVCHROMA_LOC_CENTER;
break;
case 2:
st->codecpar->chroma_location = AVCHROMA_LOC_TOPLEFT;
break;
case 3:
st->codecpar->chroma_location = AVCHROMA_LOC_TOP;
break;
case 4:
st->codecpar->chroma_location = AVCHROMA_LOC_BOTTOMLEFT;
break;
case 5:
st->codecpar->chroma_location = AVCHROMA_LOC_BOTTOM;
break;
default:
st->codecpar->chroma_location = AVCHROMA_LOC_UNSPECIFIED;
}
}


So we do indeed have a default which is "UNSPECIFIED" which makes FFMpeg act the same as before.
The problem is that when FFMpeg calls the Avisynth APIs asking for frame properties, Avisynth will return "null" after it has been asked for 'cause PropClearAll() removed each and every property.
Here's the tricky bit: "null" gets interpreted by FFMpeg as... well... 0, which is a valid value for the frame properties, so it never goes to default which is "UNSPECIFIED".

It does go there if the Avisynth APIs version are lower than 9 (it used to be 8.1 but it caused a segmentation fault in 8.0 and it was a pain to solve so Stephen bumped them to 9), so effectively unless someone is using Avisynth 3.7.1 or 3.7.2 (so if they're using older version like 3.7.0 etc) the values will be set to default and it will be as it was before thanks to this bit of code:

if (avs_library.avs_get_version(avs->clip) >= 9) {

frame = avs_library.avs_get_frame(avs->clip, framedata);
avsmap = avs_library.avs_get_frame_props_ro(avs->env, frame);

I think Stephen would agree with this.


EDIT: Oh, he already replied and yeah it's like I thought


If avs_prop_get_int is reporting the absence of a frame property as NULL, then it's equal to zero as guaranteed by C99. Zero is a valid value for four of those frame properties, which does not match it being absent.

[MasterNobody]

Shouldn't ffmpeg check error value of avs_prop_get_int calls? As far as I looked it should be GETPROPERROR_UNSET in case of absent property.

[qyot27]

After looking more at the Readme and headers, avs_prop_get_type can be used to check the status of a frame property and whether it's been set or not. This fixes the issue on my end:
https://github.com/qyot27/FFmpeg/commit/a3d542ab0fe4c7d4b10a4769f88658c6f52918b9

I don't know if this approach is better or worse than checking the error value, though it does provide a way of knowing whether the property is set before an error would occur.

[FranceBB]

I don't know if this approach is better or worse

it's better.
Thank you for this. I look forward to try it on Monday.

[StvG]

After looking more at the Readme and headers, avs_prop_get_type can be used to check the status of a frame property and whether it's been set or not. This fixes the issue on my end:
https://github.com/qyot27/FFmpeg/commit/a3d542ab0fe4c7d4b10a4769f88658c6f52918b9

I don't know if this approach is better or worse than checking the error value, though it does provide a way of knowing whether the property is set before an error would occur.

Another variant: int error = 0;
/* Field order */
int64_t field = avs_library.avs_prop_get_int(avs->env, avsmap, "_FieldBased", 0, &error);
if (error != 0) { // no property
st->codecpar->field_order = AV_FIELD_UNKNOWN;
}
else {
switch (field) {
case 0:
st->codecpar->field_order = AV_FIELD_PROGRESSIVE;
break;
case 1:
st->codecpar->field_order = AV_FIELD_BB;
break;
case 2:
st->codecpar->field_order = AV_FIELD_TT;
break;
default:
st->codecpar->field_order = AV_FIELD_UNKNOWN;
}
}

[Dogway]

Are the weights of this function ops correct? I plan to add a score to expressions, I searched on instructions tables (https://www.agner.org/optimize/instruction_tables.pdf) but I'm not sure I got it right or whether it depends on CPU arch. I based it on Haswell.

function ex_lutperf(string str) {

str= " "+str+" "
a = FindStrInstance(str,"+") * 1
s = FindStrInstance(str,"-") * 2
m = FindStrInstance(str,"*") * 2
d = FindStrInstance(str,"/") * 4
t = FindStrInstance(str,"?") * 3
e = FindStrInstance(str,"^") * 6
mn = FindStrInstance(str," min " ) * 2
mx = FindStrInstance(str," max " ) * 2
ab = FindStrInstance(str," abs " ) * 2
r = FindStrInstance(str," sqrt " ) * 2
l = FindStrInstance(str," log " ) * 2
c = FindStrInstance(str," clip " ) * 3
x = FindStrInstance(str," exp " ) * 6
si = FindStrInstance(str," sin " ) * 6
co = FindStrInstance(str," cos " ) * 6
at = FindStrInstance(str," atan " ) * 6
a2 = FindStrInstance(str," atan2 ") * 6

return a+s+m+d+t+e+mn+mx+ab+r+l+c+x+si+co+at+a2 }

[pinterf]

Are the weights of this function ops correct? I plan to add a score to expressions, I searched on instructions tables (https://www.agner.org/optimize/instruction_tables.pdf) but I'm not sure I got it right or whether it depends on CPU arch. I based it on Haswell.

function ex_lutperf(string str) {

str= " "+str+" "
a = FindStrInstance(str,"+") * 1
s = FindStrInstance(str,"-") * 2
m = FindStrInstance(str,"*") * 2
d = FindStrInstance(str,"/") * 4
t = FindStrInstance(str,"?") * 3
e = FindStrInstance(str,"^") * 6
mn = FindStrInstance(str," min " ) * 2
mx = FindStrInstance(str," max " ) * 2
ab = FindStrInstance(str," abs " ) * 2
r = FindStrInstance(str," sqrt " ) * 2
l = FindStrInstance(str," log " ) * 2
c = FindStrInstance(str," clip " ) * 3
x = FindStrInstance(str," exp " ) * 6
si = FindStrInstance(str," sin " ) * 6
co = FindStrInstance(str," cos " ) * 6
at = FindStrInstance(str," atan " ) * 6
a2 = FindStrInstance(str," atan2 ") * 6

return a+s+m+d+t+e+mn+mx+ab+r+l+c+x+si+co+at+a2 }
sqrt, log, exp, sin, cos, atan, atan2 are not single instruction functions, their core can be 20-100 more complex than a simple abs. What is common: operand load and conversion to float (and scale if needed) before the operation, and the final (scale back and ) check range, convert back to integer and store to memory.

[pinterf]

Avisynth+ 3.7.2 test8
https://drive.google.com/uc?export=download&id=1G79jALjVbOS-2ZbWL1YZGmfe32dJhdli

all 3.7.2 changes so far:
20220227 3.7.2-WIP
------------------
- PropDelete: accept a non-empty array string as list of property names to remove
Parameter is not optional, and has no name. It can be either a string (as before) or an array of strings
propDelete("_Matrix") # old syntax, still accepted
propDelete(["_Matrix", "_ColorRange"])
- PropCopy: new string parameter "props" as list of property names to remove
"props": a non-empty array of strings

old syntax, still accepted:
propCopy(org,true) # merge from all org's properties
propCopy(org,false) # erase all then copy all org's properties (exact copy)
new syntax
propCopy(org,true,props=["_Matrix", "_ColorRange"]) # merge
propCopy(org,props=["_Matrix", "_ColorRange"]) # erase all then copy only selected
- Histogram Levels: stop using shades of grey on top of bars.
- Histogram Levels: use bar color 255 for RGB instead of Y's 235. (and scaled eqivivalents)
- Fix: Histogram "Levels": prevent crash when factor=0.0
- Fix: Histogram "Levels": fix regression incorrect "factor" applied for U/V part drawing when format was subsampled (non-444)
Regression since 20160916 r2666 (commit 986e2756)
- Histogram "Audiolevels" and StereoOverlay to deny planar RGB
- Histogram "Luma": support 10-16 and 32 bits
- Histogram: give parameter name "factor" and type 'float' for Histogram's unnamed optional parameter used in "Level" mode.
Other modes just ignore this parameter if given.
- Fix: Histogram "color" may crash on certain dimensions for subsampled formats.
Regression since 20180301 r2632.
- Fix: Histogram "color" and "color2" mode check and give error on Planar RGB
- Fix: missing Histogram "color2" CCIR rectangle top and bottom line (black on black)
Regression since 3.6.2-test1 (commit 1fc82f03)
- Fix: Compare to support 10-14 bits
was: factor was always using 65535 (2^16-1) instead of (2^bit depth - 1)
was: 16 bit luma/rgb color values were used for drawing graph
- Fix: Compare
'channels' parameter default to "Y" when input is greyscale;
instead of "YUV" which was giving error because of U and V does not exist for this format.
- ShowRed/Green/Blue/Alpha/Y/U/V
- support YUY2 input
- support YV411 output
- (not changed: ShowU/ShowV may give error for 420, 422 or 411 format outputs when clip dimensions are
not eligible for a given output subsampling (check for appropriate mod2 or mod4 width or height)
- Copy alpha from source when target is alpha-capable
- Fill alpha with maximum pixel value when target is alpha-capable but source ha no alpha component
- Delete _Matrix and _ChromaLocation frame properties when needed.
- More consistent behaviour for YUV and planar RGB sources.

Default pixel_type is adaptive. If none or empty ("") is given for pixel_type then target format is
- YUV444 when source is Y, YUV or YUVA
- RGB32/64 (packed RGB) when source is RGB24/32/48/64
- RGBP (planar RGB) when source is RGBP or RGBAP

When 'rgb' is given for pixel_type then then target format is

- RGB32/64 (packed) when source is RGB24/32/48/64 - old, compatible way
- RGB planar when source is planar RGB(A) or YUV(A) or Y - changed from rgb32/64 because all bit depth must be supported

When 'yuv' is given (new option!) for pixel_type then then target format is

- YUV444 for all sources

Also there is a new option when pixel_type is still not exact, and is given w/o bit depth.
pixel_type which describes the format without bit depth is automatically extended to a valid video string constant:

y, yuv420, yuv422, yuv444, yuva420, yuva422, yuva444, rgbp, rgbap

Examples:

32 bit video and pixel_type 'y' will result in "Y32"
16 bit video and pixel_type 'yuv444' will result in "YUV444P16"
8 bit video and pixel_type 'rgbap' will result in "RGBAP8"

- Fix #263. Escaping double-quotes results in error
- Allow top_left (2) and bottom_left (4) chroma placements for 422 in colorspace conversions, they act as "left" (0, "mpeg2")
in order not to give error with video sources which have _ChromaLocation set to other than "mpeg2"
See https://trac.ffmpeg.org/ticket/9598#comment:5
- Fix: Expr LUT operation Access Violation on x86 + AVX2 due to an unaligned internal buffer (<32 bytes)
- Fix: Chroma full scale as ITU Rec H.273 (e.g +/-127.5 and not +/-127) in internal converters, ColorYUV and Histogram
- Fix #257: regression in 3.7.1: GreyScale to not convert to limited range when input is RGB. Regression in 3.7.1
Accepts only matrix names of limited range as it is put in the documentation.
- Fix #256: ColorYUV(analyse=true) to not set _ColorRange property to "full" if input has no such
property and range cannot be 100% sure established. In general: when no _ColorRange for input and
no parameter which would rely on a supposed default (such as full range for gamma), then an
output frame property is not added.
When no _ColorRange for input and no other parameters to hint color range then
- gamma<>0 sets full range
- opt="coring" sets limited range
- otherwise no _ColorRange for output would be set
- Overlay (#255): "blend": using accurate formula using float calculation. 8 bit basic case is slower now when opacity=1.0.
Higher bit depths and opacity<1.0 cases are quicker.
Mask processing suffered from inaccuracy. For speed reasons mask value 0 to 255 were handled
as mask/256 instead of mask/255. Since with such calculation maximum value was not the expected 1.0 but rather 255/256 (0.996)
this case was specially treated as 1.0 to give Overlay proper results at least the the two extremes.
But for example applying mask=129 to pixel=255 resulted in result_pixel=128 instead of 129. This was valid on higher bit depths as well.
Note 3.7.2 Test2 has a regression of broken maskless mode for 0<opacity<1 which was fixed in 3.7.2 test 3
- Fix: Attempt to resolve deadlock when an Eval'd (Prefetch inside) Clip result is
used in Invoke which calls a filter with GetFrame in its constructor.
(AvsPMod use case which Invokes frame prop read / ConvertToRGB32 after having the AVS script evaluated)
Remark: problem emerged in 3.7.1test22 which is trying to read frame properties of the 0th frame in its constructor.
A similar deadlock situation was already fixed earlier in Neo branch and had been backported but it did not cover this use case.
Note: Prefetch(1) case was fixed in 3.7.2 Test3

[Dogway]

Thank you! Do you think we should treat filters as to not overflow video signal max value (for HBD)? Or keeping "range_max" as "range_size-1" albeit illegal video signal?

EDIT: I think copypasta typo in propCopy "as list of property names to copy"
Do you think a negative selection propCopy is possible? Like I want to copy all frameprops from clip to mask, including custom ones (future scene changes, stats), but not "_ColorRange", "_Transfer", etc.
Also I noticed that "props" is val type? "propCopy(org,true,props="_ChromaLocation")", not sure if intended.
And got a crash with ArrayDel(arr,-1), I was trying (with -1) to not delete an index if a condition was met

[kedautinh12]

Why test 8 don't have Cuda ver??

[guest]

Why test 8 don't have Cuda ver??

Curious how it jumped from test 3 to test 8.....

And no "normal" x86, only XP option.

[kedautinh12]

Curious how it jumped from test 3 to test 8.....

And no "normal" x86, only XP option.

x86_xp can use in win 11, i checked it. You can compare between test 3 and test 8 with changelogs

[guest]

x86_xp can use in win 11, i checked it. You can compare between test 3 and test 8 with changelogs

OK, thanks for that...

But who would use x86 much these days, anyway..

[Reel.Deel]

Curious how it jumped from test 3 to test 8.....


The intermediate test versions were released in the issues section but only for x64. I'm working on the online/offline documentation (https://github.com/AviSynth/AviSynthPlus/pull/264) and found a few bugs, pinterf fixed those issues and provided test versions along the way :)

But who would use x86 much these days, anyway..

I don't use x86 but still have it installed, does no harm.

[pinterf]

Why test 8 don't have Cuda ver??
I stopped including "CUDA" in the name, it's basically a non-XP 64 bit version of Avisynth+ which _can_ accept specially written CUDA Avisynth plugins. Knowing the fact that almost no one is using these Avisynth-Neo CUDA-aware interface I removed it from the name. Such an Avisynth+ build does not use CUDA at all, it is just able to host such filters; the only known plugin set which is compatible with this interface is https://github.com/pinterf/AviSynthCUDAFilters which is there for people/developers who'd like to play with this interface.

[pinterf]

Thank you! Do you think we should treat filters as to not overflow video signal max value (for HBD)? Or keeping "range_max" as "range_size-1" albeit illegal video signal?
I hope all internal Avisynth filters keep this rule, along with the external plugins I maintain. When it is not fulfilled, e.g. you encounter >1023 pixel values for a 10 bit clip, it is a bug, please report it.


EDIT: I think copypasta typo in propCopy "as list of property names to copy"
Do you think a negative selection propCopy is possible? Like I want to copy all frameprops from clip to mask, including custom ones (future scene changes, stats), but not "_ColorRange", "_Transfer", etc.
So this idea was not only in my mind, good, I was thinking about it, even a regex or wildcard-support version.


Also I noticed that "props" is val type? "propCopy(org,true,props="_ChromaLocation")", not sure if intended.
No. It is array of string. But in Avisynth - when the function signature expects array - a single parameter is reported as a single-element array. PropCopy syntax does not need array delimiter brackets for a single property name.


And got a crash with ArrayDel(arr,-1), I was trying (with -1) to not delete an index if a condition was met
Thanks to the report.

[FranceBB]

And no "normal" x86, only XP option.

XP version is compiled with Zc:threadSafeInit and using v141_xp instead of v142 and it targets SSE2 (but so do non XP builds so who cares), that's it, nothing special about it, it's compatible with every OS from XP onward without any issues, so you can use it in XP, Vista, 7, 8, 8.1, 10, 11 ;)

[Dogway]

I hope all internal Avisynth filters keep this rule, along with the external plugins I maintain. When it is not fulfilled, e.g. you encounter >1023 pixel values for a 10 bit clip, it is a bug, please report it.

I got deceived by the levels chart in the previous Reel.Deel's post (https://forum.doom9.org/showthread.php?p=1964603#post1964603).
The valid signal/data range only applies to legal range signals. Full range still occupies the full size minus 1.

I checked on ITU-R BT.2100-2 Table 9 (https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.2100-2-201807-I!!PDF-E.pdf#page=12), and fixed my values.

# data_max = Select (bitd, [255.,255.], [1019.,1023.], [4079.,4095.], [16319.,16383.], [65279.,65535.], [16711679.,16776960.], [ 1., 1.]) # Legal upper bound for video signal in legal range
# data_min = Select (bitd, [ 0., 0.], [ 4., 0.], [ 16., 0.], [ 64., 0.], [ 256., 0.], [ 65536., 0.], [ 0., 0.]) # Legal lower bound for video signal in legal range


With that said the chart in the Levels wiki ("http://avisynth.nl/index.php/Levels") should be fixed.

[Reel.Deel]

With that said the chart in the Levels wiki ("http://avisynth.nl/index.php/Levels") should be fixed.

I fixed it the rst docs (locally):

https://i.ibb.co/LZ1RHG7/avisynthplus-levels.png"]https://i.ibb.co/LZ1RHG7/avisynthplus-levels.png

I figured you misunderstood my question. I'll fix the wiki later, the levels page and also the autoscale (http://avisynth.nl/index.php/Autoscale_parameter) page.

[magnetite]

I stopped including "CUDA" in the name, it's basically a non-XP 64 bit version of Avisynth+ which _can_ accept specially written CUDA Avisynth plugins. Knowing the fact that almost no one is using these Avisynth-Neo CUDA-aware interface I removed it from the name. Such an Avisynth+ build does not use CUDA at all, it is just able to host such filters; the only known plugin set which is compatible with this interface is https://github.com/pinterf/AviSynthCUDAFilters which is there for people/developers who'd like to play with this interface.

Is DTL's MVTools2 mod an exception, since it uses DX12 compute shaders? Or is that outside of CUDA and doesn't require a special Avisynth+ build to run?

[DTL]

" outside of CUDA and doesn't require a special Avisynth+ build to run?"

Yes. All DX12 interfacing is inside MAnalyse only. Planned to extended to MDegrainN to reuse uploaded to accelerator frames.
But it require Windows10 (or newer) to have 'native' DX12 interface (and drivers from hardware accelerator manufacturer to support DX12-ME + compute shaders 5). I read about attempts to integrate DX12 to Win7 but it looks need significant re-write of user-application too to support these hacks. Also it was reported to run at AMD hardware that is not CUDA compatible at all - as was expected from Microsoft+DX hardware abstraction layers. May be new enough intel hardware will also support it someday (or already - not tested).

It is really some selection for developers: which API to hardware accelerators to implement/support - CUDA may be compatible between Windows/UNIX builds but limited to NVIDIA HW manufacturer only. The DirectX compatible between many HW accelerators manufacturers but limited to Windows host OS only (may be some hacks for UNIX exist or may be developed in the future). Microsoft invest into DirectX hardware abstraction infrastructure (user-side API and requirements for drivers developers / hardware developers) and naturally want to get profit - so UNIX 'official' DirectX support may be limited if any at all.

As I understand the Avisynth developers do not like idea to be limited by Windows only so the development of core support for offloading some computing to accelerator via Microsoft->DirectCompute (DX+compute shaders) API is unlikely. So it is currently possible only inside some single or sequenced filters from the single plugins pack with some additional interfacing about upoladed to accelerator resources for reusing.

[wonkey_monkey]

What would be the recommended way in C++ to check if the current AviSynth environment supports properties?

[pinterf]

What would be the recommend way in C++ to check if the current AviSynth environment supports properties?
Something like this commit:
https://github.com/pinterf/AjkMedian/commit/f99fdaada9af8360235e05baf1eb78af160354f6
With this technique your plugin will pass existing properties further, even if property read-write is not implemented. Same V8 check is enough for reading properties.

When you actively want to set properties I recommend a second check against V9 interface version, with which you can use an additional interface function env->MakePropertyWritable when the original frame content must be untouched. Otherwise after a MakeWritable or NewVideoFrameP you can use the getFramePropsRW

https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/FilterSDK/FilterSDK.html#what-s-new-in-the-api-v8
https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/FilterSDK/FilterSDK.html#what-s-new-in-the-api-v9
and the env->CheckVersion part under
https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/FilterSDK/Cplusplus_api.html#getenvproperty-v8

[wonkey_monkey]

Thanks. I must get around to updating my plugins one day soon...

[kedautinh12]

Wow, new avatar monkey :D

[ryrynz]

Pinterf needs one. Dogway needs a Dog. Reeldeel can have a movie reel.

[wonkey_monkey]

Would consideration be given to changing ImageSource so that it selects the most appropriate colourspace for the depth of the input image? By default 16-bit PNGs produce an 8-bit clip, which seems at odds with other source filters. And whether or not there is any transparency could decide whether it should be RGB[24/48] or RGB[32/64] (or maybe it should default to planar RGB nowadays).

As I understand it JPEG is also usually YCbCr, which could also be returned as a valid colourspace, instead of converting to RGB...?

[DTL]

JPEG is 'full-range' typically I think. Also it can have 4:4:4 non chroma subsampled form (and looks 4:2:2 too). So in the old Avisynth to be more compatible without additional range properties it looks was converted to RGB type. Keeping YCrCb (also with chroma subsampled) may confuse users with unnatural to video 'full-range' YCrCb.
Also JPEG is typically for static pictures encoding same as for old non-moving pictures pixel-based from computers. So to load JPEG same as BMPs and other pixel-based formats it may be converted to the same non-video RGB. To indicate it is static picture. Not picture from moving picture sequence.

[qyot27]

Would consideration be given to changing ImageSource so that it selects the most appropriate colourspace for the depth of the input image? By default 16-bit PNGs produce an 8-bit clip, which seems at odds with other source filters. And whether or not there is any transparency could decide whether it should be RGB[24/48] or RGB[32/64] (or maybe it should default to planar RGB nowadays).

As I understand it JPEG is also usually YCbCr, which could also be returned as a valid colourspace, instead of converting to RGB...?
The problem is that DevIL doesn't support outputting YUV or planar RGB. It *might* be able to do RGB48/RGBA64, but the way the plugin currently works doesn't let that go through. Apart from that, technically FFMS2 does have an FFImageSource function (https://github.com/FFMS/ffms2/blob/master/src/avisynth/avisynth.cpp#L320) that'll do whatever FFmpeg does.

The real solution would be to move to a different image library that does support YUV and high bit depth images easily. From a basic search, the only one I feel probably comes close to both the goal ImageSource had in using DevIL and that supports YUV/HBD stuff is SAIL (https://github.com/HappySeaFox/sail). If that happens, it'd be down the road a ways.

[pinterf]

Avisynth+ 3.7.2 test 12 (20220303) (https://drive.google.com/uc?export=download&id=1PMdVp0U0Jq9b88ahg20SBotU_UB-w3oP)
20220303 3.7.2-WIP
------------------
- propCopy: able to specify that the property list is negative.
bool "exclude" = false # default: "props" is positive list

propCopy(org,true,props=["_Matrix", "_ColorRange"], exclude=false) # merge only two properties
propCopy(org,true,props=["_Matrix", "_ColorRange"], exclude=true) # merge all, except listed ones
propCopy(org,props=["_Matrix", "_ColorRange"]) # erase all then copy only selected
propCopy(org,props=["_Matrix", "_ColorRange"], exclude = true) # erase all, then copy all, except listed ones

- Version()
New optional parameters

int length, int width, int height, string pixel_type, clip c

Version clip defaults:
length=240, width = -1, height = -1 (-1: automatically sized to fit for font size 24)
pixel_type = "RGB24"

When 'clip' (a format template) is specified then pixel_type, length,
fps data, width and height are defined from it.
If any additional 'length', 'width', 'height', 'pixel_type' parameter is given, it overrides defaults.
When width and height is given and is <= 0 then it is treated as 'automatic'

Covers feature request Issue #261

- BlankClip: allow 'colors' size more than actual number of planes.
If an array is larger, further values are simply ignored.
- BlankClip, AddBorders, LetterBox: no A=0 check for non-YUVA
- Fade filter family new parameters
int 'color_yuv'
array of float 'colors'
similar to BlankClip
- MergeRGB, MergeARGB
- add MergeARGB parameter "pixel_type", similar to MergeRGB
- accept pixel_type other than packed RGB formats, plus a special one is "rgb"
- output format is planar rgb(a) (MergeRGB/MergeARGB) when
- pixel_type = "rgb" or
- pixel_type is empty and
- either input is planar RGB
- either input is different from 8 or 16 bits (no packed RGB formats there)
- pixel_type is explicitely set to a valid planar rgb constant e.g. "RGBP10"
- Accept planar RGB clip in place of input clips and the appropriate color plane is copied from them
- Fill alpha channel with zero when MergeRGB output pixel_type format is specified to have an alpha plane
- frame property source is the R clip; _Matrix and _ChromaLocation are removed if R is not an RGB clip

[gispos]

Avisynth+ 3.7.2 test 12 (20220303) (https://drive.google.com/uc?export=download&id=1PMdVp0U0Jq9b88ahg20SBotU_UB-w3oP)

Thanks, for 32bit Is there only the XP version?

[kedautinh12]

Thanks, for 32bit Is there only the XP version?

No, it's can use in another windows ver

[pinterf]

Yes. XP and newer.

[gispos]

I assumed that, but I thought that the XP version is neutered. :)

[LigH]

It may contain a more compatible but slightly outdated threading code?

[pinterf]

I assumed that, but I thought that the XP version is neutered. :)
There is still an option for the v141_xp toolset in Visual Studio 2022 though I'm still using Visual Studio 2019; my developer PC migration is in progress since January.
But I hate it, for some reason building an XP version takes at least 25 minutes, while normal version is ready in 3-5 minutes.
Threading code - static variable initializing - is not thread safe in XP versions (-Zcthreadsafeinit- option is needed), I met this phenomenon only once, in masktools2, when I spent some days understanding a thing that normally was impossible to happen, then had to do a workaround.

[FranceBB]

Speaking of compilation, has anyone ever done a speed test when comparing the normally compiled x64 version (non XP one) which targets SSE2 and a re-compiled x64 version (non XP one) that targets AVX2 to see if there are any speed benefits?
The reason why I'm asking this is that IF there are speed benefits, we might have the x86/x64 XP version still compiled with Zc:threadSafeInit using v141_xp targeting SSE2 and the x86/x64 non XP ones compiled with v142 and targeting AVX2 instead so that those with a newer OS but an older CPU will still be able to "fallback" on the XP versions which are SSE2 proof and those with new beefy machines can benefit from AVX2 for the C/C++ part of the code that doesn't have intrinsics.
What do you think?

[pinterf]

Speaking of compilation, has anyone ever done a speed test when comparing the normally compiled x64 version (non XP one) which targets SSE2 and a re-compiled x64 version (non XP one) that targets AVX2 to see if there are any speed benefits?
The reason why I'm asking this is that IF there are speed benefits, we might have the x86/x64 XP version still compiled with Zc:threadSafeInit using v141_xp targeting SSE2 and the x86/x64 non XP ones compiled with v142 and targeting AVX2 instead so that those with a newer OS but an older CPU will still be able to "fallback" on the XP versions which are SSE2 proof and those with new beefy machines can benefit from AVX2 for the C/C++ part of the code that doesn't have intrinsics.
What do you think?
Targeting SSE2 means only that the DLL needs at least SSE2.
Most internal codes however benefit SSE4.1, AVX, AVX2. Almost all internal filters have both SSE2/SSE4.1 and AVX/AVX2 versions. Choosing the best available version is dynamic. Support for modern processors does not mean that the whole DLL is compiled for e.g. AVX2. One single DLL contains all processor versions, it's the filter writer's task to run the appropriate version, e.g. if I'd let run SSE4.1 version of a resizer on an old AMD it would crash.

So Avisynth built for XP still contain all AVX/AVX2 optimizations. These AVX+ optimizations are not available on XP because Windows XP as an operating system does not support saving extra AVX/AVX2 registers so their use is disabled. Even if the processor has AVX2 instruction set, it won't be reported as a processor feature under XP, so the function dispatcher will use at most SSE4.1 versions of internal filters.

[FranceBB]

I know, but I was wondering about the plain C++ part of the Avisynth core, whether it would benefit or not, but at this point I think everything in the core (I mean in the frameserver itself including all functions) has the relative assembly optimization which will be picked dynamically every time it loads and it detects the instructions set of the host CPU, right?

The reason why I asked is that I know that for instance some external plugins are in C++ only, so it's up to the compiler to generate optimized code given that there are no manually written intrinsics, however if I got it right this doesn't apply to the AVS core, so to the frameserver itself, 'cause everything inside the core HAS manually written intrinsics anyway and those will be picked dynamically at run time so that each CPU goes to the appropriate codepath, so the compiler doesn't have to optimize anything at all given that everything has already been written by a human being, right?

[DTL]

"and the x86/x64 non XP ones compiled with v142 and targeting AVX2 instead"

AVX is a decade+ old. If you can build the executable it is better to set optimization to your current CPU architecture. If your CPU is AVX512 - set it to compiler and it may use extra register file available in AVX512-capable chip. It is 4 times larger in compare with old AVX. Also if possible - try intel optimizing compiler and multi file interprocedure optimization. It may also make some visible performance gain. Currently several different C/C++ compilers available and may generate significantly different executable (even from intrinsics - it is not hardcoded ASM but only a 'good hint' to compiler). So the more AVS goes away from hardcoded asm to intrinsics - the more it depend on exact compiler even in 'pseudo-asm' of its part.

Also typically the large complex projects are shipped with 'fail-safe' compiler options to make building somehow working executable easier. But it not mean it is fastest possible - because more complex compiler options like interprocedure optimization may cause build fail. So if someone have time to experiment - it may take a day or days to try different compiler options and different compilers to make best speed executable even without touching program text at all. Each compile is slow so it takes hours to test several compile options with full rebuild.

[wonkey_monkey]

According to the Wiki page on Expr:

string scale_inputs = "none"

"floatUV": (since v3.5) chroma pre and post shift by 0.5 for 32 bit float pixels, thus having them in the range of 0..1 instead of -0.5..+0.5 during Expr evaluation

I'm on 3.7.1 (compiled from source) but when I tried this parameter value, I got this error:

https://i.imgur.com/Bdx6i8K.png

Which is wrong, the Wiki or the source code?

[Reel.Deel]

Blankclip(pixel_type="YUV444PS", colors=[1,1,1])
Expr("x .5 -", scale_inputs="floatUV")

Works for me, I'm using 3.7.2 test12.

[LigH]

Check if you really use the AviSynth version you believe to use, and the color mode this feature may require...

[wonkey_monkey]

I figured it out. scale_inputs is case sensitive, which I didn't expect, and also the error message needs updating to include "floatUV" as one of the reported options (line 5958 of exprfilter.cpp)

[pinterf]

I figured it out. scale_inputs is case sensitive, which I didn't expect, and also the error message needs updating to include "floatUV" as one of the reported options (line 5958 of exprfilter.cpp)
yay. I have approximately zero motivation fixing it nowadays. My mind is all elsewhere. On my morning bike commute I dropped some tinned food at a humanitarian aid center in the central railways station for Ukrainan refugees.

[wonkey_monkey]

Sorry if you took it as a demand for immediate action - it wasn't - but I'm just reporting a bug...

[pinterf]

Sorry if you took it as a demand for immediate action - it wasn't - but I'm just reporting a bug...
No, no, nothing wrong with it, it's just difficult to express my feelings properly.

[pinterf]

I think parameter values (e.g. matrix names) are usually case insensitive. Names of frame properties however are case sensitive.

[wonkey_monkey]

No, no, nothing wrong with it, it's just difficult to express my feelings properly.

No problem. I find some of my feelings on the matter quite easy to express with four-letter words, but... forum rules and all that.

[pinterf]

Parameter value is now case insentive and the error message contains floatUV as well. Thanks for the report.

[SeregaDS]

yay. I have approximately zero motivation fixing it nowadays. My mind is all elsewhere. On my morning bike commute I dropped some tinned food at a humanitarian aid center in the central railways station for Ukrainan refugees.

Thank you from Ukraine!
p.s. sorry for offtop

[Dogway]

Many camcorders and video capable cameras now are using 16-255 that contain extra highlight information (superwhites) to be retreived later on processing. You can even display this range on a capable device (there are such projectors).

Recovering this post to ask about adding official "_ColorRange" mode 2 as a new range parameter. I had it in TransformsPack for some time but I wasn't sure whether it was a marginal case. I see it now in repeated places, also used in some HLG 10-bits. It has several names, SMPTE+, Camera Range or Extended Range, so whatever fits. Not for now but when you get back to AviSynth, take your time.

The allowed range changes in this case as it uses the Limited Range flag, for the higher bound it uses the data max (https://github.com/Dogway/Avisynth-Scripts/blob/9502f348ac29a8c0438b0eadf47ae4b2424a9522/ExTools.avsi#L7201) values. For example 64-1019 for 10-bit HLG.

[ryrynz]

No, no, nothing wrong with it, it's just difficult to express my feelings properly.

Except when it comes to doing something outside, your excitement is palpable.

[FranceBB]

Except when it comes to doing something outside, your excitement is palpable.

Ferenc Pinter, a skilled developer and an amazing human being for everything he does online and offline. ;)

[pinterf]

Perhaps our last chance is to talk to Zavulon and Geser and ask them to unify their force together behind the scenes to stop this one man show. (Characters from Night Watch/Day Watch book series - Ночной дозор/Дневной дозор - one of my favourites, re-read them many times.)

[pinterf]

Recovering this post to ask about adding official "_ColorRange" mode 2 as a new range parameter. I had it in TransformsPack for some time but I wasn't sure whether it was a marginal case. I see it now in repeated places, also used in some HLG 10-bits. It has several names, SMPTE+, Camera Range or Extended Range, so whatever fits. Not for now but when you get back to AviSynth, take your time.

The allowed range changes in this case as it uses the Limited Range flag, for the higher bound it uses the data max (https://github.com/Dogway/Avisynth-Scripts/blob/9502f348ac29a8c0438b0eadf47ae4b2424a9522/ExTools.avsi#L7201) values. For example 64-1019 for 10-bit HLG.
If supported, this parameter has some side effects as well; conversions must somehow support and act upon this value.
Or do you think this value must be treated simply as passthrough and being not forbidden?

Btw, sometimes I was missing another possible extension of this flag: when a limited chroma plane is extracted to a single Y clip. The plane then would be flagged as "Limited-Chroma".

[Dogway]

I was thinking only on acknowledging the index value as I don't have plans to work with it for the time being, so a passthrough I guess.
I searched for some officiality on papers but didn't find any, just some words on the fact that "super-white" is a thing. Here (https://www.itu.int/dms_pub/itu-r/opb/rep/R-REP-BT.2408-4-2021-PDF-E.pdf#[{"num":173,"gen":0},{"name":"XYZ"},54,770,0]).
This blog post from the BBC assumes the practice. Harsh vs. Normal (https://www.bbc.co.uk/rd/blog/2020-06-lut-format-conversion-hdr-video-production)
Also found this (https://www.nacinc.jp/wp-content/uploads/2014/12/PRM-4220_Brochure_JP.pdf) product brochure of a Dolby reference monitor where they acknowledge SMPTE+ as a valid range.

I recall also reading about mixed ranges for luma and chroma, not sure if that's what you refer to.

[FranceBB]

just some words on the fact that "super-white" is a thing. Here (https://www.itu.int/dms_pub/itu-r/opb/rep/R-REP-BT.2408-4-2021-PDF-E.pdf#[{"num":173,"gen":0},{"name":"XYZ"},54,770,0]).
This blog post from the BBC assumes the practice. Harsh vs. Normal (https://www.bbc.co.uk/rd/blog/2020-06-lut-format-conversion-hdr-video-production)


In this case this is indicated for signals routed through SDI cables and in that case we can have Super Whites and Super Blacks, but there's a catch: the last 4 bits of the full 10bit signal MUST NOT be used (same goes for the first 4 bits). Those packets are important 'cause we rely on timecodes to have everything synced with the central clock which is the very same generator inside the same location which connects each and every device connected (from VTRs to Hardware Encoders to Playout Ports etc). If a timecode "skip" is detected, there are all kind of alarms as the signal might be compromised etc. Also, without having this, a signal might be out of phase which is why every signal which is routed through SDI is ALWAYS referenced. (I mean, you can have things that are not, but nobody in his right mind would ever do that). Anyway, this is because those bits are reserved to the synchronization packets which compose the timing reference signal, which is why you cannot have a real Full Range signal carried through SDI cables, which is what is used across the world by literally every broadcaster. That being said, you almost always have Limited TV Range signals in 0.0-0.7V for luma which of course corresponds to 64-940 and -0.3V, 0.3V for chroma (https://i.imgur.com/HRLItw3.jpeg) (yes, chroma is "lower" in voltage than luma 'cause we go from -0.35V which correspond to 64 to +0.35V which corresponds to 960). In other words, even if you have a Full Range signals, 0–3 and 1020–1023 are never used and also the overwhelming majority of signals are Limited TV Range anyway and whenever there's something slightly out of range, that's almost always clipped out by some hardware to get a real Limited TV Range signal out to the user. In the screenshot I posted, we have Limited TV Range Luma with Limited TV Range Chroma + Overshooting which exceed the +0.35V (so outside of the 960) and you can see it in the FlatLumaChroma (on the right had side, chroma, in the middle there's luma).

Oh and by the way, this is actually the reason why Studio RGB exists, given that RGB is supposed to be always Full Range (but you can't work in Full Range in SDI) and this is why the LUTs I've got from several studios all work in Studio RGB (so RGB but in Limited TV Range).


I recall also reading about mixed ranges for luma and chroma, not sure if that's what you refer to.

I've never ever seen that, but hey, everything can happen eheheheh


By the way, the article from the BBC is surprisingly describing almost everything we've been doing in the past for things like football games.

[Dogway]

Ah that's good to know, I had been wondering for some time what happened to synchronization in Full Range signals... The papers don't state that "Video Data" range also applies to Full Range. This can come handy if/when I decide to encode a few things, the issue though is whether there is an intermediary conversion (well there is if I encode as YCbCr Full Range), the conversion to Full RGB is going to be flawed as the range equations assume the full signal range for Full Range flagged streams.

For example:
ConvertBits(10,fulls=false,fulld=true)
Saturates the full range.

I've never ever seen that, but hey, everything can happen eheheheh
I think the mixed plane range thing was for xvYCC extended gamut, but don't call me on that.

[FranceBB]

Ah that's good to know, I had been wondering for some time what happened to synchronization in Full Range signals... The papers don't state that "Video Data" range also applies to Full Range. This can come handy if/when I decide to encode a few things, the issue though is whether there is an intermediary conversion (well there is if I encode as YCbCr Full Range), the conversion to Full RGB is going to be flawed as the range equations assume the full signal range for Full Range flagged streams.

For example:
ConvertBits(10,fulls=false,fulld=true)
Saturates the full range.




I know, but I don't think we should change the way our conversions work only because of an SDI limit. On the other hand, this is prompting me to experiment a bit instead of trusting the proprietary Blackmagic and AJA hardware conversion to get it right, so I'm gonna try to change the card from Narrow Range to Full and see what happens and if it gets the conversion "right" for SDI (it probably does). :)

[DTL]

"the last 4 bits of the full 10bit signal MUST NOT be used (same goes for the first 4 bits). "

Not 4 bits but 4 code values of 10bit encoding. It is 2 bits.

The idea before strange float moving to 0..1.0 range in float encoding was very simple: Base levels encodings are 8bit and any higher is fractional refining inbetween 8bit codelevels. So in 8bit data paths (with possible SDI parts) sync (+ANC) reserved code values are 0 and 255. The 1,2,3 in 10bit are dangerous because if somwhere performed truncating 2 LSBs to 8bit it will result in 8bit-zero that is reserved codevalue (and may cause data parsing error). But if all processing in 10bit (or higher) - it still safe. In non-SDI workflows all codevalues from 0 to MAX may be used because there is no need to SDI sync and ANC data parsing from bitstream.

"a signal might be out of phase which is why every signal which is routed through SDI is ALWAYS referenced"

SDI is designed as self-describing bitstream format (that is sync codevalues sequencies are for) so can be re-phased to any local reference clock by 'frame sync' hardware units (it can parse incoming SDI bitstream without external reference syncs and write to internal frame buffer and read using external clock reference). For file-based workflows it is deprecated info because file based do not need special line and frame start-end detection codevalues or special reserved sequencies (and even in most cases do not keep line and frame blanking data with sync code values sequencies).

So keeping special reserved code values todays may be never need by anyone in full file-based workflows. If even some SDI transmitter still required to someone and being feed by 0..255/0..1023 file - it may simply clamp levels to 1..254 or 4..1019 in active lines codevalues because it knows it is frame data and not SDI service (or ANC) data (service data is placed in SDI bitstream outside time of sending active line data). It will be slight distortion but may be not any visible.

[FranceBB]

So keeping special reserved code values todays may be never need by anyone in full file-based workflows. If even some SDI transmitter still required to someone and being feed by 0..255/0..1023 file - it may simply clamp levels to 1..254 or 4..1019 in active lines codevalues because it knows it is frame data and not SDI service (or ANC) data (service data is placed in SDI bitstream outside time of sending active line data). It will be slight distortion but may be not any visible.

Yep, which is why I don't think we should change any of the current behaviour in avisynth for that matter, but yeah you're right. :)

[Dogway]

Great explanation, well that basically tells that full range for broadcast is a big nono. I guess "full range" is a concept that started to rise with full file-based workflows where these constraints don't apply.
By the way reading around on the papers for HDR they even encourage full range for PQ, in this space over/undershoots are minimal so no data is clipped.

[DTL]

"encourage full range for PQ, in this space over/undershoots are minimal so no data is clipped."

Full-range PQ may be used as poor-people HDR in 10bit-only limited workflows (and end-user distribution). Also for film-look tuned content without significant 'normal' over/undershoots. Mathematically PQ is the most awful non-linear transfer and most prone to additional over/under shoots and other distortions if filtered/processed in non-linear form. So 10bit full-range PQ encoding simply prefer to minimize tone distortions like banding and allow to have more 'frequency' distortions. Clipping of over/under shoots to full range integer will cause a bit less sharpness and may cause additional ringing distortions (depending on processing/displaying). But in HD/UHD it may be less critical. I hope as the hype around HDR fades away the full-range PQ 10bit transfer will be removed from general use (or the 12..12+ bits forms with 'standard moving pictures' narrow range encoding will be used or float).

[Dogway]

About the Data Range in Full Range signals, extracted from ITU BT.2100:
Note 9b – Some digital image interfaces reserve digital values, e.g. for timing information, such that the permitted video range of these interfaces is narrower than the video range of the full-range signal. The mapping from full-range images to these interfaces is application-specific.

From BT.2408:
The full range representation is useful for PQ signals and provides an incremental advantage against visibility of banding/contouring and for processing. Because the range of PQ is so large, it is rare for content to contain pixel values near the extremes of the range. Therefore, over-shoots and under-shoots are unlikely to be clipped.

Full-range PQ may be used as poor-people HDR in 10bit-only limited workflows (and end-user distribution). Also for film-look tuned content without significant 'normal' over/undershoots. Mathematically PQ is the most awful non-linear transfer and most prone to additional over/under shoots and other distortions if filtered/processed in non-linear form. So 10bit full-range PQ encoding simply prefer to minimize tone distortions like banding and allow to have more 'frequency' distortions. Clipping of over/under shoots to full range integer will cause a bit less sharpness and may cause additional ringing distortions (depending on processing/displaying). But in HD/UHD it may be less critical. I hope as the hype around HDR fades away the full-range PQ 10bit transfer will be removed from general use (or the 12..12+ bits forms with 'standard moving pictures' narrow range encoding will be used or float).

I forgot to say that I would be using 12-bit which is the recommended bitdepth for minimizing banding, although 12-bit profiles are not existant in x265. I think this may be caused becaused 12-bit is composed of 10-bit + 2-bit dual layer in profile 7.
I don't even have an HDR TV, but I directly see the benefits in details rendition in dark areas after tonemapping.

I don't know why you affirm that PQ is the worst space for over/undershoots, log space which shares the same principles (highlight rolloff) is used in compositing explicitly to avoid over/undershoots, you can read more about it here (https://library.imageworks.com/pdfs/imageworks-library-cinematic_color.pdf#page=37).

[FranceBB]

Because the range of PQ is so large, it is rare for content to contain pixel values near the extremes of the range. Therefore, over-shoots and under-shoots are unlikely to be clipped.

That is correct. Being it a totally logarithmic curve like Slog, Clog, LogC etc, it starts high, so it appears as if it sits in the middle, as shown by the waveform monitor: Img1 (https://i.imgur.com/teEiuao.png) - Img2 (https://i.imgur.com/fP2SS3N.png) - Img3 (https://i.imgur.com/TjRoAm9.png)

so it's highly unlikely values will ever be clipped out.


log space which shares the same principles (highlight rolloff) is used in compositing explicitly to avoid over/undershoots

yep.



I forgot to say that I would be using 12-bit which is the recommended bitdepth for minimizing banding, although 12-bit profiles are not existant in x265.

Uh? there's --profile main12 along with main422-12 and main444-12 afaik (untested, though).


I think this may be caused because 12-bit is composed of 10-bit + 2-bit dual layer in profile 7.


Yeah, you're right, that's the thing, the standard is indeed 10bit only and the only way you have to get 12bit would be via Dolby Vision with dual layer, 'cause if you encode directly to 12bit I have no idea how many hardware decoders are out there, capable of decoding it correctly, unfortunately :(
And it's a shame, really, 'cause most masterfiles are 12bit MJPEG2000 4:4:4 (and rarely some of them DNXHQX 4:2:2 or 4:4:4 12bit), so it would make sense for the public to get something like that rather than a dithered down 10bit version. On the other hand, as long as you're gonna have 1000 nits in PQ, you're gonna be fine with 10bit. The only reason why sometimes in productions you can find pseudo-full-range PQ is that they're trying to take advantage of the extra head given by not having to stay within the narrow range (Limited TV Range) limit to overcome the 10bit limitation and have more stops and therefore more nits. This is a bit like what Sony did for HLG by doing it in 8bit full range, hoping that using all the 255 values of the full range would overcome the limitation and allow people to shoot HLG with 8bit based cameras. They somehow succeeded in the sense that people were no longer stuck on BT709 SDR 100 nits, but the problem was that you couldn't have more than 699 nits (I don't remember how many stops) with full range 8bit HLG, 'cause you needed 10bit to get to 1000 as you didn't physically have enough values to correctly represent the signal. So, in a nutshell, history is repeating itself and I suspect we're gonna have everything in 12bit Narrow Range (Limited TV Range) in not so many years into the future. In SDI this is achieved with something called "multi-link" connection in which you're basically making use of multiple SDI cables working together to create the final output and I think you have MSB and LSB there too but I'm not familiar at all 'cause here in Italy we're stuck with 10bit (although I think the UK side works in 12bit, but I don't know their workflows, so...) ;)

[Selur]

Uh? there's --profile main12 along with main422-12 and main444-12 afaik (untested, though).
I agree 12bit x265 encoding works fine here,...

[Dogway]

Yep sorry, I've been out of touch on encoders for some time, good news then for 12-bit on x265. I think I got it mixed with x265 not supporting IPTPQc2 matrix, not sure if that's possible. Anyway, just waiting for LCEVC support in x265.

[DTL]

"From BT.2408:
Quote:
The full range representation is useful for PQ signals and provides an incremental advantage against visibility of banding/contouring and for processing. Because the range of PQ is so large, it is rare for content to contain pixel values near the extremes of the range. Therefore, over-shoots and under-shoots are unlikely to be clipped."

1. The typical Avisynth processing workflow is not linear domain data but transfer-function domain data. So if applying any 'sharpening' in this domain - it still easy to get over/under shoots below zero (black) and over max system white.

2. I think most of real content have real zero black level (video black and numerical zero in scene light). Not some 'HDR-dark level' converted into far from system black levels code levels in PQ transfer domain. So some sharpening applied to the film-look 'makeup' will cause below-black level undershoots. If using 'full range' with system black code level 0 and resticted negative numbers - it will cause clipping of below-black data undershoots. Also any artificial generated content like 3D render or CG must also have non-zero HDR blacks to follow this idea of non-clipping undershoots in full-range' I not sure if HDR content providers follow this idea. Also if even every HDR content provider uses non-zero HDR darks to save from clipping undershoots below zero code level - it actually creates 'common use HDR non-zero dark that it assumed as HDR black' designed to save from clipping undershoots below zero code level. And this practice will be equal to simply using non-zero code level to encode zero-light level and have footroom for data undershoots.

Due to limitations of physical optics - it never can create zero brightness level at sensor from non-totally dark scene (if scene even do have zero emitting areas like special light traps). So real optical cameras have adjustments to compensate lens glare/flare and increase contrast to 'infinity' adjusting master black to put some scene non-zero black to output black code levels. No any real physical camera can shoot 'bipolar HDR' with increased range below 'standard' and above 'standard'. All 'HDR' is about less clipping of highlights, not about better capturing of low light real scene parts. So the low scene data levels are totally artificial product of glare/flare compensation and manual master black and other camera adjustments (like black stretch and tons of other non-real scene light tricks of camera manufacturers). The real dynamic range of typical lens is about 500 (may be some thousands for outstanding lens) for 'high key scene' and it fit in SDR 8bit good.

The footroom for SDR undershoots below black is about 14/253 - about 5% of total data range and I think it was selected after long debates of the great engineers of the past to make things not very bad. It eats a lot of expensive range but may be add to the quality. So total disabling of undershoots for HDR looks like not good idea.

Also most 'video-look' makeup footage do have significant over/undershoots and samples values may travel below/under system black code level.

" stuck on BT709 SDR 100 nits"

SDR may have recommended something around 100 nits for indoor setups but it is about colour imaging without strict real physical brightness mapping. So SDR content may be viewed greatly at sunlight-viewable outdoor TVs having 2000..4000+ nits many years before the 'HDR hype' was started. Also good SDR display for medium bright room also go far above 100 nits in nominal white. Also to view full SDR of about 1000..3000 in good colours from the darkest areas it is required 1000+nits for nominal white because eye colours sensitivity start degrade below 10 nits and already not great at 1 nit and colours fades out in saturation at about 0.01 nit and below. So the good part of HDR physically mapped PQ range is black and white only for viewer.

[qyot27]

AviSynth+ 3.7.2 has been released. (https://github.com/AviSynth/AviSynthPlus/releases/tag/v3.7.2)

32-bit MSVC builds were crashing if FFmpeg tried to access frame properties (32-bit GCC builds were unaffected, and so were non-Windows builds). This got fixed, which warranted making a release so users would have a real, working build.

C interface Win32 access: fix issue by adding V8 interface function…
… names to avisynth.def

or else names are decorated (Issue #276)
e.g. DLL published _avs_get_frame_props_ro@8 instead of avs_get_frame_props_ro
ShowRed/Green/Blue/Alpha/Y/U/V: addition to earlier fixes:
When clips are planar and both source and destination format have alpha plane,
then it will be copied instead of filled with 255d.
Additional checking is done for alpha plane size when ShowU/V, because when
source is subsampled the original alpha plane cannot be copied (larger).
ConvertBits:
Does not get frame 0 in constructor for frame properties if 'fulls' is directly specified. (magiblot)
May make script initialization much quicker (Issue #275)
#275
Trim, AudioTrim: bool 'cache' (default true) parameter.
Workaround for Issue #274, lower memory consumption but may be slower.
Benefits heavily depend on how trimmed clips are used later.
Expr: scale_inputs to case insensitive and add floatUV to error message as an allowed value.
propCopy: able to specify that the property list is negative.
bool "exclude" = false # default: "props" is positive list

propCopy(org,true,props=["_Matrix", "_ColorRange"], exclude=false) # merge only two properties
propCopy(org,true,props=["_Matrix", "_ColorRange"], exclude=true) # merge all, except listed ones
propCopy(org,props=["_Matrix", "_ColorRange"]) # erase all then copy only selected
propCopy(org,props=["_Matrix", "_ColorRange"], exclude = true) # erase all, then copy all, except listed ones

Version()
New optional parameters

int length, int width, int height, string pixel_type, clip c

Version clip defaults:
length=240, width = -1, height = -1 (-1: automatically sized to fit for font size 24)
pixel_type = "RGB24"

When 'clip' (a format template) is specified then pixel_type, length,
fps data, width and height are defined from it.
If any additional 'length', 'width', 'height', 'pixel_type' parameter is given, it overrides defaults.
When width and height is given and is <= 0 then it is treated as 'automatic'

Covers feature request Issue #261

BlankClip: allow 'colors' with array size more than the number of actual planes.
If an array is larger, further values are simply ignored.
BlankClip, AddBorders, LetterBox: no A=0 check for non-YUVA
Fade filter family new parameters
int 'color_yuv'
array of float 'colors'
similar to BlankClip
MergeRGB, MergeARGB
add MergeARGB parameter "pixel_type", similar to MergeRGB
accept pixel_type other than packed RGB formats, plus a special one is "rgb"
output format is planar rgb(a) (MergeRGB/MergeARGB) when
pixel_type = "rgb" or
pixel_type is empty and
either input is planar RGB
either input is different from 8 or 16 bits (no packed RGB formats there)
pixel_type is explicitely set to a valid planar rgb constant e.g. "RGBP10"
Accept planar RGB clip in place of input clips and the appropriate color plane is copied from them
Fill alpha channel with zero when MergeRGB output pixel_type format is specified to have an alpha plane
frame property source is the R clip; _Matrix and _ChromaLocation are removed if R is not an RGB clip
PropDelete: accept a non-empty array string as list of property names to remove
Parameter is not optional, and has no name. It can be either a string (as before) or an array of strings
propDelete("_Matrix") # old syntax, still accepted
propDelete(["_Matrix", "_ColorRange"])
PropCopy: new string parameter "props" as list of property names to remove
"props": a non-empty array of strings

old syntax, still accepted:
propCopy(org,true) # merge from all org's properties
propCopy(org,false) # erase all then copy all org's properties (exact copy)
new syntax
propCopy(org,true,props=["_Matrix", "_ColorRange"]) # merge
propCopy(org,props=["_Matrix", "_ColorRange"]) # erase all then copy only selected
Histogram Levels: stop using shades of grey on top of bars.
Histogram Levels: use bar color 255 for RGB instead of Y's 235. (and scaled eqivivalents)
Fix: Histogram "Levels": prevent crash when factor=0.0
Fix: Histogram "Levels": fix regression incorrect "factor" applied for U/V part drawing when format was subsampled (non-444)
Regression since 20160916 r2666 (commit 986e275)
Histogram "Audiolevels" and StereoOverlay to deny planar RGB
Histogram "Luma": support 10-16 and 32 bits
Histogram: give parameter name "factor" and type 'float' for Histogram's unnamed optional parameter used in "Level" mode.
Other modes just ignore this parameter if given.
Fix: Histogram "color" may crash on certain dimensions for subsampled formats.
Regression since 20180301 r2632.
Fix: Histogram "color" and "color2" mode check and give error on Planar RGB
Fix: missing Histogram "color2" CCIR rectangle top and bottom line (black on black)
Regression since 3.6.2-test1 (commit 1fc82f0)
Fix: Compare to support 10-14 bits
was: factor was always using 65535 (2^16-1) instead of (2^bit depth 1)
was: 16 bit luma/rgb color values were used for drawing graph
Fix: Compare
'channels' parameter default to "Y" when input is greyscale;
instead of "YUV" which was giving error because of U and V does not exist for this format.
ShowRed/Green/Blue/Alpha/Y/U/V
support YUY2 input
support YV411 output
(not changed: ShowU/ShowV may give error for 420, 422 or 411 format outputs when clip dimensions are
not eligible for a given output subsampling (check for appropriate mod2 or mod4 width or height)
Copy alpha from source when target is alpha-capable
Fill alpha with maximum pixel value when target is alpha-capable but source ha no alpha component
Delete _Matrix and _ChromaLocation frame properties when needed.
More consistent behaviour for YUV and planar RGB sources.

Default pixel_type is adaptive. If none or empty ("") is given for pixel_type then target format is
YUV444 when source is Y, YUV or YUVA
RGB32/64 (packed RGB) when source is RGB24/32/48/64
RGBP (planar RGB) when source is RGBP or RGBAP

When 'rgb' is given for pixel_type then then target format is

RGB32/64 (packed) when source is RGB24/32/48/64 old, compatible way
RGB planar when source is planar RGB(A) or YUV(A) or Y changed from rgb32/64 because all bit depth must be supported

When 'yuv' is given (new option!) for pixel_type then then target format is

YUV444 for all sources

Also there is a new option when pixel_type is still not exact, and is given w/o bit depth.
pixel_type which describes the format without bit depth is automatically extended to a valid video string constant:

y, yuv420, yuv422, yuv444, yuva420, yuva422, yuva444, rgbp, rgbap

Examples:

32 bit video and pixel_type 'y' will result in "Y32"
16 bit video and pixel_type 'yuv444' will result in "YUV444P16"
8 bit video and pixel_type 'rgbap' will result in "RGBAP8"

Fix #263. Escaping double-quotes results in error
Allow top_left (2) and bottom_left (4) chroma placements for 422 in colorspace conversions, they act as "left" (0, "mpeg2")
in order not to give error with video sources which have _ChromaLocation set to other than "mpeg2"
See https://trac.ffmpeg.org/ticket/9598#comment:5
Fix: Expr LUT operation Access Violation on x86 + AVX2 due to an unaligned internal buffer (<32 bytes)
Fix: Chroma full scale as ITU Rec H.273 (e.g +/-127.5 and not +/-127) in internal converters, ColorYUV and Histogram
Fix #257: regression in 3.7.1: GreyScale to not convert to limited range when input is RGB. Regression in 3.7.1
Accepts only matrix names of limited range as it is put in the documentation.
Fix #256: ColorYUV(analyse=true) to not set _ColorRange property to "full" if input has no such
property and range cannot be 100% sure established. In general: when no _ColorRange for input and
no parameter which would rely on a supposed default (such as full range for gamma), then an
output frame property is not added.
When no _ColorRange for input and no other parameters to hint color range then
gamma<>0 sets full range
opt="coring" sets limited range
otherwise no _ColorRange for output would be set
Overlay (#255): "blend": using accurate formula using float calculation. 8 bit basic case is slower now when opacity=1.0.
Higher bit depths and opacity<1.0 cases are quicker.
Mask processing suffered from inaccuracy. For speed reasons mask value 0 to 255 were handled
as mask/256 instead of mask/255. Since with such calculation maximum value was not the expected 1.0 but rather 255/256 (0.996)
this case was specially treated as 1.0 to give Overlay proper results at least the the two extremes.
But for example applying mask=129 to pixel=255 resulted in result_pixel=128 instead of 129. This was valid on higher bit depths as well.
Note 3.7.2 Test2 has a regression of broken maskless mode for 0<opacity<1 which was fixed in 3.7.2 test 3
Fix: Attempt to resolve deadlock when an Eval'd (Prefetch inside) Clip result is
used in Invoke which calls a filter with GetFrame in its constructor.
(AvsPMod use case which Invokes frame prop read / ConvertToRGB32 after having the AVS script evaluated)
Remark: problem emerged in 3.7.1test22 which is trying to read frame properties of the 0th frame in its constructor.
A similar deadlock situation was already fixed earlier in Neo branch and had been backported but it did not cover this use case.
Note: Prefetch(1) case was fixed in 3.7.2 Test3

[FranceBB]

Nice one, Stephen. I'm gonna update as soon as I get to work ;)

[qyot27]

...and in addition to the builds that went up yesterday, we now have both an installer and a filesonly package for macOS 11 or higher on ARM64/Apple Silicon. I'm still unsure of exactly whether we should refer to it by architecture (ARM64), marketing name (Apple Silicon), or the name of the chip (M1, which the packages for 3.7.2 do use just for brevity, but it won't make much sense once M2, M3, etc. show up).

[Ceppo]

I have a simple request, if it is hard to make no reason to worry about it.

Is possible to have avisynth+ detect 1(...) and 0 as true and false as c++ does? I'm lazy, and I find it more readable.

[wonkey_monkey]

You can use "!=0" to convert a number to boolean in a way that agrees with C++ with only three extra characters (plus possibly a space) ;)

If you're really lazy, I might be able to alter my script modification plugin to do this automatically for you. You'll have to use my Avisynth+ DLL though, and your scripts won't be compatible for other people unless they're also using it.

[Ceppo]

I didn't think about it :eek:, thanks! :cool:

[Ceppo]

If you're really lazy, I might be able to alter my script modification plugin to do this automatically for you. You'll have to use my Avisynth+ DLL though, and your scripts won't be compatible for other people unless they're also using it.
My original intent was to pass a boolean parameter writing 1 or 0 instead of true/false, I don't need it for writing scripts, just to test different parameters... so I'll gladly accept your proposal :D

[wonkey_monkey]

Not quite sure what you're trying to do then, but in any case I was thinking only of tertiary conditionals, so what I had in mind wouldn't work after all.

[Ceppo]

I was in multi tasking mode, so maybe my idea didn't make sense, probably it doesn't.

What I wanted to do was for example,
CTelecine(sse=true) -> CTelecine(sse=1)
Doing this would make my semantic bug hunting less painful. But to do that avisynth would need to convert int to bool if the parameter is bool... so well I don't know if this is legit.

[Selur]

Using latest AviSynth+ 3.7.2

When I use:
ClearAutoloadDirs()
LoadPlugin("I:\INNOIN~1\64bit\Avisynth\AVISYN~1\LSMASHSource.dll")
LoadPlugin("I:\INNOIN~1\64bit\Avisynth\AVISYN~1\MosquitoNR.dll")
LWLibavVideoSource("G:\TESTCL~1\test.avi",cache=false,dr=true,format="YUV420P8", prefer_hw=0)
MosquitoNR()
ConvertToRGB32(matrix="Rec601")
return last
my code crashes when loading the the file:
AVS_linkage = m_env->GetAVSLinkage();
const char* infile = m_currentInput.toLocal8Bit(); //convert input name to char*
std::cout << "Importing " << infile << std::endl;
AVSValue arg(infile);
m_res = m_env->Invoke("Import", AVSValue(&arg, 1)); // <- here it dies
see: https://github.com/Selur/avsViewer/blob/441500fe4f46ece0e48542a61daec95eb019ff3b/avsViewer.cpp#L142
When i use
ClearAutoloadDirs()
LoadPlugin("I:\INNOIN~1\64bit\Avisynth\AVISYN~1\LSMASHSource.dll")
LoadPlugin("I:\INNOIN~1\64bit\Avisynth\AVISYN~1\MosquitoNR.dll")
LWLibavVideoSource("G:\TESTCL~1\test.avi",cache=false,dr=true,format="YUV420P8", prefer_hw=0)
MosquitoNR()
#ConvertToRGB32(matrix="Rec601")
return last
the script is loaded, but when I later invoke ConvertToRGB32 it crashes, there.
see: https://github.com/Selur/avsViewer/blob/441500fe4f46ece0e48542a61daec95eb019ff3b/avsViewer.cpp#L875

Same happens when using just 'version()' as script.

With previous AviSynth+ versions I had the issue the other way around.
32bit failed, 64bit worked (see: https://forum.doom9.org/showthread.php?t=183787)

-> any idea?

Cu Selur