AviSynth+ thread Vol.2

[tormento]

tormento
I sent you a message.
Thank you. I think you should share with anyone here.

[tormento]

Sorry if the question has already been discussed, I couldn't find the answer searching.

I stumbled across Resizing with Colorspace Correction (https://usage.imagemagick.org/resize/#resize_colorspace) and I asked myself if normal resizing that we do with AVS+ internals, jpsdr's MT resizers or avsresize are correct.

Should I convert everything to (linear) RGB before resizing or it is implemented already in the most often used filters?

Originals (https://visibleearth.nasa.gov/images/55167/earths-city-lights)

https://usage.imagemagick.org/resize/earth_lights_direct.png
Not gamma aware resizing

https://usage.imagemagick.org/resize/earth_lights_colorspace.png
Gamma aware resizing

[Selur]

iirc ReSampleHQ (http://avisynth.nl/index.php/ResampleHQ) does it, but I don't think other resizers do.

[LigH]

As far as I remember I would agree with Selur.

I would not generally recommend converting everything to RGB, it will be slightly lossy especially when all YUV components have 8 bit depth only (except YCoCg/YCgCo which uses a symmetrically shifted color space, relative to the RGB cube).

[tormento]

I would not generally recommend converting everything to RGB, it will be slightly lossy especially when all YUV components have 8 bit depth only (except YCoCg/YCgCo which uses a symmetrically shifted color space, relative to the RGB cube).
AFAIK VapourSynth works in RGB by design.

So, every time VS opens a YUV source, it adds a lossy step to the operation?

[tormento]

ReSampleHQ does it, but I don't think other resizers do.
It's a pity that it hasn't been updated since ages, above all if the other resizers don't take gamma in account.

We need a reply from jpsdr for ResampleMT and from stvg for AVSresize.

[LigH]

I don't use VapourSynth. I only reply with my experience in AviSynth. Matching this thread's topic.

[FranceBB]

iirc ReSampleHQ (http://avisynth.nl/index.php/ResampleHQ) does it, but I don't think other resizers do.

Correct.


original=ImageSource("D:\earth_lights_lrg.jpg")

imagemagic_direct=ImageSource("D:\earth_lights_direct.png").Subtitle("ImageMagic Default")
imagemagic_RGB=ImageSource("D:\earth_lights_colorspace.png").Subtitle("ImageMagic Default RGB")

downscale_resample=ResampleHQ(original, 500, 250).Subtitle("Avisynth Resample HQ")

downscale_point=PointResize(original, 500, 250).Subtitle("Avisynth PointResize")
downscale_gauss=GaussResize(original, 500, 250).Subtitle("Avisynth GaussResize")
downscale_bilinear=BilinearResize(original, 500, 250).Subtitle("Avisynth BilinearResize")
downscale_bicubic=BicubicResize(original, 500, 250).Subtitle("Avisynth BicubicResize")
downscale_sinc=SincResize(original, 500, 250).Subtitle("Avisynth SincResize")
downscale_lanczos=LanczosResize(original, 500, 250).Subtitle("Avisynth LanczosResize")
downscale_blackman=BlackmanResize(original, 500, 250).Subtitle("Avisynth BlackmanResize")
downscale_sin=SinPowerResize(original, 500, 250).Subtitle("Avisynth SinPowerResize")
downscale_spline=Spline64Resize(original, 500, 250).Subtitle("Avisynth Spline64Resize")

v1=StackHorizontal(imagemagic_direct, imagemagic_RGB, downscale_resample)
v2=StackHorizontal(downscale_point, downscale_gauss, downscale_bilinear)
v3=StackHorizontal(downscale_bicubic, downscale_sinc, downscale_lanczos)
v4=StackHorizontal(downscale_blackman, downscale_sin, downscale_spline)

StackVertical(v1, v2, v3, v4)


https://i.postimg.cc/nhwF0434/resize-test.png

[poisondeathray]

I asked myself if normal resizing that we do with AVS+ internals, jpsdr's MT resizers or avsresize are correct.

Not by default, but avsresize and fmtc can perform linear scaling - you linearize the transfer, scale, then readjust the transfer back


Should I convert everything to (linear) RGB before resizing or it is implemented already in the most often used filters?


Discussed in other thread (many pages) . Technically linear is more correct for everything, all filters, all math operations, not just scaling. 1+1 =2 . There is no baked in gamma error in all math.

But basically for downscaling it can be beneficial for small bright areas - that's where general public can see it the most .




I would not generally recommend converting everything to RGB, it will be slightly lossy especially when all YUV components have 8 bit depth only (except YCoCg/YCgCo which uses a symmetrically shifted color space, relative to the RGB cube).

I agree in general, but gamma linearization is usually done in float, so it can be RGB<=>YUV can be lossless when done properly.

Also, "gamma" is a RGB concept. If the topic is "gamma scaling error" when scaling (or any filter for that matter), it should be done in float RGB. Actually every high end professional software already does linear gamma scaling and works in float RGB - eg. resolve, nukex




AFAIK VapourSynth works in RGB by design.

No


So, every time VS opens a YUV source, it adds a lossy step to the operation?
No



It's a pity that it hasn't been updated since ages, above all if the other resizers don't take gamma in account.

We need a reply from jpsdr for ResampleMT and from stvg for AVSresize.
Not sure about ResampleMT, but as above, AVSResize and FMTC can do linear scaling, but it's not by default

[tormento]

Not by default, but avsresize and fmtc can perform linear scaling - you linearize the transfer, scale, then readjust the transfer back

[…]

Not sure about ResampleMT, but as above, AVSResize and FMTC can do linear scaling, but it's not by default
Could you be so kind to show me a couple of examples of this back and forth for both?

[poisondeathray]

Could you be so kind to show me a couple of examples of this back and forth for both?


It's just the transfer argument to linear. You are "linearizing the gamma". So if you were starting with a RGB web image, in srgb



#assume starting with 8bit sRGB web image

#srgb to linear in 32bit float
z_ConvertFormat(pixel_type="RGBPS", colorspace_op="rgb:srgb:709:f=>rgb:linear:709:f")

#resize step in current pixel type, which is RGBPS
z_ConvertFormat(width, height, resample_filter="spline36", colorspace_op="rgb:linear:709:f=>rgb:linear:709:f")

#transfer from linear back to srgb (8bit srgb in this case)
z_ConvertFormat(pixel_type="RGBP", colorspace_op="rgb:linear:709:f=>rgb:srgb:709:f")



If it was some video source, you 'd change the appropriate parameters such as matrix, pixel type etc...

[Boulder]

There's a couple of linear light downscaling examples at the end of the Avsresize wiki page.
http://avisynth.nl/index.php/Avsresize

[DTL]

Sorry if the question has already been discussed, I couldn't find the answer searching.

I asked myself if normal resizing that we do with AVS+ internals, jpsdr's MT resizers or avsresize are correct.

Should I convert everything to (linear) RGB before resizing or it is implemented already in the most often used filters?



It is really question about digital static imaging where RGB or YUV 4:4:4 (do not know what for if they can keep RGB) is possible. Even old/classic JPEG can store and unpack 4:4:4.

But with digital moving pictures general public world you mostly live with old poor and ugly 4:2:0 and can not benefit from resize in linear domain any significantly. Attempt to release such conditioned footage may cause more distortions at displaying because many displays perform scaling in System Transfer domain. Scaling in linear is more expensive and was NOT put to industry standards for digital displays. With digital-analog workflows and CRT displays it may be close to not possible at all because linear conversion was _after_ DAC and inside CRT device physically. This mean ADC and DAC work in System Transfer domain upscale mode. And this (partially and silently) was inherited to 'fully digital' seasons at the market. Also broadcast waveform monitors (Tektronix and Leader rasterizers and (all ?) CRT based oscilloscopes) make upscale only in System Transfer domain and do not have switch to make scale in linear domain. All this make a great pressure to continue use conditioning for System Transfer domain instead of moving to nice and lower distorted RGB with linear domain scaling.

There may be possibly more letters about how it is happen in this civilization. But later.

What is really important about AVS+ core close to this question:

Conversions of YUV 4:2:x to RGB 4:4:4 may be performed in different ways :

1. Current way in AVS+ core (and may be plugins ?) is upscale UVs to Y size (YUV 4:4:4) and perform dematrix to RGB.

This is not perfect and can not work well with that resize in linear idea because of a chicken and egg problem: To make resize in linear RGB we need RGB to perform dematrix and to get RGB in System Transfer we need to resize YUVs first. So it is already not possible to do perfectly.
I think it is not possible to convert Transfer in 4:2:0 form of digital data without resize (need to check this better).

This is also subject to add a Note to Resize documentation how ConvertToRGB/444 is working internally when upconverting from 4:2:0.

2. Second way to know and test is downsample Y to UV size and make dematrix of RGB in UV size. This will make half banded (H or H+V) RGB. This RGB can be converted in linear transfer without resize. Also we have full-band Y (in System Transfer domain). We can apply linear transfer conversion to it. It is not perfect step but close to nothing more to do with old ugly subsampled chroma formats. Y is close to G channel in properties (spectrum ?).
And can do attempt to transfer fine details from it (using sort of DPIDraw() or some other way like high band frequencies extract from Y and add to RGB half-banded ?) into RGB to get better result.

The sad issue of Transfer conversion from/to linear is frequency spectrum changes and this distort conditioning against Gibbs ringing (and even aliasing) created by production downsizers. This mean display (processing) upsizing must be performed in Conditioning domain and it must be tagged in metadata for receiver. But this tagging is not exist yet even. Nor in any industry standards. But if Conditioning domain is linear RGB - the conversion to subsampled YUV also adds distortions and degrage the total idea to add quality by production in linear RGB. To fix this - the distrubution must be also in RGB. And this is not true for public distribution formats.

Yes - pro NLEs may do scale in linear RGB. But they also can do data exchange in RGB or YUV 4:4:4. And these production formats do not reach poor endusers so this knowledge is partially useless for poor home endusers.

[tormento]

Thanks to everybody for their answers.

Conversions of YUV 4:2:x to RGB 4:4:4 may be performed in different ways :

1. Current way in AVS+ core (and may be plugins ?) is upscale UVs to Y size (YUV 4:4:4) and perform dematrix to RGB.

So, at the end of the day, is it worth to convert a 4:2:0 AVC or HEVC video to RGB to resize it? I am particularly worried about descaling procedure, where we have the inversion of the kernel matrix and the operation needs to be as precise as possible.

If positive, being descale existing only as jpsdr plugin, not knowing how they work internally, what is the least destructive way to upconvert to RGB, descale and downconvert to 420 again?

When the native resolution is half of the original one, we have an easy way to do it, i.e. descale Y plane only and work in 444. But when the native resolution is a random one?

[Emulgator]

But when the native resolution is a random one?
One would have to run FFT on source, find the ringing patterns, guess the source resolution and the resizer kernel used
then behind that maybe guess the sensor pattern (?!) and that resizer kernel used.
Then reverse those resizer kernels in reverse order in their respective spaces.
While doing so, correct old faults that had been incurred at that stage only.
So no artistic or neutralising grading, only Y/Chroma shifts that came in at that stage: Camera AGC, broadcast phase shifts, tape recording oversaturations etc.)
Then start anew, and this time grading/resizing/compositing as RGB in 32bit float in linear light please, only finally giving in to the requirement of distribution as YUV, again.
Patience & Persistence.

[tormento]

One would have to run FFT on source
I know how to find native ;)

My question was about a good strategy to descale converting to RGB and lose as tiny information as possible.

[DTL]

So, at the end of the day, is it worth to convert a 4:2:0 AVC or HEVC video to RGB to resize it? I am particularly worried about descaling procedure, where we have the inversion of the kernel matrix and the operation needs to be as precise as possible.

If positive, being descale existing only as jpsdr plugin, not knowing how they work internally, what is the least destructive way to upconvert to RGB, descale and downconvert to 420 again?


To keep most of quality you simply need to keep RGB after you somehow create it from input sources. You can only go to YUV 4:4:4 if compression require it. Simply never go back to the ugly and lossy chroma subsampled 4:2:x domain.

[jpsdr]

If positive, being descale existing only as jpsdr plugin, not knowing how they work internally,
How they work is described in the first post of my Internaly multi-threaded desampling functions thread.

[tormento]

How they work is described in the first post of my Internaly multi-threaded desampling functions thread.
I see no reference to how color spaces are internally treated.

[DTL]

I think they are same as AVS(+) internal resizers - completely separated planes resize without changing colour space.

[FranceBB]

Yep, the results from the MT version are pretty much like for like with the internal ones in the core. I remember testing them to see the differences a while back and then extending the test to the different CPU assemblies to catch any discrepancy there too. If I remember correctly, DTL did the same and ran a similar batch of tests. Anyway the point is that it's actually intended to be like that.

[tormento]

I was looking at avslibplacebo documentation (http://www.avisynth.nl/index.php/Avslibplacebo) and, for libplacebo_Resample, I've found the parameters

int trc = 1
The colorspace's transfer function (gamma / EOTF) to use for linearizing.

sigmoidize, linearize
Whether to linearize/sigmoidize before scaling.
Only relevant for RGB formats.
When sigmodizing, linearize should be true
Default: True.

sigmoid_center
The center (bias) of the sigmoid curve.
Must be between 0.0 and 1.0.
Default: 0.75.

sigmoid_slope
The slope (steepness) of the sigmoid curve.
Must be between 1.0 and 20.0.
Default: 6.5.

I just need to understand if that parameter means that it automatically converts to linear RGB when resampling or not. Any idea?

Whati is sigmoidization?

The default value for trc is BT_1886 (ITU-R Rec. BT.1886 (CRT emulation + OOTF). Does it fit for standard AVC BT.709 material or is it better to use Gamma values? In the latter case, which one is a good "default" one for BD mastered content?

[DTL]

I do not know any industry standards on spectrum shaping (including production resize). The general practic - you feed your master footage to master control monitor and adjust resizers (sharpeners/softerens/etc) to make image look/make-up you like. Simply work as program director. For each program different look/make-up may be used in fine details of the image. In medium and large size details generally all types of resizers make equal result.

" just need to understand if that parameter means that it automatically converts to linear RGB when resampling or not. Any idea?"

If it says
sigmoidize, linearize
Default: True.

it looks both convert to linear domain and to sigmoid domain are enabled by default.

"Whati is sigmoidization?"

Additional non-linear transfer domain. Created by some developers in the past for image resizing.

Try to look into imagemagic documentation.
AI Overview
ImageMagick offers a technique called "sigmoidal contrast" that can be applied before or during resizing to improve the visual quality, particularly for enlarging images or when dealing with certain types of content like line drawings or CG art. This technique applies a sigmoidal curve to the image's colorspace, effectively enhancing contrast in mid-tones while compressing highlights and shadows.

Applying Sigmoidal Contrast in ImageMagick:
The primary option for applying sigmoidal contrast is -sigmoidal-contrast <geometry>, which is used with the convert command. The geometry argument takes two values:
factor: This controls the amount of contrast enhancement. Higher values result in more pronounced contrast.
midpoint%: This specifies the point along the color range (from 0% to 100%) where the most significant contrast enhancement occurs. For example, 50% would center the enhancement in the mid-tones.

May be look as S-shaped transfer function.

[LigH]

Wikipedia: Sigmoid function (https://en.wikipedia.org/wiki/Sigmoid_function)

Helpful to limit infinite input ranges to finite output ranges. Also known as "soft clipping" in audio tasks.

[real.finder]

I have some questions about MT in avs+

is MT_SERIALIZABLE single thread? what cpu core will be used if there are more than one MT_SERIALIZABLE filter in the encoding script? if it always CPU 0 (first core) then will it better if it switch core (thread) for every filter that MT_SERIALIZABLE?

also IIRC in old avs mt
if we have for example:-
source call (MT_SERIALIZABLE)
some MT_MULTI_INSTANCE filter
another MT_MULTI_INSTANCE filter
yet another MT_MULTI_INSTANCE filter
some MT_NICE_FILTER filter

It was advised not to repeat "setmtmode(2)" for every filter since this may affect performance
so it will be like

setmtmode(3) #set mode to MT_SERIALIZABLE
source call
setmtmode(2) #set mode to MT_MULTI_INSTANCE
some MT_MULTI_INSTANCE filter
another MT_MULTI_INSTANCE filter
yet another MT_MULTI_INSTANCE filter
setmtmode(1) #set mode to MT_NICE_FILTER
some MT_NICE_FILTER filter

so the question now how avs+ deal with this case?

[takla]

I have some questions about MT in avs+


http://avisynth.nl/index.php/SetFilterMTMode#Choosing_the_correct_MT_mode

See also the "Example" section of that link.

SetFilterMTMode("FFVideoSource", 3)
or
SetFilterMTMode("FFVideoSource", MT_SERIALIZED)

[real.finder]

http://avisynth.nl/index.php/SetFilterMTMode#Choosing_the_correct_MT_mode

See also the "Example" section of that link.

SetFilterMTMode("FFVideoSource", 3)
or
SetFilterMTMode("FFVideoSource", MT_SERIALIZED)

already know this, it's not what I asked about

[qyot27]

Prefetch can be used multiple times in-script to alter the threading use per-filter. AFAIK, avoiding bottlenecks still requires structuring the script such that easily-scalable-threading doesn't get clogged up by other filters that don't scale nearly as well, regardless of what you actually do with Prefetch.

The correct part of the Wiki article was this, which notes the change in behavior and demonstrates multiple uses of Prefetch:
http://avisynth.nl/index.php/SetFilterMTMode#Prefetch

Controlling which CPU gets the task is something I imagine requires a much more robust level of device awareness*, and what actually controls that right now would have to be the OS's thread scheduler. Core-wise, I do know that on the Apple M1 (4P+4E), macOS prefers allocating the P cores to the process before the E cores, so Prefetch(4) will use the Performance cores, and only when you request more than that will it activate the Efficiency cores. I would presume most OSes act the same way, but I don't have any other Windows or Linux devices with heterogenous chips in them to actually test it.

*this is actually one aspect of several that I consider to be what a future bump to 3.8 would entail, btw



The behavior of the old MT fork isn't directly comparable to the way Plus does it. The old fork would change the threading model across the entire core, so multiple calls to SetMTMode had to be inserted between filters that had different scalability. Plus tethers the threading model to the individual filter and switches as necessary. Planning scripts in an optimal way is still required to avoid bottlenecks, but the fine threading control is delegated to Prefetch (hence the summary above this), not to changing the entire core's MT mode.

[real.finder]

Controlling which CPU gets the task is something I imagine requires a much more robust level of device awareness*, and what actually controls that right now would have to be the OS's thread scheduler.

thanks for the detailed information!

so, if there are 2 MT_SERIALIZED filters in-script, they will use the same thread? will this be changed in the future?

[DTL]

Filters (external plugins) can use their own multithreading. The MT_SERIALIZED only (at least ?) ensures the AVS+ core will send (request ?) frames in sequential order to single only instance of a filter in a total filter graph. This is important for filters with temporal processing. In other MT modes AVS+ core creates several instances of a filter and splits the total frame number to process to some blocks and process in different threads. This causes a random number of frames processed by each instance of a filter (with blocks like cache number of frames requested in Prefetch()) and can break the design of temporal filters.

So if filter need to do temporal processing it tries to request MT_SERIALIZED mode (like MAnalyse with temporal=true).

[rgr]

Filters (external plugins) can use their own multithreading. The MT_SERIALIZED only (at least ?) ensures the AVS+ core will send (request ?) frames in sequential order to single only instance of a filter in a total filter graph. This is important for filters with temporal processing. In other MT modes AVS+ core creates several instances of a filter and splits the total frame number to process to some blocks and process in different threads. This causes a random number of frames processed by each instance of a filter (with blocks like cache number of frames requested in Prefetch()) and can break the design of temporal filters.

So if filter need to do temporal processing it tries to request MT_SERIALIZED mode (like MAnalyse with temporal=true).

So what would a sample script with such a filter look like, for example, this one?

BSSource("input.mp4")
crop(...)
QTGMC(NoiseProcess=1,(...),TR=3)
lsfplus
Levels(...)

I've noticed that simply adding Prefetch to the end of such a script causes random frames to have errors (appearing to be decoding errors, e.g., part of the frame shifted to the right/left).

[DTL]

" part of the frame shifted to the right/left"

I not sure if QTGMC uses MAnalyse with temporal predictors enabled (may be at some preset ?) . If even temporal predictors enabled and not serialized MT mode used it expected to cause only very slight degradation of motion estimation quality (but at the repeating pattern of the frames).

Temporal predictors are the previous frame motion vectors. They are not present only for 0 frame of a clip in normal (serialized) mode. But in group of frames MT mode (standard) they are invalid (not present) for each starting frame of a group (of cache number of frames ?) and this can cause some repeatable ME quality degradation in total clip.

To test it you can simply enable and disable temporal predictors for MAnalyse and try to force different MT modes with SetMTMode().

From mvtools documentation:

temporal

Use temporal predictors from previous frame motion vectors. Not compatible with SetMTMode (classic Avisynth) or other than MT_SERIALIZED mode for Avisynth+, and requires a linear access to work correctly. Note: Since 2.7.32 the filter registers itself automatically MT_SERIALIZED instead of MT_MULTI_INSTANCE under Avisynth+ when temporal=true is given.

[ENunn]

i'm having an issue with some of my script encodes. i don't know if this is a plugin bug or an avisynth bug or what

whenever i use a plugin that adds grain i'm getting a glitch that shows up for only one frame, and it only happens in my encodes.

https://i.imgur.com/0TdaFqn.png

this does NOT show up when i preview my script in avspmod.

https://i.imgur.com/TOFNwf1.png

i have tried using multiple grain plugins (f3kdb, addgrainc, neo_dfttest's dither function), encoding to multiple codecs in ffmpeg (x264, prores, utvideo), and i have also tried encoding in virtualdub, and they all have the same issue. it seems like the only fix for it is to lower the prefetch value or not use prefetch at all.

i don't know if anyone else has this problem, but i figured i'd share whats going on on my end.

[Boulder]

You could try using something like Preroll(10) right after your source loading line.

[ENunn]

You could try using something like Preroll(10) right after your source loading line.

didn't fix it unfortunately

[Emulgator]

Post your script. Try different source filters.

[poisondeathray]

whenever i use a plugin that adds grain i'm getting a glitch that shows up for only one frame, and it only happens in my encodes.


If you encode it several separate times, does the glitch always occur on the exact same frame in the same position? Or is it a random distribution , or maybe more than 1 frame and you didn't see it?

Also check your cpu / hardware temperatures during encoding and perform HW stability checks . A single frame preview in avspmod is not going to "tax" the system as much as full encoding . No prefetch is going to be less resource intensive than prefetch(high number)

[ENunn]

Post your script. Try different source filters.

LoadVirtualDubPlugin("C:\Program Files (x86)\AviSynth+\plugins64+\ccd_64bit.vdf", "ccd", 0)
v = AVISource("i:\virtualdub\fix\vhs\opening to blind fury 1992 vhs.avi")
a = lwlibavaudioSource("i:\recordings\opening to blind fury 1992 vhs.flac")
audiodub(v,a)
preroll(10)
#delayaudio(.100)
#delayaudio(.150)
assumetff().converttoyuv422(matrix="rec601", interlaced=true).convertbits(10)
Levels(30, 1,940, 0, 1020, coring=false,dither=true).tweak(bright=0, cont=1.00, hue=-0, sat=0.850, coring=false, dither=true).convertbits(8)
#turnRight().Histogram().TurnLeft()
Trim(291, 3042)
Mergechroma(separatefields().vinverse().weave())
converttorgb32(matrix="rec601", interlaced=true,chromaresample="point")
e=separatefields().selecteven().CCD(5,0)
o=separatefields().selectodd().CCD(5,0)
interleave(e,o).weave()
converttoyv24(matrix="rec601", interlaced=true,chromaresample="point")
qtgmc(preset="Faster", tr2=2, ezdenoise=0.0, sharpness=0.0, sourcematch=3, border=false)
neo_dfttest(sigma=5, tbsize=3)
LSFplus(ss_x=1.5, ss_y=1.5, secure=true, Spwr=2, SdmpHi=0, soothe=false, preblur="on", keep=25)
FineDehalo(rx=2.5, ry=2.5)
neo_f3kdb(grainy=64, grainc=64, dynamic_grain=true)
converttoyv12(matrix="rec601", interlaced=false,chromaresample="point")
prefetch(10)

tried lwlibavvideosource, had the same result

If you encode it several separate times, does the glitch always occur on the exact same frame in the same position? Or is it a random distribution , or maybe more than 1 frame and you didn't see it?
nope. it's different every time. when it shows up its only for a frame but i wouldn't be surprised if it shows up multiple times on the encode. i haven't really checked the whole video.

Also check your cpu / hardware temperatures during encoding and perform HW stability checks . A single frame preview in avspmod is not going to "tax" the system as much as full encoding . No prefetch is going to be less resource intensive than prefetch(high number)

everything's cool and stable.

[DTL]

Can we enable multi-processor compilation for Visual Studio (at least) for AVS+ in the cmake settings ? If it not cause additional bugs it is visibly faster in the initial building of the project.

[real.finder]

nope. it's different every time. when it shows up its only for a frame but i wouldn't be surprised if it shows up multiple times on the encode. i haven't really checked the whole video.


maybe one filter you use has wrong mt mode, maybe ccd

[qyot27]

Can we enable multi-processor compilation for Visual Studio (at least) for AVS+ in the cmake settings ? If it not cause additional bugs it is visibly faster in the initial building of the project.
It's the user's job to select an appropriate number of parallel jobs when they build the project; we should not be in the business of making any assumptions about that. There is no one-size-fits-all setting - even in the current/near future generation of CPUs, what works as a reasonable value for a 9850HX won't fit Nova Lake, to say nothing of any recent Threadripper PRO or Epyc or Xeon.

Other CPU architectures (and x86, if using MinGW or Linux/macOS/BSD/Haiku) already have Ninja if you want full core saturation by default. And while it isn't a concern with AviSynth+, in principle, setting the number of parallel jobs too high can cause problems in the build tools themselves if you don't have enough RAM. The llvm-mingw compilation process will repeatedly die after a while on a Samsung Galaxy Book4 Edge (Snapdragon X Elite X1E-80-100, 12 cores, 16GB RAM) if allowed to just use all cores, because it will run out of memory.

[ENunn]

maybe one filter you use has wrong mt mode, maybe ccd

nah this has been happening long before i started using ccd. i posted about this issue before on both videohelp and doom9. here's a post i made of it happening on an older script.
(https://forum.doom9.org/showthread.php?t=185782) at first i thought it was the median filter causing it, but nah its something with all the plugins i've tried that add grain.

[real.finder]

nah this has been happening long before i started using ccd. i posted about this issue before on both videohelp and doom9. here's a post i made of it happening on an older script.
(https://forum.doom9.org/showthread.php?t=185782) at first i thought it was the median filter causing it, but nah its something with all the plugins i've tried that add grain.

maybe post a minimal script with small sample that causing it so we can see what is the problem

[DTL]

It's the user's job to select an appropriate number of parallel jobs when they build the project; we should not be in the business of making any assumptions about that. There is no one-size-fits-all setting - even in the current/near future generation of CPUs, what works as a reasonable value for a 9850HX won't fit Nova Lake, to say nothing of any recent Threadripper PRO or Epyc or Xeon.



Eh - I mean we have a simple switch like on or off in Visual Studio project settings :

https://i.ibb.co/5XcZ9ghY/2025-11-07-101416109.png (https://ibb.co/V0B6NcYH)

It looks /MP compiler option only and no more adjustments. With current cmake project generation it looks this option is missed and compilation takes only 1 core and slower. With this setting enabled the compiler auto-selects everything and task manager shows all cores loaded and compilation going faster. If this not causes any issues it expected to be enabled.

Also it is *nice to have* .editorconfig file for Visual Studio to set even strings formatting. pinterf made this file for mvtools project:

root = true

# NOTE: editorconfig-core-c has a hardcoded limitation of 50 chars per .ini section name

[*.{cpp,h,hpp,c,asm,hlsl}]
indent_style = space
indent_size = 2
trim_trailing_whitespace = true
end_of_line = crlf
insert_final_newline = true

[Sources/fstb/*.{cpp,h,hpp,c,asm}]
indent_style = tab
tab_width = 3
[Sources/conc/*.{cpp,h,hpp,c,asm}]
indent_style = tab
tab_width = 3

[*.md]
indent_style = space
trim_trailing_whitespace = false


After using of the project files from current cmake generator it looks default Visual Studio settings not match AVS+ files and strings tabs spaces (ident_size ?) are bigger. And may be hidden CR/LF at the end of lines not match too.

[DTL]

i'm having an issue with some of my script encodes. i don't know if this is a plugin bug or an avisynth bug or what

whenever i use a plugin that adds grain i'm getting a glitch that shows up for only one frame, and it only happens in my encodes.

https://i.imgur.com/0TdaFqn.png

it seems like the only fix for it is to lower the prefetch value or not use prefetch at all.

i don't know if anyone else has this problem, but i figured i'd share whats going on on my end.

Looks like some part of processing may have issues with multithreading (not correct lock of buffers to process or other).

I got somehow similair look of the frame when I test not MT-friendly implementation of filter in AVS+ core.
https://i.ibb.co/XrVyrRrP/2025-11-08-104614796.png (https://ibb.co/DHLGH2HF)

Try to decrease number of filters in script to make smallest possible script with same issue to attempt to narrow filter with this issue.

The next sad feature found with this implementation: After script load (access from frame 0 in VirtualDub) or any random seek operation system return some sequence of corrupted frames but if continue making sequential frame access over about (MT cache number ?) frames - the distortions stops and next frames returned OK. As I remember I have very close looking issue with first several corrupted frames with mvtools2 encodings. But at the long footage with slow start from black frames it is not very annoying. But the source of these errors may be close like some still present issue in AVS+ multithreading. It is at least subject to note.

If tested with Prefetct(4,1) instead of Prefetch(4) - the number of corrupted frames become significantly lower (like about 2..3). As I remember deafult cache number of frames is threads x2 (?) so Prefetch(4) equal to Prefetch(4,8) (?). It looks like something going out of sync at clip start or random seek operation and going in sync after about cache_number_frames sequential access. Even if some non-MT correct implementation in the filter exist.

Some working tech demo of planes-sequential processing workflow for AVS+ filters created: Idea description https://github.com/AviSynth/AviSynthPlus/issues/460 and current sources and release for test with Invert() filter example - https://github.com/DTL2020/AvisynthPlus-plane-base-filter-example/releases
Old AVS and AVS+ filtergraph uses frame-based (full structure of planes) dataflow between filters and this causes worse data locality in CPU caches and performance degradation. Better performance solution (for compatible filters) is process single plane in a sequence of compatible filters.

Complete implementation requires both redesign of AVS+ core and some changes in compatible filters. This API update do not change plane processing core of the filters and only changes processing order of the planes in the filtergraph. So required redesign of filters is not very much. Mostly addition of 2 new methods of connecting filters with partial frame processing (first plane) and requesting of each residual plane processing from output filters to input filters via filtergraph.

[Emulgator]

Good find, DTL.

[rgr]

" part of the frame shifted to the right/left"

I not sure if QTGMC uses MAnalyse with temporal predictors enabled (may be at some preset ?) . If even temporal predictors enabled and not serialized MT mode used it expected to cause only very slight degradation of motion estimation quality (but at the repeating pattern of the frames).

In addition to shifting parts of the image, there are also errors like this one -- part of the image simply becomes a black spot.

https://gcdnb.pbrd.co/images/U0SLZXSG7DK8.png?o=1

[DTL]

New testbuild to play with - https://github.com/DTL2020/AviSynthPlus/releases/tag/post-3.7.5_m1_test5

Finally working implementation with multithreading and manual control of size of the part of the plane to process in between sequence in filtergraph via new script function.

Test script:

SetRowsRegionSize(32000)

ColorBarsHD(2500, 1000)

Invert()
Invert()
Invert()
Invert()
Invert()
Invert()
Invert()
Invert()

Prefetch(4)


Test results at i3-2120 CPU (128k L1 (32k per each of 4 cores ?), 512k L2, 3.0MB L3 caches) :

For 8 Invert() in a sequence and Prefetch(4) - AVS+ 3.7.5 release 123 fps.
Test 5 release depending on SetRowsRangeSize() setting:
1024k - 130fps
512k - 272 fps
256k - 336fps
64k - 405 fps
32k - 412 fps

It looks best optimized for about 1/4 of L1 cache size (if it is enough in size like 32k and more ? or L1 cache size of each core ?). Max performance increases about 3.34x in comparison with full frame all planes cycling in each filter (with frame size 2500x1000 and YV24 format of 4:4:4).

Need to make program text smaller in next versions and attempt to add the same processing modes in some more visible results filters like Levels(). A sequence of Invert() are not clear to check if they are all really processing all parts of each frame in a sequence.

Currently this tech demo only supports simple fiter chains without splitting of output clips to several consumer filters. Standard filters expected to use standard GetFrame() connection pin/method. But new filters need to solve this issue (they currently ask sources about available best performance modes and will try to put requests on partially processed frames/planes).

[ENunn]

maybe post a minimal script with small sample that causing it so we can see what is the problem

lwlibavvideosource("path\to\file.mkv", fpsnum=30000,fpsden=1001,rap_verification=false,prefer_hw=3)
assumetff()
qtgmc(preset="Faster", tr2=2, ezdenoise=0.0, sharpness=0.0, sourcematch=3, border=false)
neo_dfttest(sigma=5, tbsize=3)
neo_f3kdb(grainy=64, grainc=64, dynamic_grain=true)
prefetch(10)

i don't think a small sample isn't gonna show my issue consistently. but here's a link to one though. (https://mega.nz/file/nglGBbyZ#OyQngGohJHphtNiwG13kpK3P44ch5hVHwOhXgvuzoFw)

i did an encode with prefetch set to 3. the glitching STILL HAPPENS!! i don't get it. this is a different source but the issue still applies.
https://i.imgur.com/quOTuJT.png

good info dtl!!

Try to decrease number of filters in script to make smallest possible script with same issue to attempt to narrow filter with this issue.

i did some tests. i made separate scripts. one with only qtgmc, one with only qtgmc and neo_dfttest, one with only qtgmc and neo_f3kdb, and one with qtgmc, neo_f3kdb, and neo_dfttest.

qtgmc - no glitching
qtgmc and neo_dfttest - glitching
qtgmc and neo_f3kdb - no glitching
qtgmc, neo_dfttest, and neo_f3dkb - i didn't notice any glitching on this encode but as mentioned before, it happens randomly and the glitches are different every time. it could also be happening in dark scenes and i don't notice it.

so it looks like it ISN'T the grain plugins that are causing it, but neo_dfttest. welp, time for a new denoiser.

[real.finder]

so it looks like it ISN'T the grain plugins that are causing it, but neo_dfttest. welp, time for a new denoiser.

seems its old problem https://github.com/HomeOfAviSynthPlusEvolution/neo_DFTTest/issues/6