AviSynth+ thread Vol.2

[VoodooFX]

Heads-up:
I encountered some bug creating "special effects" with that unofficial compile kedautinh12 is posting.
And some another with official avs. I need to remember what I was doing...

[guest]

Heads-up:
I encountered some bug creating "special effects" with that unofficial compile kedautinh12 is posting.
And some another with official avs. I need to remember what I was doing...

"unofficial compile" of what ??

[VoodooFX]

"unofficial compile" of what ??
AviSynth+

[gispos]

Make some tech demo of YV12 to RGB (planar only for now, RGBP8) - https://github.com/DTL2020/ConvertYV12toRGB/releases/tag/0.0.1a . AVX2 only.

It sort of draft-preview quality (using sort of point-resize for UV upsampling and completely not-aware of 'chroma-location') and 265bit regs data shuffling is not completely debugged so columns are not arranged properly. Only should run on mod64 frame widths (1920 or 3840 is good) or will not process last non-mod64 columns. Playing with 64 YUV and RGB samples at once not very easy in debugging - to place everything in correct order over scanline.

For sort of point-resize of UV plane it uses 8bit unpack with itself (single instruction) for H-scale and for V-scale it make UV line doubling in load address advancing (so it is universal 4:2:2 and 4:2:0 to RGB convert engine). So UV upscaling takes close to minimal possible time in this example.

Also can use internal OpenMP (threads) param to check AVS MT vs internal MT (not yet tested at all).

It just some quick performance test of such approach. I not have AVX2 chip at home so can not run performance test, only check if it output something like RGB decoded frame in SDE and not crash with default 640,480 frame.

Expected script for performance test is:

LoadPlugin("DecodeYV12toRGB.dll")
ColorBars(3840, 2160, pixel_type="YV12")

DecodeYV12toRGB(threads=1)
Prefetch(N) # N is number of threads, or try internal OpenMP threads at filter.


Interleaving of 32samples R, G, B AVX registers into RGB24 (or RGB32 with empty alpha) will take some more design ideas how to make it any fast. So currently planar RGB wins for processing but create 3 different store streams into RAM (though total speed is comparable to RGB24 interleaved).

Interleaving of planar RGB into RGB24 of mod3 channels step may be so SIMD-unfriendly that it may be better to do RGB32 instead.
It runs like a pig. (That's what they say in Germany):)

ConvertToRGB32() = ~35 fps

Your code and ConvertToRGB32() = ~116 fps
DecodeYV12toRGB(threads=1)
ConvertToRGB32()

All without prefetch, nice!

But, the picture is not clean, I have strong stripes.

https://i.postimg.cc/kXG2XDQd/Screenshot-4.jpg (https://postimg.cc/0bgkVkXt)

[gispos]

For consumers? Still 4:2:0, but 10bit instead of 8bit and with a different chroma location (top left, aka 4:2:0 type 2) than the default MPEG-2 one (left, aka 4:2:0 type 0).
Hmm... I have a lot of YUV422P10 lying around here. And after ConvertBits(8) it comes out YV16.

Consumer pfff :D

[DTL]

"the picture is not clean, I have strong stripes."

Yes - it is not finally debugged. It is only first tech demo to estimate performance of single pass processing. Also I got some ideas how to make output of RGB32 interleaved (sorry for our lovely RGB24 interleaved - it is really much more complex to design and debug). The AVX2 looks like not have dual-input permutes for 8bit bytes (also not have 8bit permutes at all - so accumulating of G and R after initial B-shuffling require permutes to temporal reg and masked blends to output) so it will be somehow slower in compare with our nice AVX512 instructions sets (having VBMI and VL families). Also AVX2 do not have dual-input permutes at all.

AVX512_VBMI have great _mm512_permutex2var_epi8 instruction for filling RGB32 output from planar RGB regs to whatever position required (without any temporals and masked blends). Also it is really 64-bytes all flat field to run at - not 2x128bit poor processing of AVX2 for lots of operations (so is the current bad running build require lots of debug between 2x128bit lanes operations of AVX2).

I like to have AVX512 chips only and forgot poorly designed AVX2 of semi-256bit real opeation. It is mostly dual-speed SSE2 2x128bit operation but not true 256bit.

[gispos]

I'm excited to see how it continues!

Where does this difference come from?
1920 x 1080 YUV422P10

ConvertToRGB32(matrix="Rec709")
~97 fps

ConvertBits(8)
ConvertToRGB32(matrix="Rec709")
~113 fps

I cannot see any difference in the quality.

[FranceBB]

Consumer pfff :D

LOL yeah TX Ready mezzanine files are of course 4:2:2 10bit for me too over here ehehehehe
But I gotta admit, it's a bit like a drug, once I started watching those for my favorite movies and tv series ehm I mean for work-related purposes, normal consumer Satellite and Terrestrial feeds feel like unwatchable... :rolleyes:

[DTL]

"Where does this difference come from?
1920 x 1080 YUV422P10

ConvertToRGB32(matrix="Rec709")
~97 fps

ConvertBits(8)
ConvertToRGB32(matrix="Rec709")
~113 fps"

When you process >8bit the AVS engines may generally run with 16bit words (all 10 12 14 16 bit are typically processed as 16bit and it typically require 32bit integer intermediates). So when you first downconvert to 8bit the UV upscale and dematrix run with 8bit words (and typically 16bit intermediates) and it is faster.

So possibly 1920 x 1080 YUV422P16 will run at the same speed as 1920 x 1080 YUV422P10.

New version: https://github.com/DTL2020/ConvertYV12toRGB/releases/tag/0.0.2a

Now it can write RGB32 interleaved. At i3-9100T CPU the 4 and 1 AVS-threaded performance is about equal. It looks even 1 core of AVX2 can fill all RAM bandwidth of poor DDR-SDRAM.

Single threaded at i3-9100T is 38 fps for ConvertToRGB32() and 180 fps for DecodeYV12toRGB(threads=1) at 4K YV12 decode.

At i5-9600K:
1 thread 300fps
2 threads 355 fps
4 threads 366 fps

As theory says it can not run over about 500 fps at poor DDR4-SDRAM. The AVX2 units greatly overperform poor SDRAM at such simple tasks. So intel not like to put AVX512 into very poor end-users chips with very slow RAM.

[StainlessS]

It looks even 1 core of AVX2 can fill all RAM bandwidth of poor DDR-SDRAM.
Just curious, is that single channel, or dual channel DRAM that you're using ?

[My 1 litre (lenovo tiny/HP mini/Dell micro) machines all came with single channel DRAM. (1 DIMM used, 1 DIMM free), all mine are now dual channel]

EDIT: Below 9th Gen type 'T' CPU's (low power, as used in 1 litre desktop machines + others)

############## :LGA 1151: 6th Gen -> 9th Gen CPU's ##############
Intel Core i9-9900T (2.1 - 4.4 GHz 8C/16T 14nm HD630 12MB Cache DDR4-2666_Max128GB PCIe3.0) £140
Intel Core i7-9700T (2.0 - 4.3 GHz 8C/8T 14nm HD630 12MB Cache DDR4-2666_Max128GB PCIe3.0) £130 [EDIT: down from £140]

Intel Core i5-9600T (2.3 - 3.9 GHz 6C/6T 14nm HD630 9MB Cache DDR4-2666_Max128GB PCIe3.0) £75 [EDIT: up from £65]
Intel Core i5-9500T (2.2 - 3.7 GHz 6C/6T 14nm HD630 9MB Cache DDR4-2666_Max128GB PCIe3.0) £45 [EDIT: Price drop from £62]
Intel Core i5-9400T (1.8 - 3.4 GHz 6C/6T 14nm HD630 9MB Cache DDR4-2666_Max128GB PCIe3.0) £52 [EDIT Price rise from £50, the i5-9500T above, is faster, cheaper]

Intel Core i3-9300T (3.2 - 3.8 GHz 4C/4T 14nm HD630 8MB Cache DDR4-2400_Max64GB PCIe3.0) Price not known
Intel Core i3-9100T (3.1 - 3.7 GHz 4C/4T 14nm HD630 6MB Cache DDR4-2400_Max64GB PCIe3.0) £22( ~ CPU 2nd user price [EDIT: Price drop from £28])


EDIT: Approx prices of 2nd user Intel Processors [Not necessarily in-stock], click on required socket type, and In_Stock buttons if required.
https://uk.webuy.com/search?stext=Intel+processor&sortBy=prod_cex_uk_price_asc&categoryFriendlyName=Intel+Processors&Manufacturer=Intel

[VoodooFX]

Heads-up:
I encountered some bug creating "special effects" with that unofficial compile kedautinh12 is posting.
And some another with official avs. I need to remember what I was doing...

One located for "unofficial" avs (https://gitlab.com/uvz/AviSynthPlus-Builds/-/tree/main/IntelLLVM/x86):

b = BlankClip(height=4, width=2, color_yuv=$000000, pixel_type="Y8")
w = BlankClip(height=4, width=2, color_yuv=$FFFFFF, pixel_type="Y8")

clp = StackHorizontal(w,b).Blur(1).BicubicResize(64,64)
turn = clp.Turn180

StackVertical(clp, turn)


Bug:

https://i.imgur.com/6PqrjPv.png


Another for "official" avs I can't remember what exactly I was doing, I'll try tomorrow.

[DTL]

Just curious, is that single channel, or dual channel DRAM that you're using ?

[My 1 litre (lenovo tiny/HP mini/Dell micro) machines all came with single channel DRAM. (1 DIMM used, 1 DIMM free), all mine are now dual channel]

EDIT: Below 9th Gen type 'T' CPU's (low power, as used in 1 litre desktop machines + others)

############## :LGA 1151: 6th Gen -> 9th Gen CPU's ##############
Intel Core i3-9100T (3.1 - 3.7 GHz 4C/4T 14nm HD630 6MB Cache DDR4-2400_Max64GB PCIe3.0) £22( ~ CPU 2nd user price [EDIT: Price drop from £28])


[/url]

Oh - I think it is impossible in 201x to have 1channel - but it is really so
https://i.ibb.co/Y7gRH9k/lenovo-sch.png (https://imgbb.com/)

It is lenovo monoblock PC. May be really 2nd channel left for RAM upgrade option.

For i3-9100T CPU intel promises about 37.5 GB/s MAX RAM bandwidth so 1channel may be around 15..17 GB/s only.

So 1channel of DDR4 is about 180fps, 2channel is about 360 fps.

Checked at Xeon Gold6134 (not sure how many RAM channels installed - CPU-Z can not run properly, but may be at least 3 of DDR4 dual ranks installed at least ?):

1 thread 270 fps
2 threads 500 fps
4 threads 670 fps
8 threads 750 fps


The ConvertToRGB32 performance is much lower at low threads number (compute bounded)
1 thread 40 fps
2 threads 75 fps
4 threads 150 fps
8 threads 245 fps

So if 6channels of DDR4 (2666) installed it saturates at about 700 fps of RGB32 4K store. Wiki https://en.wikipedia.org/wiki/DDR4_SDRAM DDR4-2666 is about 20 GB/s per channel (?) so 6 channels is about 120 GB/s only.

Some more modern 202x general compute platforms finally will be 12channels DDR5 per CPU (something around 500 GB/s ?, finally 1 TB/s per 2 CPU board ?) - https://www.techpowerup.com/306145/intel-xeon-granite-rapids-and-sierra-forest-to-feature-up-to-500-watt-tdp-and-12-channel-memory . LGA-7529 really huge package. But to reach 1..2+ TB/s it looks require non-user-side mountable chips. Only factory assembled CPU + RAM modules.

[StainlessS]

May be really 2nd channel left for RAM upgrade option.
Yep, that's my guess too,
In the past Dell etc tended to fit eg 4 x 1GB DIMMS for 4GB total,
and people used to get a bit miffed that the had to throw away some DIMMS and buy all new DIMMS to upgrade RAM.
I prefer what they seem to be doing now, less annoying.

[DTL]

Btw using Converttorgb32() only for planar->packed causes ~30% fps drop. It seems a lot at first look. I have to test with libp2p (https://github.com/sekrit-twc/libp2p).

You can try now this repacking at AVX2 - https://github.com/DTL2020/ConvertYV12toRGB/blob/7cd0d98f166db3444168a0c19e2ecd1135a64cea/DecodeYV12toRGB.cpp#L218

It process 64 samples per pass. Though if you read planar from main host RAM it may be only RAM speed limited and not visibly depend on byte permute engine implementation.

The only practical benefit from fast conversion function is it left more CPU cores to compute something useful. And possibly draw less power if someone still care about.

[gispos]

New version: https://github.com/DTL2020/ConvertYV12toRGB/releases/tag/0.0.2a

Now it can write RGB32 interleaved. At i3-9100T CPU the 4 and 1 AVS-threaded performance is about equal. It looks even 1 core of AVX2 can fill all RAM bandwidth of poor DDR-SDRAM.

It gets better and better, ~205 fps without prefetch!

The stripes have become less, but still present.
https://i.postimg.cc/Wp6ytXkR/Screenshot-5.jpg (https://postimg.cc/5jygR5gp)


EDIT: Below 9th Gen type 'T' CPU's (low power, as used in 1 litre desktop machines + others)

############## :LGA 1151: 6th Gen -> 9th Gen CPU's ##############
Intel Core i9-9900T (2.1 - 4.4 GHz 8C/16T 14nm HD630 12MB Cache DDR4-2666_Max128GB PCIe3.0) £140
Intel Core i7-9700T (2.0 - 4.3 GHz 8C/8T 14nm HD630 12MB Cache DDR4-2666_Max128GB PCIe3.0) £130 [EDIT: down from £140]

Intel Core i5-9600T (2.3 - 3.9 GHz 6C/6T 14nm HD630 9MB Cache DDR4-2666_Max128GB PCIe3.0) £75 [EDIT: up from £65]
Intel Core i5-9500T (2.2 - 3.7 GHz 6C/6T 14nm HD630 9MB Cache DDR4-2666_Max128GB PCIe3.0) £45 [EDIT: Price drop from £62]
Intel Core i5-9400T (1.8 - 3.4 GHz 6C/6T 14nm HD630 9MB Cache DDR4-2666_Max128GB PCIe3.0) £52 [EDIT Price rise from £50, the i5-9500T above, is faster, cheaper]

Intel Core i3-9300T (3.2 - 3.8 GHz 4C/4T 14nm HD630 8MB Cache DDR4-2400_Max64GB PCIe3.0) Price not known
Intel Core i3-9100T (3.1 - 3.7 GHz 4C/4T 14nm HD630 6MB Cache DDR4-2400_Max64GB PCIe3.0) £22( ~ CPU 2nd user price [EDIT: Price drop from £28])


EDIT: Approx prices of 2nd user Intel Processors [Not necessarily in-stock], click on required socket type, and In_Stock buttons if required.
https://uk.webuy.com/search?stext=Intel+processor&sortBy=prod_cex_uk_price_asc&categoryFriendlyName=Intel+Processors&Manufacturer=Intel
I was looking at new CPU's and was shocked by the power consumption.... had immediately lost the desire.

Then I looked at older ones.
I might like this one, it also has AVX-512 if not neutered by Intel and a TDP of 65 watts.
What do the experts think of the CPU?
https://ark.intel.com/content/www/us/en/ark/products/212279/intel-core-i711700-processor-16m-cache-up-to-4-90-ghz.html

[DTL]

11th intel may be good candiate because it typically have all cores AVX512 and not very bad set of AVX512-family instructions support. But sadly it is only 2 channels and DDR4. So it limited to about 50 GB/s only. I not see >2 channels RAM chips of this consumer series of CPUs.

So it may be really hard choise between 11th intel having AVX512 but only 2ch of DDR4 or newer intels with DDR5 (though 2ch only but about twice faster). It is generally balance between user's lovely plugins - either compute bound (AVX512 better if implemented) or memory bound (2ch DDR5 will be better even with AVX2 only). No one for all good solution at shrinking and dying desktops market.

Also DDR5 is sort of beginnig nowdays and I still not see as it run at 100 GB/s at 2ch. So may be 4..6 ch of more cheap DDR4 may be easily faster.

The 4ch RAM require more expensive chip case design and more expensive motherboard. So for running MS Word and Internet Explorer at home it is enough 1..2 ch of any mass market DDR SDRAM and no-AVX512 chip. In the old decades large market of end-users high performance desktops support low prices and good progress close to every year. Some promises progress speed increase and reach 'singularity' in about 202x or a bit later. They were too low in knowledge in product marketing cycle and the expected 'singularity' turn to about death of end-users desktops at all. The mining at GPU only delayed it a bit. Now the end-users desktops market is close to death and prices are high and performance progress is very low. It may be more noisy and hot but do its job.

So may be if plan to work with video processing in future years it may be recommended to look at second-hand Xeons (or may be complete second-hand workstation with Xeon with full-blood AVX512 and 6ch of at least DDR4 RAM, or cheap Xeon with 4ch DDR4).

You can look to list of classic Xeon W for workstations https://ark.intel.com/content/www/us/en/ark/products/series/125035/intel-xeon-w-processor.html - it range from very poor 1250 with 2ch of DDR4 to much more advanced 33xx with 8ch of DDR4.

[Boulder]

Have you tested these in a real life situation where you have more stuff in the script and the encoder's work eating the CPU cycles and memory bandwidth as well? It's often a very different case where you may notice only minor improvements even if some step is several times faster than earlier.

Building a PC from second hand parts is usually the most cost effective way of getting more power to the encoding jobs. And the 5xxx series Ryzens for example can be tamed so that they won't consume that much electricity at the expense of less performance. In any case, it's best to think of performance per watt since that is what matters.

[StainlessS]

My Lenovo M70Q Tiny Gen 2, has i5-11400T (35W TDP) has AVX512,
I dont know why but Lenovo tends to use the i5-xx400T which are the lower speed i5 cpu's,
Dell and HP tend to sell i5-xx500T standard speed i5's. [EDIT: i5-xx600T are high speed]
I only decided upon the lenovo 11th gen i5, so I could (one day) try out the AVX512 thingy, but CPU speed is a little disapointing.

It seems that ODD gen intel CPU's sell fewer, and EVEN gen bigger jump in performance, so I myself would opt for 12th gen
(which I might do as some point for a low power type T cpu).

Here is a 2nd user 12th gen i5-12500T machine that was online a couple of weeks ago, with 32GB DDR5 [SOLD @ £500.00],
Lenovo M80Q Gen3 Tiny/i5-12500T/32GB DDR5/2x 1TB SSD/W11/A
https://uk.webuy.com/product-detail?id=sdeslenm80qg365a&categoryName=desktops-windows&superCatName=computing&title=lenovo-m80q-gen3-tiny-i5-12500t-32gb-ddr5-2x-1tb-ssd-w11-a&referredFrom=search&queryID=ea994f54baf6f5116ae928f474865748&position=34
Intel 15-12500T:- https://www.intel.co.uk/content/www/uk/en/products/sku/96140/intel-core-i512500t-processor-18m-cache-up-to-4-40-ghz/specifications.html
Not an i7, but it still seemed like a reasonable deal {I watched it for about a week, then was 'gutted' when it sold :( }.
EDIT: I think was PCIe 4.0 too {double speed over PCIe 3.0 for recent nvme SSD}.

Here an i7-12700T (again low power)
Dell 7000 Micro/i7-12700T/16GB DDR4/256GB SSD/W10/B £690.00.
https://uk.webuy.com/product-detail?id=sdesdel7000mic34b&categoryName=desktops-windows&superCatName=computing&title=dell-7000-micro-i7-12700t-16gb-ddr4-256gb-ssd-w10-b&referredFrom=search&queryID=5f76971477c09cf6166e18151135d313&position=18
Intel i7-12700T:- https://www.intel.co.uk/content/www/uk/en/products/sku/134596/intel-core-i712700t-processor-25m-cache-up-to-4-70-ghz/specifications.html

Note, the 12 gen CPUs drop from 14nm to 10nm Lithography { "Intel 7" is 10nm, maybe an attempt by intel to deceive ??? }.

I kinda like the tiny machines, but current (dell 12th gen) tiny's come with 90W PSU instead of 65W of previous gens, and for the higher power non 'T' comes with a 120W (or maybe its 130W) PSU.
Running non 'T' in eg Dell 7000 Micro is supposed to be quite noisy. [EDIT: Under load.]

Gispos, If you decide to opt for a new CPU, do tell what you decided and how it went.

EDIT:
How that works in practice is that those new third-generation 10nm chips will be referred to as “Intel 7,” instead of getting some 10nm-based name (like last year’s 10nm SuperFin chips).
https://www.theverge.com/2021/7/26/22594074/intel-acclerated-new-architecture-roadmap-naming-7nm-2025

[DTL]

DDR5 also greatly differs in performance from DDR5-4000 to DDR5-8000 and not all CPUs support all clock rates. So low speed DDR5 may be not very faster in compare with fast DDR4. And DDR4 can go up to about DDR4-4800.

If operate with large frame size and many threads and RGB colour formats the main host RAM speed may be significant. AVS very like to use large software caches around each filter as I see.

[gispos]

Gispos, If you decide to opt for a new CPU, do tell what you decided and how it went.

You know, with me it always takes an eternity until I have decided. With my monitor, I think it was 6 months. Then there will probably already be quantum computers.;)

[DTL]

New version - https://github.com/DTL2020/ConvertYV12toRGB/releases/tag/0.0.3

Possibly bugs with UV columns positions fixed. Added 'matrix' param - so for UHD with 2020(ncl ?) matrix it should be

DecodeYV12toRGB(matrix=2)

Currently only limitedYV12->limitedRGB32 conversion. As a benefit for monitor on typical PC-RGB display it not clips superwhites, but raises black to 16 (lower contrast). On professional 'video' display or supported limited range in RGB it should display everything right.

"Have you tested these in a real life situation where you have more stuff in the script and the encoder's work eating the CPU cycles and memory bandwidth as well? It's often a very different case where you may notice only minor improvements even if some step is several times faster than earlier."

Real situation with host RAM speed is about:

As 199x..200x endusers PC general purpose shows - the home PCs at typical general purpose designed software (games/office/internet) typically only compute limited and close to zero benefit from 2ch to 4ch RAM config. So general purpose home PCs typically of very low RAM channels to be cheaper.

Only special designed software for large datasets processing may be host-RAM speed limited. So for datacenters and other scientific (and may be video-proc) there are special Xeon families:

1. Many cores + low RAM ch (2 to 6) - for compute-bound applications
2. Lower cores + high RAM ch (8 to 12) - for memory-bound applications
3. Somehow balanced - many cores + high RAM ch (8) - for many types of applications

At the datacenter design for special computing it may be visibly profitable to select right family of Xeons to get more performace per investment. Also if compute task can not be offloaded to better compute accelerator and require general-purpose CPUs.

If it can be offloaded - there is no need to buy too slow DDR-SDRAM - the good compute unit in 202x years is something like 6RU chassis with 2 kW dual-PSU and 6..12 Tesla GA100 or may be better new GH100 accelerator boards mounted. With aggregated RAM speed about 24+ TB/s HBM2e/HBM3 and it is 1000x faster in compare with DDR3 2ch home enduser PCs.

Also Xeon Max finally go into HBM RAM - https://wccftech.com/intel-announces-the-worlds-first-x86-cpu-with-hbm-memory-xeon-max-sapphire-rapids-data-center-cpu/ 1 TB/s RAM bandwidth per CPU is much better for AVX512/1024 . https://wccftech.com/intel-unleashes-hbm-powered-xeon-cpu-max-sapphire-rapids-xeon-gpu-max-ponte-vecchio-for-data-centers/
So it is again some intermediate period of changing RAM type. 2023 is the very beginning of x86 general purpose CPUs with HBM RAM integrated. Marketing title is "Xeon CPU MAX 'Sapphire Rapids HBM' ".

Having Xeon Max in the undertable compute box is a new dream for 202x season. Xeon Max can directly run AVS+ and its plugins without offloading to external special programming accelerator (like Tesla A100 board).

*waiting for vendors to ship Performance Workstations Xeon Max based for UHD processing*. Xeon Platinum 9462 - 32 Core (2.7 / 3.1 GHz) - $7995 US may be cheaper in compare with A100 external accelerator unit (though only about 1/2 of RAM speed also). It is finally next-generation compute platform ready to use at home and compatible with all accumulated PC software. May be HP will announce some Workstations on new platform in 2023 or 2024 at least. Currently I hear poor users of old platforms can not run RAW 6K realtime NLE and need to use low-res proxy and offline render with RAW sources. The next-gen solution will easily run 6K RAW in realtime and save development time and make NLE workflow more simple ans less buggy. Also it mean in end of 2023 and next years the current 6ch old Xeons at DDR-SDRAM will become second-hand throw-away hardware at prices for poors.

Also in Max series listed Xeon 9460 - https://ark.intel.com/content/www/ru/ru/ark/products/232595/intel-xeon-cpu-max-9460-processor-97-5m-cache-2-20-ghz.html
Also can be expanded with 8ch of slow DDR5-4800 over internal 64 GB of HBM. Dual-CPU Workstation should be possible with 128 GB of HBM total.

For higher demanding applications intel provides 4 OAM: 512GB HBM2e, 512 Xe Cores, 2400W TDP, 208 TFLOPS, 12.8 TB/s memory. But require redesign AVS core to use in-accelerator memory management and filters running.

About displaying/monitoring of YV12 at Windows PCs - may be it is better to found some way to feed 4:2:0 data to display accelerator directly to decode to RGB and send to display. If GDI not support AVS YV12 format - the DirectX typically work with sort of NVIDIA-designed NV12 format. It is about same as YV12 (equal in datasize and transfer performance) but UV planes samples are interleaved and located under Y plane (so memory transfer line size is equal for Y and UV-interleaved lines). If you can test if your software can feed NV12 to display it may be better to make simple
ConvertYV12toNV12()
or add NV12 into AVS core (really hard because or many data-compute) filters.

https://learn.microsoft.com/en-us/windows-hardware/drivers/display/4-2-0-video-pixel-formats
NV12 is the preferred 4:2:0 pixel format.

Example of some poor non-SIMD conversion of YV12 to NV12 from MAnalyse to feed to DX12-ME accelerator - https://github.com/DTL2020/mvtools/blob/0a6b093507bc757457783d537bc6cfaaa273989d/Sources/MVAnalyse.cpp#L2407 (UV part only)

So if possible to feed Y and UV of NV12 separately to GDI API it is fastest format for display - you directly provide pointer to RAM Y plane of YV12 and it is DMA-uploaded to accelerator from RAM. No CPU-core load required (though PCIe board still uses CPU RAM controller and bus). You need only to interleave small UV planes and provide pointer to upload to accelerator.

In a perfect world - but really the DMA to accelerator may require special PHY/Virtual-RAM addresses lines mapping/alignment and stride so GDI may use internal or some intermediate library or driver function on host-CPU routine to relocate provided Y-plane buffer for DMA transfer preparation. So to make best AVS to accelerator transfer in YV12/NV12 it may require special AVS core redesign to meet DMA transfer requirements to display accelerators. It depend on display API used.

Also some ConvertYV12toNV12() filter may prepare all-aligned buffers for DMA to save from one more full-frame Y-plane memory remap.

[gispos]

New version - https://github.com/DTL2020/ConvertYV12toRGB/releases/tag/0.0.3

Possibly bugs with UV columns positions fixed. Added 'matrix' param - so for UHD with 2020(ncl ?) matrix it should be

DecodeYV12toRGB(matrix=2)


There are no more stripes, also the speed is still high, ~210 fps for me without prefetch.
But no matter what I set as matrix the colors do not match the original or to ConvertToRGB32().

Everything is a bit darker. Is that your hint on that?

Currently only limitedYV12->limitedRGB32 conversion. As a benefit for monitor on typical PC-RGB display it not clips superwhites, but raises black to 16 (lower contrast). On professional 'video' display or supported limited range in RGB it should display everything right.


Edit:
Looked at it again more closely.
You write limited range, 16..235, Yes everything light is darker and everything dark becomes lighter, is there a chance that you can still change this?

[DTL]

Yes - it is limitedYUV->limitedRGB.

Can you change range using GDI tools (programming of display accelerator if possible) ?

The ConvertToRGB32() possibly write Full range, but it cause clipping of superwhites (it is no good for monitoring anyway). Also it maps 16lvl to black. It helps to get full contrast on sRGB typical PC display.

I take coefficients for matrix from https://gist.github.com/yohhoy/dafa5a47dade85d8b40625261af3776a .

I not know now if it possible to make Full range (or partial-Full like 16..254 maps to 0..255 RGB without clipping of superwhites) playing with coefficients only. So no change to current processing engine from its current performance. If add special conversion to Full (standard mapping of 16..235 to 0..255 RGB) it will be somehow slower and also somehow lower in precision. Though the performance penalty may be invisible. Need to try.

Here is release of ver 0.1.0 https://github.com/DTL2020/ConvertYV12toRGB/releases/tag/0.1.0

Added RGB post processing with gain and offset - you can adjust any levels mapping (Full or black offset only and others)

offset - signed short (16bit) value to add to 8bit RGB value after decoding, default 0 (no offset)
gain - scaled to 64 multiplier (64 mean 1.0float - no gain), default 64 (no gain).

Most close to ConvertToRGB32(matrix="Rec709") output is about
DecodeYV12toRGB(matrix=1, gain=74, offset=-16)

gain 74 is (255/219) * 64 = 74.52 (may be 75 can work a bit better with still no bugs)

For still not known reason scale of 128 with better precision (7bit) not work correctly (with ColorBarsHD()) so only 6bit of 64 multiplier currently looks like working.

Black offset only without superwhites clipping is something like
DecodeYV12toRGB(matrix=1, gain=68, offset=-16)

Postprocessing part work always so it make some performance penalty (can not check now without AVX2 CPU). Performance penalty not depend on gain/offset values (also no check for validity so too extreme values will cause 16bit signed short computing over/underflow and other issues).

[gispos]

Postprocessing part work always so it make some performance penalty (can not check now without AVX2 CPU). Performance penalty not depend on gain/offset values (also no check for validity so too extreme values will cause 16bit signed short computing over/underflow and other issues).
So it is still ~189 fps for me.

You write for previews, I think it's also good for animation videos.
It more precisely favors the color boundaries than ConvertToRGB32. Now whether this is better for normal videos... what do others say.
https://vimeo.com/826648468?share=copy

Edit:
With normal videos I see almost no difference.
But it might be really interesting for the anamation film lovers.
So if you have a YV12 video you get the color fringes clean. I don't know if there are other filters that can do this.

What else I need to get rid of:
With all the test I noticed that some things affect the speed of my PC.
I have a TV card that always runs as soon as the PC is turned on. The costs but smooth 20% when I measure fps.
Even the browser makes only 30 fps out of 35 fps.
Embarrassing, but I had never thought of it, or say I would not have believed that makes such a difference.

So now I have a CPU with 20% more power. :)

[DTL]

"So it is still ~189 fps for me."

It is good enough. Next week I will try to do Intel C compiled binary and also AVX512 compiled (both VS2019 and IC 19) to see if it will benefit from AVX512 larger register file. The main program text is intrinsics-based so depend on C-compiler used. Also may be someone having LLVM compiler may try to do AVX2-limited and AVX512-limited compiled binaries to test performance.

I not check if current implementation run out of AVX2 register file size or not (so may use temp store/load some data to/from L1D cache and it adds performance penalty). Also it may depends on compiler used.

"It more precisely favors the color boundaries than ConvertToRGB32"

It should be equal to lowest quality resize for 'classic moving pictures' - ConvertToRGB32(chromaresample=point). Though for pixel-art it is good and make sharper chroma.

"With normal videos I see almost no difference."

For not very sharp 4K footages (and also soft film transfers) it may be not visible difference from bicubic resize chroma. So UHD1/2 may be treated as dual/quad upsampled FullHD and not need as precise subsampling processing as SD and HD.

"I don't know if there are other filters that can do this."

You can compare with ConvertToRGB32(chromaresample=point).

New version 0.2.0 - https://github.com/DTL2020/ConvertYV12toRGB/releases/tag/0.2.0

Uncached load looks like not work in most of chips (may work in future ?). Uncached store works very fine in > 1 thread. Though it may be mostly benefit if display accelerator loads result directly from RAM. Each use case may require cached of not store. With AVSmeter it looks much better to use non-cached store.

Test results at i5-11600 (AVX512 builds just very slightly faster, so looks most of data fit AVX2 register file well) with 2ch of DDR4-3200 RAM:

1thread cached store 255 fps, non-cached store 268 fps
2thread cached store 400 fps, non-cached store 532 fps
4thread cached store 340 fps, non-cached store 811 fps

at Xeon Gold 6134 (looks like 6ch of DDR4-2666 ?)
1thread cached store 256 fps, non-cached store 295 fps
2thread cached store 488 fps, non-cached store 596 fps
4thread cached store 694 fps, non-cached store 1175 fps
6thread cached store 760 fps, non-cached store 1570 fps (ConvertToRGB32 is 216 fps)
and some hyperthreading example:
8thread non-cached store 1725 fps
10thread non-cached store 1780 fps

Finally 1500+ fps with 6ch DDR4 is running. It looks with non-cached store it good to check if AVX512 version of processing may double fps per 1 thread.

AVS multithreading looks like visibly faster in compare with internal OpenMP (may use slower workunit size per thread and having more threads sync penalty). But takes much more RAM.

Also some redesigned non-cached read version - https://github.com/DTL2020/ConvertYV12toRGB/releases/tag/0.2.1
At i5-9600K the cl=false is slightly faster (about 730 and 760 fps). At i5-11600 the cl=false is significantly slower (about 800 and 560 fps). So it greatly depend on CPU family and may be use case in surrounding data source/sink.

[gispos]

"So it is still ~189 fps for me."

Next week I will try to do Intel C compiled binary and also AVX512 compiled (both VS2019 and IC 19) to see if it will benefit from AVX512 larger register file. The main program text is intrinsics-based so depend on C-compiler used. Also may be someone having LLVM compiler may try to do AVX2-limited and AVX512-limited compiled binaries to test performance.

I don't know if you are using AvsPmod.
Have here a test version to allow a real test (not only fps measured without display).

Under menu 'Options' there are the 2 entries
'Prefetch RGB display conversion' = Prefetch(2,2)
'Fast YV12 display conversion' = Your plugin

Your plugin is used when:
The function exists, the option is turned on, the video is YV12, the video width is mod 64.

For me with a real 4K video YUV420P10 and ConvertBits(8)
ConvertToRGB32()
~23 fps

ConvertToRGB32() with option 'Prefetch display conversion'
~32 fps

DecodeYV12toRGB(threads=1, matrix=1, gain=74, offset=-16) no prefetch
~40 fps

With option prefetch it becomes jerky ~46 fps then 30 fps and so on, I think that then my RAM or Bus limit is exceeded.

It is a pity that, as you have already written, the quality is only "simple".
I compared it with a 4K video and saw small differences... so this AvsPmod version will probably remain the only one that contains this function.

Too bad, but I think no one will want to use it in AvsPmod for preview.
Make one with 'bicubic'... it can be slower.... must only be faster than ConvertToRGB32. :)

https://drive.google.com/file/d/1Q6lfOZmMcW8KojRCcd1JF3_z1I-RreES/view?usp=sharing

[DTL]

"I don't know if you are using AvsPmod."

I not know what is AvsPmod (expect it is some script editor with some preview window and some frame number control ?). Also pinterf also ask where is this strange 4K ConvertToRGB32 is used - https://github.com/AviSynth/AviSynthPlus/issues/354#issuecomment-1549689053 . May you can write all required for AvsPmod directly at github ?

"the video width is mod 64."

It may be tested with other widths (or plugin may be simply fixed for +1 64-columns process). As I see with Asd-g plugins he also uses some mod64 processing for all widths and it may be feature of new AVS+ core to provide mod64 real row pitch for all frame widths.

"and ConvertBits(8)"

May be better to design version with input in 10..16 bits and skip this 1 more convert. It is anyway internally upconverted to 16bit to process. Or make 10->8 internally with close to zero time (right bitshift to 2). Will try later.

[Dogway]

Yes - it is limitedYUV->limitedRGB.

Can you change range using GDI tools (programming of display accelerator if possible) ?

The ConvertToRGB32() possibly write Full range, but it cause clipping of superwhites (it is no good for monitoring anyway). Also it maps 16lvl to black. It helps to get full contrast on sRGB typical PC display.

I take coefficients for matrix from https://gist.github.com/yohhoy/dafa5a47dade85d8b40625261af3776a .

I not know now if it possible to make Full range (or partial-Full like 16..254 maps to 0..255 RGB without clipping of superwhites) playing with coefficients only. So no change to current processing engine from its current performance. If add special conversion to Full (standard mapping of 16..235 to 0..255 RGB) it will be somehow slower and also somehow lower in precision. Though the performance penalty may be invisible. Need to try.


You can use coefficients for range conversion within the transformation matrix. Take a look at my YUV_to_RGB() (https://github.com/Dogway/Avisynth-Scripts/blob/8c45afa6142e043a0ff83148e48ad9db3f316968/TransformsPack%20-%20Models.avsi#L212) function.

In this case using Rec709 primaries with D65 illuminant, for TV to PC range the coefficients are:
m=[1.16895, 0 , 1.799772,\
1.16895,-0.214149,-0.535015,\
1.16895, 2.120637, 0]
Then the conversion goes as follows. You still have to substract footroom to Y and center chroma to 0 for the matrix to work.

ConverttoYUV444(chromaresample="bicubic",param1=0.1750,param2=0.4125)
YUV=ExtractClip()
bi=8
range_PC = "ymin - "
range_TV = ""
UVf = bi < 32 ? "range_half - " : ""

Expr(YUV[0],YUV[1],YUV[2], ex_dlut("x "+range_PC+" "+string(m[0])+" * "+range_TV+" z "+UVf + string(m[2])+" * + ", bi, false), \
ex_dlut("x "+range_PC+" "+string(m[0])+" * "+range_TV+" y "+UVf + string(m[4])+" * + z "+UVf + string(m[5])+" * + ", bi, false), \
ex_dlut("x "+range_PC+" "+string(m[0])+" * "+range_TV+" y "+UVf + string(m[7])+" * + ", bi, false), optSingleMode=true, format=bi>16?"RGBPS":"RGBP"+string(bi))

[DTL]

"DecodeYV12toRGB(threads=1, matrix=1, gain=74, offset=-16) no prefetch
~40 fps"

With versions 0.2.x and later you can try additional options cl/cs true/false (all 4 cases may be tested) for aditional performance tuning. Typically cs=false make things faster and cl=true/false depend on CPU chip and other.

[FranceBB]

I not know what is AvsPmod

Said DTL, without the blink of an eye, not knowing he was talking to the current and only AVSPmod mod maintainer, Gispos XD


Test results at i5-11600 (AVX512 builds just very slightly faster, so looks most of data fit AVX2 register file well) with 2ch of DDR4-3200 RAM:

1thread cached store 255 fps, non-cached store 268 fps
2thread cached store 400 fps, non-cached store 532 fps
4thread cached store 340 fps, non-cached store 811 fps

at Xeon Gold 6134 (looks like 6ch of DDR4-2666 ?)
1thread cached store 256 fps, non-cached store 295 fps
2thread cached store 488 fps, non-cached store 596 fps
4thread cached store 694 fps, non-cached store 1175 fps
6thread cached store 760 fps, non-cached store 1570 fps (ConvertToRGB32 is 216 fps)
and some hyperthreading example:
8thread non-cached store 1725 fps
10thread non-cached store 1780 fps




Interesting. Tomorrow I'll try to run some benchmarks too with the normal AVSPmod mod version (current ConverttoRGB32() function) and the new yv12 to RGB32 function.
p.s sorry DTL for "ghosting" you on PMs, but believe me, I clock in the office at 08.00AM and I leave at 07.00PM these days, yet I never find time to do the stuff I wanna do... :(

[DTL]

"Tomorrow I'll try to run some benchmarks"

Do your company plan to buy new Xeon Max platforms ? In 2023 or 2024 ? It will be nice to see how the all AVS software will run there.

[FranceBB]

Do your company plan to buy new Xeon Max platforms ? In 2023 or 2024 ?

I wish...
I think it will all depend on whether we'll get the football rights back in Italy.
Currently they're saying "no" to any request I put forward... :(

[gispos]

"the video width is mod 64." ... "and ConvertBits(8)"

These were your specifications. YV12 and width mod 64.
To be able to use your plugin.

Update, I had tested with a video YUV420P10:
video=LWLibavVideoSource(SourceFile, cache=True, indexingpr=False, format="")
audio=LWLibavAudioSource(SourceFile, cache=True)
audioDub(video, audio)
ConvertBits(8)

That was ~40 fps with your plugin

But if I do this with ColorBars(3840, 2160, "YV12") these are over 100 fps to ConvertToRGB32() with only ~33 fps

Apparently the source filter for this video takes quite a long time.
Must compare this with some other videos and with DirectShowSource.

So at over 100 fps (I had 120) I'm thinking of leaving this option.

[pinterf]

Anyway, I added AVX2 code path for YV24 to RGB32/RGB24 conversion. My gain (i7-11th gen) was typically +50% fps shown by AvsMeter.
At least the 2nd part of the ConvertToRGB32 conversion chain is enhanced.

[DTL]

And what about caching control for storing ?

[pinterf]

And what about caching control for storing ?
In my opinion this feature is too specific need, at least to involve it into the parameters of the filters. I'd better leave Avisynth to be independent of actual processor architectures and operating systems, even if it would seem that introducing a hardware specific parameter makes it quicker for some actual processors in 2023.

[gispos]

Anyway, I added AVX2 code path for YV24 to RGB32/RGB24 conversion. My gain (i7-11th gen) was typically +50% fps shown by AvsMeter.
At least the 2nd part of the ConvertToRGB32 conversion chain is enhanced.
I had noticed that if a ConvertBits(8) is used before the ConvertToRGB32() it becomes faster for formats > 8bit.
Does this have an influence on the quality? Is that still needed with the update?

[DTL]

In my opinion this feature is too specific need, at least to involve it into the parameters of the filters. I'd better leave Avisynth to be independent of actual processor architectures and operating systems, even if it would seem that introducing a hardware specific parameter makes it quicker for some actual processors in 2023.

If adding of cache control at end-user side with more filter params is difficult - it may be recommended to make fixed non-cached store because in all current tests it shows about twice more memory performance and may only add some penalty with very small frame sizes (fitting in L2/L3 caches x number_of_threads). So for processing of FHD/UHD1/UHD2 frame sizes with massive-multicore CPUs in 202x it may add to general performance.

The _stream_si256() instruction (intrinsic) is not depend on OS used and also work in the AVX2 instructions set.

I had noticed that if a ConvertBits(8) is used before the ConvertToRGB32() it becomes faster for formats > 8bit.
Does this have an influence on the quality? Is that still needed with the update?

Processing in >8 bit typically will create better precision/quality. You can compare with Subtract() and see if you got more or less precision/quality loss.

[pinterf]

I had noticed that if a ConvertBits(8) is used before the ConvertToRGB32() it becomes faster for formats > 8bit.
Does this have an influence on the quality? Is that still needed with the update?

Case:
444 source over 8 bits
Target: RGB32 or RGB24

Conversion chain:
- 4:4:4 -> Planar RGB (bit depth kept)
- ConvertBits(8)
- 8 bit Planar RGB -> RGB32/24

But when you convert the source YUV clip to 8 bits before ConvertToRGB32, then direct YV24->RGB32 process is done. (No planar RGB workaround)

Reasons:
- Packed RGB exists only at 8 or 16 bits
- YUV444P16 to RGB64 is not implemented in SIMD code, only in slow C.

In my AVX2 update only the last element of a 8 bit chain YV24->RGB32/24 was made quicker.

[DTL]

"direct YV24->RGB32 process is done."
"only the last element of a 8 bit chain YV24->RGB32/24 was made quicker."

Example of AVX2 YV24 to RGB32 64 columns per SIMD pass started from

https://github.com/DTL2020/ConvertYV12toRGB/blob/bd88a2b5be6c84bb5650860eafb14fa6a4ea2716/DecodeYV12toRGB.cpp#L226

(point resize of UV of YV12 to YV24 is https://github.com/DTL2020/ConvertYV12toRGB/blob/bd88a2b5be6c84bb5650860eafb14fa6a4ea2716/DecodeYV12toRGB.cpp#L214 to https://github.com/DTL2020/ConvertYV12toRGB/blob/bd88a2b5be6c84bb5650860eafb14fa6a4ea2716/DecodeYV12toRGB.cpp#L224 )

May be transfer it to AVS core as option for AVX2 processing ? Though I not sure if it keep same precision for dematrix.

Though still slow UV upsize from YV12 to YV24 if using standard AVS resampling engine may eat most of performance gain.

[flossy_cake]

Is there any possibility of Avisynth supporting namespaces at some point in future?

I need to use globals for reliable messaging between frames in ScriptClip, and this is eating a lot of namespace.

Also user cannot make multiple calls to my function because both calls would be writing to the same globals.

ScriptClip(local=true/false) doesn't seem to work, I'm guessing due to caching or something. It only seems to work if I declare globals outside and before the ScriptClip, then I can write to them inside the ScriptClip and the messaging between frames seems to work properly. Maybe I am doing something wrong here, I will keep trying.

I tried to save namespace by using a single array containing all my globals but couldn't get it to work as there doesn't seem to be any way to SET an element of an array through a string-key, only through an integer-index. It is possible to GET an element of the array through a string-key, though, but it must be explicitly defined before hand like eg. dictionary = [["one", 1], ["two", 2]] then you can go like dictionary["one"] but to set dictionary["one"] to something is not possible, only ArraySet(dictionary, 1, 0) to write integer 1 to index 0. Cannot go ArraySet(dictionary, 1, "one").

For now I will just prefix all my globals with some random string that nobody would use like


global 5dl31h_prevFrameYDiff
global 5dl31h_prevFrameUDiff
global 5dl31h_prevFrameVDiff

[pinterf]

"direct YV24->RGB32 process is done."
"only the last element of a 8 bit chain YV24->RGB32/24 was made quicker."

Though still slow UV upsize from YV12 to YV24 if using standard AVS resampling engine may eat most of performance gain.

The main two differences why the quick YV12->YV24 chroma is not used: default UV resampler is not pointresize. Secondly the default chroma location is "mpeg" ("left") and not "center".

The YV12->YV24 U and V resize method would recognize and work upon the very quick special case only when both parameters are set properly: pointresize and center.

(Overlay was using this method when some of its modes were supported only in 4:4:4 mode internally; Any 4:2:0 and 4:2:2 from and to conversion used this special conversion
See here: https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/filters/overlay/444convert.cpp
)

[FranceBB]

Happy 23rd Birthday, Avisynth! :D

https://i.imgur.com/nJNOO0h.png

(at least according to Wikipedia, 19th of May 2000)
I wasn't here on Doom9, back then, of course, as I only started using this amazing frameserver in June 2006 as a home user (and professionally only in 2013), but I'm sure some of the people who are still here were.
The world has changed a lot since then and so did Avisynth, constantly evolving towards a better future.

[StainlessS]

Is there any possibility of Avisynth supporting namespaces at some point in future?

I need to use globals for reliable messaging between frames in ScriptClip, and this is eating a lot of namespace.

Also user cannot make multiple calls to my function because both calls would be writing to the same globals.

ScriptClip(local=true/false) doesn't seem to work, I'm guessing due to caching or something. It only seems to work if I declare globals outside and before the ScriptClip, then I can write to them inside the ScriptClip and the messaging between frames seems to work properly. Maybe I am doing something wrong here, I will keep trying.

I tried to save namespace by using a single array containing all my globals but couldn't get it to work as there doesn't seem to be any way to SET an element of an array through a string-key, only through an integer-index. It is possible to GET an element of the array through a string-key, though, but it must be explicitly defined before hand like eg. dictionary = [["one", 1], ["two", 2]] then you can go like dictionary["one"] but to set dictionary["one"] to something is not possible, only ArraySet(dictionary, 1, 0) to write integer 1 to index 0. Cannot go ArraySet(dictionary, 1, "one").

For now I will just prefix all my globals with some random string that nobody would use like


global 5dl31h_prevFrameYDiff
global 5dl31h_prevFrameUDiff
global 5dl31h_prevFrameVDiff


Hi Flossy,
Martin53 and Gavino developed a multi-instance method of scripting. .

Here, two templates

Bare_MI.avs

Function BARE_MI(clip c,Bool "Show") {
myName="BARE_MI: "
IsAvsPlus=(FindStr(UCase(versionString),"AVISYNTH+")!=0) HasGScript=RT_FunctionExist("Grunt")
Assert(IsAvsPlus||HasGScript,RT_String("%sEssential either GScript or AVS+",myName))
Assert(RT_FunctionExist("GScriptClip"), RT_String("%sEssential GRunt installed"),myName))
Assert(RT_FunctionExist("MSuper"),RT_String("%sEssential MvTools2 installed"),myName))
FuncS="""
Function Fn@@@(clip c,Bool Show,String Fmt) {
c n = current_frame
Return Last
}
#######################################
# Unique Global Variables Initialization
#######################################
Global Prev@@@= -666 # Init vars, Prev = -666 forces initalize
#######################################
# Unique Runtime Call, GScriptClip must be a one-liner:
#######################################
ARGS = ""
c.GScriptClip("Fn@@@(last, "+ARGS+")", local=true, args=ARGS)
"""
#######################################
# Unique Identifier Definition
#######################################
GIFunc ="BARE_MI" # Function Name, Supply unique name for your multi-instance function.
GIName =GIFunc+"_InstanceNumber" # Name of the Instance number Global
RT_IncrGlobal(GIName) # Increment Instance Global (init to 1 if not already exists)
GID = GIFunc + "_" + String(Eval(GIName))
InstS = RT_StrReplace(FuncS,"@@@","_"+GID)
# RT_WriteFile("DEBUG_"+GID+".TXT","%s",InstS) # UnComment to write each unique script function instance to log file
HasGScript?GScript(InstS):Eval(InstS):
# if CallCmd available, Auto delete DBase file on clip closure.
#RT_FunctionExist("CallCmd")?CallCmd(close=RT_String("""CMD /C chcp 1252 && del "%s" """,DB), hide=true, Synchronous=7,debug=true):NOP
}


MultiInstanceTemplate.avs

Function FFMS_MakeMultiPartScript(clip c) {
c myName="FFMS_MakeMultiPartScript: "
IsAvsPlus=(FindStr(UCase(versionString),"AVISYNTH+")!=0) HasGScript=RT_FunctionExist("GScript") HasGrunt=RT_FunctionExist("GScriptClip")
Assert(IsAvsPlus || HasGScript,myName+"Essential Avs+ OR GScript installed")
Assert(HasGrunt,myName+"Essential GRunT plugin installed, http://forum.doom9.org/showthread.php?t=139337")
Show=Default(Show,False)
Fmt = "%d ] c1AveLum=%6.2f" # Make format string only once, not at every frame
FuncS="""
Function Fn@@@(clip c,Bool Show,String Fmt) {
c
n = current_frame
If(Prev@@@ == n) { # Cache failure, requested same frame again.
RT_DebugF("%d ] Cache Failure",n,name="Fn@@@_DBUG: ")
} Else If(Prev@@@ + 1 != n) { # Init OR Rewind OR User jumped about, dont you just hate users!
if(n == 0) {
if(Prev@@@ == -666) {
RT_DebugF("%d ] Initialized to frame 0",n,name="Fn@@@_DBUG: ")
} else {
RT_DebugF("%d ] Rewind to frame 0",n,name="Fn@@@_DBUG: ")
}
} else {
RT_DebugF("%d ] User Jumped About",n,name="Fn@@@_DBUG: ")
}
}
if(Show) {
RT_Subtitle(Fmt,n, c.RT_AverageLuma(n=n))
}
Global Prev@@@=n # Previous frame (For next interation jump about detect)
Return Last
}
#######################################
# Unique Global Variables Initialization
#######################################
Global Prev@@@= -666 # Init vars, Prev = -666 forces initalize
#######################################
# Unique Runtime Call, GScriptClip must be a one-liner:
#######################################
ARGS = "Show,Fmt"
c.GScriptClip("Fn@@@(last, "+ARGS+")", local=true, args=ARGS)
"""
#######################################
# Unique Identifier Definition
#######################################
GIFunc ="FFMS_MMPS" # Function Name, Supply unique name for your multi-instance function.
GIName =GIFunc+"_InstanceNumber" # Name of the Instance number Global
RT_IncrGlobal(GIName) # Increment Instance Global (init to 1 if not already exists)
GID = GIFunc + "_" + String(Eval(GIName))
InstS = RT_StrReplace(FuncS,"@@@","_"+GID)
# RT_WriteFile("DEBUG_"+GID+".TXT","%s",InstS) # UnComment to write each unique script function instance to log file
HasGScript ? GScript(InstS) : Eval(InstS) # Use GSCript if installed (loaded plugs override builtin)
Return Last
}


If you need any follow-up, then start new thread.

EDIT: MultiInstanceTemplate.avs stolen from here:
https://forum.doom9.org/showthread.php?t=176386&highlight=MakeMultiPartScript

EDIT: Some script names ending in "_MI"
DirtBox_MI :- https://forum.doom9.org/showthread.php?t=175708
Dupped_MI :- https://forum.doom9.org/showthread.php?p=1716968#post1716968
MorphDupes_MI_1.01:- https://forum.doom9.org/showthread.php?p=1764865#post1764865
DetectSub_MI :- https://forum.doom9.org/showthread.php?p=1782036#post1782036

EDIT: The "@@@" parts of function/variable names are given unique number as per multi-instance function instance.

EDIT: There is also this, for creating unique names (file or variable).

RT_LocalTimeString(Bool "file"=True)
Returns current local time as a string.
Where digits Y=Year, M=Month, D=Day, H=Hour, M=Minute, S=Second, m=millisecond.
When bool file==False, then string in format "YYYY-MM-DD HH:MM:SS.mmm"
When bool file==True (Default) string is in format "YYYYMMDD_HHMMSS_mmm"
Also when file==True, function first waits until the system tick count is incremented (about every 10ms)
before inquiring system time. This is to prevent 2 consecutive calls returning the same time string.
Perhaps useful for temporary filename generation.


EDIT:
And see Gavino stuff here [try move as much code as you can out of Scriptclip script and into function].
GRunT does not change the behaviour of ScriptClip regarding string usage, even when local scope is used, as string memory in Avisynth is orthogonal to the scope of variables and, as you say, is not released until script destruction.

The RTE script string (as a whole) is created only once when the containing script is loaded. However, that string itself is parsed afresh on every frame, which means that any identifiers and string literals within it are repeatedly added to the string heap.

Usually this is not significant, but for large run-time scripts, coupled with lots of source frames, it can add up. In fact, I discovered this was the source of a memory leak in SRestore (see here).

The solution is to move the code inside the run-time script to another function, reducing the run-time script itself to a simple function call. This effectively eliminates memory problems, and also gives a speed increase.

In other words, instead of
ScriptClip("""
... very long script ...
""")
use
function f(... some params ...) {
... previous script code ...
}
...
ScriptClip("f(...)")
Unless using GRunT, current_frame needs to be passed as a parameter to the function. (In GRunT, this is a [i]global variable.)

EDIT: Thread where some of the multi-instance stuff was debated.
Runtime variables scope and lifetime:- https://forum.doom9.org/showthread.php?p=1650250#post1650250

[flossy_cake]

Thanks for the info StainlessS

[DTL]

Asd-g make some clang and icx builds of DecodeYV12toRGB:

_clang_avx2 - arch:avx2
_clang_avx512 - arch:avx512
_icx_avx2 - arch:avx2 (intel c++ compiler 2023)
_icx_avx512 - arch:avx512 (intel c++ compiler 2023)

https://github.com/Asd-g/AviSynth-vsTTempSmooth/files/11517940/DecodeYV12toRGB.zip

May be they run somehow faster VS2019 builds ?

Also pinterf wrote next point about lower quality of DecodeTV12toRGB: It currently uses more data to process per SIMD pass but with only 16bit intermediate precision. And AVS+ internal YV24 to RGB dematrix uses 32bit intermediates - so AVS+ ConvertToRGB32 provide better precision.

So it looks we can have 2 version of dematrix functions:

1. 16bit internal processing with higher performance but lower precision.
2. 32bit internal processing with somehow lower performance but better precision.
3. float32 highest precision and probably lowest performance.

AVX2 with 512 bytes registerfile looks like allow to process 64 samples (colour - YV24 and output RGB) with 16bit intermediates or 32 samples with 32bit intermeadiates (it is version to try for performance check).

AVX512 with 2048 bytes registerfile looks like allow to process 256 samples (colour - YV24 and output RGB) with 16bit intermediates or 128 samples with 32bit intermeadiates (both need to be implemented and check for performance).

[gispos]

Asd-g make some clang and icx builds of DecodeYV12toRGB...
Thanks to all contributors!

I am so free and post it here and not in the AvsPmod thread.

Everything with zoom 100%, dll version icx_avx2

Video 1920 x 1080 YV12, playback (also with display drawing)
ConvertRGB = ~95 fps
DecodeRGB = ~112 fps
I can not see any visual difference.

ColorBars(width=3840, height=2160, pixel_type="YV12"), also playback
ConvertRGB = ~32 fps
DecodeRGB = ~100 fps

ColorBars(width=3840, height=2160, pixel_type="YUV420P10"), also playback
ConvertRGB = ~24 fps
DecodeRGB = ~92 fps

Well, for ColorBars DecodeRGB looks nicer (sharp edges).
RGB (255,255,255) becomes (253,253,253)
I took a snapshot in AvsPmod and compared.
The difference is barely visible on my monitor, and there is almost no difference in the other colors either.

If you use it you have to be careful to use the right dll with the right AVX version.
Had once tested the 512... crash.
Maybe it would be possible to create only one dll with checking the existing AVX version.

AvsPmod Pre-Release_7 also to test the dll
https://drive.google.com/drive/folders/1I7yNkFLoYmOush5Olx-jT799GphKcSwX?usp=share_link

https://i.postimg.cc/GmZHXFRH/Convert-To-RGB.gif (https://postimages.org/)

[flossy_cake]

Is there some way of getting Avisynth to print out the namespace of all global variables and function names, so that we know what names NOT to use?

I named my function AIT() but it doesn't work properly & I think it's a namespace clash with some other plugin that might also be declaring a function called AIT().

[DTL]

New version - https://github.com/DTL2020/ConvertYUVtoRGB/releases/tag/0.3.0

Now can take also 10..16bits YUV planar and convert to 8bit YV12 internally before processing.

With skipping ConvertBits(8) before YUV to RGB with one more full frame scan it also adds performance at out-of-cache frame sizes. If AVS can either have lots of ready to use SIMD functions inside for different conversions with single source scan or can dynamically compile required SIMD program for requested format conversion it can save from several host RAM read/writes for sequence of formats conversion and adds to performance.

At i3-9100T CPU both AVX2 Asd-g builds run a bit faster in compare with my VS2019 builds. So even close to pure SIMD program but intrinsics based (not fixed assembler) can have visibly different performance depending on C-compiler used even if it not very visibly run out of register file size.

About transients display - the internal AVS ColorBars(HD) are only about levels checking - not about transients for moving pictures. Maybe only for special designed pixel-art with moving pictures. So point-resize for UV work very well with pixel-art based output of ColorBars(HD). With natural footage for moving pictures it will be somehow different.

Also BicubicResize for UV default for ConvertToRGB32 with default b/c values can create some small over/undershoots with too sharp input from ColorBars(HD) data - it is expected.

[StainlessS]

Flossy,

From RT_Stats,

RT_VarExist(string)
Given the name (string) of the variable that you want to test for existence, returns true if exists, else false.
Eg, #a=32
RT_Debug(string(RT_VarExist("a"))) # would output 'false' to debugview unless '#a=32' uncommented. {Defined(a) would fail with error}.
return colorbars()

***
***
***

RT_FunctionExist(string)
Given the name (string) of the Function (plugin) that you want to test for existence, returns true if exists, else false.



Some builtin

Exist

Exist(filename)
Tests if the file specified by filename exists.

Examples:

filename = ...
clp = Exist(filename)
\ ? AviSource(filename)
\ : Assert(false, "file: " + filename + " does not exist")

Defined

Defined(var)
Tests if var is defined. Can be used inside Script_functions to test if an optional argument has been given an explicit value.
More formally, the function returns false if its argument (normally a function argument or variable) has the void ('undefined') type, otherwise it returns true.

Examples:

b_arg_supplied = Defined(arg)
myvar = b_arg_supplied ? ... : ...

FunctionExists

FunctionExists(name) AVS+
Tests if the function or filter name is defined in the script.
name can be any string – it does not need to be a legal name.
Example – see Apply below

InternalFunctionExists

InternalFunctionExists(name) AVS+
Tests if the function, filter or property name is defined natively within AviSynth+.

Unlike FunctionExists, returns false for external plugins and user-defined functions.

VarExist

VarExist(name) AVS+
Tests if the variable exists or not. Note: if variable exists, it returns true regardless of the "defined" state of the variable

http://avisynth.nl/index.php/Internal_functions#FunctionExists

EDIT: There is no way to query Global/Local scope, VarExist first scans Local list, and if var not found there, then scans Global list.
True and False are Global vars not constants, but Local vars 'Hide' Globals,
so inside function can do eg,
True = !True # (New True is Local and 'Hides' Global True).