ONNX models to use with avs-mlrt

[kedautinh12]

Originally Posted by FranceBB

Does it also happen with fp16=true?
I also have 8GB GDDR5 in my NVIDIA Quadro P4000 and I had to add that to compensate 'cause otherwise it would perform calculations in 32bit float and run out of memory.

Yes, the 32bit is very strong for even with 8gb vram GPU in 480p video

[Arx1meD]

Originally Posted by FranceBB

Does it also happen with fp16=true?

Yes. When fp16=true also appears errors.
I also noticed when the models are converted in the chaiNNer program with the selection fp32 everything works fine (for ESRGAN architecture models), and appears errors for the same model with the selection fp16.

[Stereodude]

FWIW, mlrt_ort using CUDA is faster than mlrt_ncnn to run an ONNX model (Dotzilla) on a RTX 3070Ti using a DVD source. I think it was 24fps vs. 18fps.

[takla]

Posting this example script here, so I don't forget to set colorspace stuff again...

Also, Cuda is waay slower for me, barely reaching 1/3 of the speed I get with DML.
This has also been shown by other to be the case.

With Hardware-accelerated GPU scheduling enabled, GPU usage often gets stuck at 80%.
If you run into this issue, you can disable it under Windows settings "System > Display > Graphics >> Advanced graphic settings"
Edit: Someone else ran into the same issue.

LWLibavVideoSource("C:\Users\ULTRA\Downloads\ffmpeg\INPUT.mkv").Prefetch(0)
z_ConvertFormat(pixel_type="RGBPS", colorspace_op="709:709:709:limited=>rgb:709:709:full", dither_type="error_diffusion", cpu_type="avx2", use_props=0).Prefetch(0)
mlrt_ort(network_path="C:\Program Files (x86)\AviSynth+\plugins64+\mlrt_ort_rt\models\2x_Ani4Kv2_G6i2_Compact_107500_fp32.onnx", builtin=False, provider="DML", fp16=True, num_streams=1).Prefetch(2)
z_ConvertFormat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:full=>709:709:709:limited", dither_type="error_diffusion", cpu_type="avx2", use_props=0).Prefetch(0)

[takla]

Exciting news:
ONNX is coming to FFMPEG

[DTL]

A question to NN/mlrt filters developers:

Currently RIFE uses internal float32 range remap to x255.0f (0..1 to 0..255) and all other (3) filters form avs-mlrt uses direct data upload from AVS to accelerator (in its range of 0..1).
Is it a special feature required by RIFE models/processing ? Can we use direct AVS 0..1 float range data with RIFE without issues ? Or RIFE models were specifically learned on 0..255 old PC BMP data and can not process 0..1 data range with same quality ?

The idea is to add new filters interconnection mode to AVS like GetFrameToBuffer() to direct data write into allocated upload buffer to accelerator instead of large RGBPS data copy from AVS cache to allocated upload buffer by CPU.

[WolframRhodium]

Originally Posted by DTL

Currently RIFE uses internal float32 range remap to x255.0f (0..1 to 0..255)

What do you mean by that? It uses 0..1 float range.

Originally Posted by DTL

The idea is to add new filters interconnection mode to AVS like GetFrameToBuffer() to direct data write into allocated upload buffer to accelerator instead of large RGBPS data copy from AVS cache to allocated upload buffer by CPU.

I am not a fan of this. We can hide the latency of data transfer if it is shorter than the computation time, without changing existing software stack. (Of course, this could be necessary for lightweight models, but I don't think it is significant in the near future.)

[DTL]

Originally Posted by WolframRhodium

What do you mean by that? It uses 0..1 float range.

At input it accepts 'standard' AVS float32 range of about 0..1 but at making copy into allocated by accelerator library buffer it makes additional range remap to 0..255 - https://github.com/Asd-g/AviSynthPlu.../rife.cpp#L379
in0R[w * y + x] = src0R[src_stride * y + x] * 255.0f;
in0G[w * y + x] = src0G[src_stride * y + x] * 255.0f;
in0B[w * y + x] = src0B[src_stride * y + x] * 255.0f;
in1R[w * y + x] = src1R[src_stride * y + x] * 255.0f;
in1G[w * y + x] = src1G[src_stride * y + x] * 255.0f;
in1B[w * y + x] = src1B[src_stride * y + x] * 255.0f;


The all other (3) avs-mlrt plugins to make a copy to upload buffer simply call avs_bitblt (copy without range remap).


The question: is it some obligatory feature of RIFE models or some old redundant operation and can be removed (if we can write directly to the upload buffer with 'standard' AVS float32 range mapping about 0..1) ?

"We can hide the latency of data transfer if it is shorter than the computation time, "

The idea is to free CPU and host memory bus resources. While we making some AVS filters processing we also make MPEG encoding with x26x software and it is also very slow. If we free some CPU + memory bus resources it is expected to have some performance boost with total processing of AVS+MPEG encoding (onCPU). Copy of the largest RGBPS format is very bad for cache and memory bus load.
So it is planned to test if we can simply make uncached stores at the output of
ConvertToPlanarRGB(bits=32) new filter in AVS+ core directly in the upload buffer of accelerator.
It is the special type of filters without CPU-core-readback for processing. It is filters-adapters to feed data to external accelerator (read from host RAM via DMA ?).

Updated plugin expected like:

    ncnn::Mat in0;
    ncnn::Mat in1;
    in0.create(w, h, channels, sizeof(float), 1); // allocate upload buffers to accelerator
    in1.create(w, h, channels, sizeof(float), 1);
    float* in0R{ in0.channel(0) };
    float* in0G{ in0.channel(1) };
    float* in0B{ in0.channel(2) };
    float* in1R{ in1.channel(0) };
    float* in1G{ in1.channel(1) };
    float* in1B{ in1.channel(2) };

child->GetFrameToBuffer(frame_num, buffer_description(in0R,...)); // uncached write from CPU core 
to upload buffers for DMA to accelerator in large RGBPS format

    ncnn::VkCompute cmd(vkdev);

[StvG]

Originally Posted by DTL

At input it accepts 'standard' AVS float32 range of about 0..1 but at making copy into allocated by accelerator library buffer it makes additional range remap to 0..255 - https://github.com/Asd-g/AviSynthPlu.../rife.cpp#L379...

@DTL, AviSynthPlus-RIFE - https://github.com/Asd-g/AviSynthPlu...n/src/rife.cpp is completely different plugin than avs-mlrt - https://github.com/Asd-g/avs-mlrt/bl...mlrt_RIFE.avsi

[WolframRhodium]

Originally Posted by DTL

The idea is to free CPU and host memory bus resources.

There are tons of similar improvements that could make a massive difference, e.g. fusing pointwise operations like MakeDiff() with resize. And PCIe 5.0 is fast.

[tormento]

Originally Posted by WolframRhodium

And PCIe 5.0 is fast.

Remember that somebody hasn't the money to upgrade to a modern system.

[WolframRhodium]

Originally Posted by tormento

Remember that somebody hasn't the money to upgrade to a modern system.

Then similarly the gpu processing time will be longer than the data transfer time.

[tormento]

Originally Posted by WolframRhodium

Then similarly the gpu processing time will be longer than the data transfer time.

Did you use all the possible tricks on newer builds of AVS+ to decrease the overhead from GPU to CPU transfers to have the temporal part processed?

[DTL]

If host CPU load will be lower - user can set higher quality MPEG encoding params and get more quality at the same encoding time (limited by accelerator performance).

Currently no significant optimizations for data transfer from host memory to GPU in latest AVS+ builds. The only optimization in r4483 build is faster YUV to RGBPS transform with single YUV data load to CPU instead of separate ConvertBits(32) load/store stage.

Now users can call single ConvertToPlanarRGB(bits=32) and it is faster in comparison with old sequence of
ConvertToPlanarRGB()
ConvertBits(32)
and also may be a bit higher in precision (in case of integer YUV dematrix before ConvertBits(32)).

[WolframRhodium]

Originally Posted by tormento

Did you use all the possible tricks on newer builds of AVS+ to decrease the overhead from GPU to CPU transfers to have the temporal part processed?

I do. I use an operating system trick (pinned host memory), a driver trick (CUDA graphs) and an x86-specific trick (write-combined memory, for CPU to GPU only). More tricks could be available for newer GPUs.