Topaz Video Enhance AI

[takla]

https://topazlabs.com/downloads/

Thoughts on this neat piece of software?

Personally, I think it outperforms all deblock/deblur/degrain and sharpening tools from avisynth/vapoursynth by a landslide.

Some benchmark:
Artemis High Quality v12 (1080p > 2160p) takes around 0.55 seconds per frame on my AMD RX 5700.

It is rather slow, but for the quality you get it is unmatched.

I also think all models but Artemis are redundant, because not only do they cause some artefacts (which artemis does not), they also are much slower (up to twice as slow). And just in general, scaling higher then 200% is pointtless.

[lansing]

For anime probably yes. But for live action, it'll be a hit or miss, with miss most of the time. And that processing time is just impractical. I'm not wasting a month just to upscale an one hour video, you are crazy.

[Zarxrax]

Haven't tried Video Enhance AI, but VSGAN for vapoursynth is basically the same thing, and there are lots of great user-generated models that can work with it.

[takla]

For anime probably yes. But for live action, it'll be a hit or miss, with miss most of the time. And that processing time is just impractical. I'm not wasting a month just to upscale an one hour video, you are crazy.

<0.6 sec per frame is slightly less then 13h for 20 minutes of 1080p to 2160p (sd 540p to 1080p is 0.1sec per frame, very doable). And that speed is with a midrange gpu from 2019. You can expect that time to be cut in half with the next generation of gpus.

Haven't tried Video Enhance AI, but VSGAN for vapoursynth is basically the same thing, and there are lots of great user-generated models that can work with it.
esrgan and vsgan are cuda based, which means they only work on nvidia gpus. so not possible for me to test. Edit: in the vsgan thread someone posted it takes around 1.4sec per frame. that is much longer then topaz, but depending on the model might work out much better.

[Zarxrax]

esrgan and vsgan are cuda based, which means they only work on nvidia gpus. so not possible for me to test. Edit: in the vsgan thread someone posted it takes around 1.4sec per frame. that is much longer then topaz, but depending on the model might work out much better.

Yeah, its highly dependent on the model. Depending on the model I'm using, I have seen differences of more than 5x in the encoding time. Even with the fastest models, its still quite slow.

[kedautinh12]

Waiting for avs port of vsgan

[lollo2]

Personally, I think it outperforms all deblock/deblur/degrain and sharpening tools from avisynth/vapoursynth by a landslide.

Personally, on my s-vhs captures I experimented that it is not any better than a full AviSynth restoration + upscale flow:

https://forum.videohelp.com/threads/401910-Topaz-products#post2620598

https://forum.videohelp.com/threads/399360-so-where-s-all-the-Topaz-Video-Enhance-AI-discussion/page9#post2626723

I used it just as an upscaler after the excellent avisynth denoise + sharpening, so Topaz VEAI takes advantage of the avisynth restoration; otherwise in my case is not even worth of consideration!

[PatchWorKs]

I repost the question here:

Hi there, I recently found this Upscale "render" that has an interesting workflow:

Flowframes (framerate interpolation): https://nmkd.itch.io/flowframes
Topaz Video AI (resolution upscale): https://www.topazlabs.com/video-enhance-ai
Transition Chooser to clean up scene changes and artifacting: https://www.magentacloud.de/share/lg6gou41fc

Judge result yourself:
https://www.youtube.com/watch?v=iUPVczZ2Xq4

...how to implement it with open source softwares only ?

Hope that helps/inspires.

[kedautinh12]

Personally, on my s-vhs captures I experimented that it is not any better than a full AviSynth restoration + upscale flow:

https://forum.videohelp.com/threads/401910-Topaz-products#post2620598

https://forum.videohelp.com/threads/399360-so-where-s-all-the-Topaz-Video-Enhance-AI-discussion/page9#post2626723

I used it just as an upscaler after the excellent avisynth denoise + sharpening, so Topaz VEAI takes advantage of the avisynth restoration; otherwise in my case is not even worth of consideration!

Are you sure it's better VSGAN??
https://rlaphoenix.github.io/VSGAN/

[lollo2]

I don’t know yet. Selur addressed me to some AI filters in VapourSynth, but I still have to try (TecoGAN should be the best candidate).

In any case, he seems to indicate that the VapourSynth AI model approach is better than Topaz VEAI (and in my case I am sure it is :))

[lollo2]

I repost the question here:

You already had the answer: https://forum.videohelp.com/threads/399360-so-where-s-all-the-Topaz-Video-Enhance-AI-discussion/page11#post2627997

Are you looking for something different?

[Selur]

side note: In theory TecoGan, iSeeBetter (and probably Topaz Video Enhance AI) are mainly interessting since they are designed to look at video and not single images and thus might produce better results since they also look at the adjustend frames and not a single frame.
Some folks first filter with Vapoursynth and then use Topaz Video Enhance AI, TecoGan, iSeeBetter in the hope to get a better results.

That said at least for Topaz Video Enhance AI on it's own I haven't seen anything so far that one couldn't archive or beat with Vapoursynth or Avisynth.

Problem with all the ML/AI stuff is simply the lack or control and ability to fine tune, Topaz tries to lessen this by:
a. providing different models
b. updating their models.
-> All this would be way more interessting if current graphic cards weren't so damn expensive. (and thus at least for me, who isn't earning money with it, it simply is not worth it)

Cu Selur

[lollo2]

That said at least for Topaz Video Enhance AI on it's own I haven't seen anything so far that one couldn't archive or beat with Vapoursynth or Avisynth.

I agree, as everybody who seriously tried both :)

[takla]

Personally, on my s-vhs captures I experimented that it is not any better than a full AviSynth restoration + upscale flow:

https://forum.videohelp.com/threads/401910-Topaz-products#post2620598

https://forum.videohelp.com/threads/399360-so-where-s-all-the-Topaz-Video-Enhance-AI-discussion/page9#post2626723

I used it just as an upscaler after the excellent avisynth denoise + sharpening, so Topaz VEAI takes advantage of the avisynth restoration; otherwise in my case is not even worth of consideration!

As I already said, people should use Artemis because it is the fasted and most tuned model. Furthermore, all AI models do denoising, so denoising with avisynth is pointless (will destroy too much fine detail if denoised twice). The same is true for sharpening (as long as your source is not too soft). Both of these things Artemis can do much better.

And personally, I'd save topaz output to prores (and use ffmpeg with more control for final encode)

Oh and for interlaced content I use bwdif because qtgmc makes the image look like a waxpainting.

[takla]

I repost the question here:

Hi there, I recently found this Upscale "render" that has an interesting workflow:


Judge result yourself:
https://www.youtube.com/watch?v=iUPVczZ2Xq4

...how to implement it with open source softwares only ?

Hope that helps/inspires.

Your youtube video is in 24 fps...

[lollo2]

As I already said, people should use Artemis because it is the fasted and most tuned model. Furthermore, all AI models do denoising, so denoising with avisynth is pointless (will destroy too much fine detail if denoised twice). The same is true for sharpening (as long as your source is not too soft). Both of these things Artemis can do much better.

As I already said, in my experiments Topaz VEAI is far inferior to AviSynth processing for denoising and sharpening. I gave it a second chance as an upscaler, it failed as well to improve over my AviSynth flow. In my case Topaz VEAI is useless (I posted some comparison and some restoration work in the posts I linked), but YMMV and, of course, every opinion is respectable ;-)

[kedautinh12]

Furthermore, all AI models do denoising, so denoising with avisynth is pointless (will destroy too much fine detail if denoised twice).

I think you still don't use KNLMeansCL and BM3D. Two denoiser is very good and don't detroy too much fine detail. It's still low but if you use GPU ver, it will faster

[takla]

I think you still don't use KNLMeansCL and BM3D. Two denoiser is very good and don't detroy too much fine detail. It's still low but if you use GPU ver, it will faster

I never got knlmeansCL to work on my gpu, and bm3d uses cuda, so it only works with a nvidia gpu, so I cannot test it. And for the cpu version of nlmeans, quality-to-speed ratio is inferior to what I can achieve with mvtools2 (I posted my denoise script in the mvtools2 thread) still, it is inferior to what I achieve with topaz.

[kedautinh12]

Bm3d have cpu ver. If you have cpu with many core, you will use prefetch() and got fast fps too
https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/issues/7#issuecomment-886173346

[takla]

Here are some comparison between: Raw (Original + Spline36), MDegrain (Spline36 + Adaptive Sharpen), Topaz AI (Artemis High Quality v12)

MDegrain uses Delta=4, thSAD=150, blksize=16 & overlap=8.

https://drive.google.com/drive/folders/189JbrysugCRFASdRPCj_WGZd6JwxYUoH

[kedautinh12]

Error 403, don't have permission

[lollo2]

Here are some comparison between: Raw (Original + Spline36), MDegrain (Spline36 + Adaptive Sharpen), Topaz AI (Artemis High Quality v12)
MDegrain uses Delta=4, thSAD=150, blksize=16 & overlap=8.

Here an additional comparison between a full AviSynth flow (raw capture + color correction + TemporalDegrain2 + nnedi3 + LSFmod ) and a Topaz flow (raw capture + color correction + Arthemis High Quality).

A quick video comparison (I was unable to upload the video, so used youtube, with all its limitations):

https://www.youtube.com/watch?v=vmHJyF_XxT0 (here is easy to find the winner)

Some image comparison, use zoom and fullscreen options to compare the details:

https://imgsli.com/NjcyMTE (here is easy to find the winner)

https://imgsli.com/NjcyMTI (here AviSynth is better except in the red border of the panel, but looking to any other area, i.e. the hand and the harm of the girl, the difference is evident)

[kedautinh12]

In 2 example, i choice avisynth cause your topaz result too blur and banding color and avisynth better for removed artifacts in example 1. Maybe your hobbies in detail video and mine don't same

[takla]

Error 403, don't have permission

Fixed. For anyone comparing those images: Please download the whole folder and view with external viewer, google drive is pretty bad for this.

[takla]

Here an additional comparison between a full AviSynth flow (raw capture + color correction + TemporalDegrain2 + nnedi3 + LSFmod ) and a Topaz flow (raw capture + color correction + Arthemis High Quality).

A quick video comparison (I was unable to upload the video, so used youtube, with all its limitations):

https://www.youtube.com/watch?v=vmHJyF_XxT0 (here is easy to find the winner)

Some image comparison, use zoom and fullscreen options to compare the details:

https://imgsli.com/NjcyMTE (here is easy to find the winner)

https://imgsli.com/NjcyMTI (here AviSynth is better except in the red border of the panel, but looking to any other area, i.e. the hand and the harm of the girl, the difference is evident)

Yeah. Not much difference there, so I'd pick what ever is faster (which is most likely avisynth)

[lollo2]

Yeah. Not much difference there, so I'd pick what ever is faster (which is most likely avisynth)

Yes, I think that all our considerations really depends on the videos we have. That's why everybody should experiment on his own and do not follow any general guideline ;-)

[lollo2]

In 2 example, i choice avisynth cause your topaz result too blur and banding color and avisynth better for removed artifacts in example 1. Maybe your hobbies in detail video and mine don't same

As I said, I always prefer AviSynth, but I am working on s-vhs capture, so my results are very limited. If you use pristine blue-ray video, or something like that, you may find completely different results. Experiment yourself to find the best procedure ;-)

[MysteryX]

I looked at some of the sample images on their website, denoise, sharpen, upscale.

It tends to gives a plastic effect, and I genuinely hate over-processed pictures with that plastic effect.

[ognirats]

It won't even give out good results with anime since it has been trained on real life video

[ognirats]

If there is anything that I hate about this piece of software than that are its users. Bunch of gullible morons that think their compressed af standard definition source would sunddenly be 4K.

[takla]

If there is anything that I hate about this piece of software than that are its users. Bunch of gullible morons that think their compressed af standard definition source would sunddenly be 4K.

:o Yep

[PatchWorKs]

Some image comparison, use zoom and fullscreen options to compare the details:

https://imgsli.com/NjcyMTE (here is easy to find the winner)

https://imgsli.com/NjcyMTI (here AviSynth is better except in the red border of the panel, but looking to any other area, i.e. the hand and the harm of the girl, the difference is evident)
Can you please test same frames in SwinIR (https://github.com/JingyunLiang/SwinIR) an post results ? https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667 (https://colab.research.google.com/gist/JingyunLiang/a5e3e54bc9ef8d7bf594f6fee8208533/swinir-demo-on-real-world-image-sr.ipynb)

https://raw.githubusercontent.com/JingyunLiang/SwinIR/main/figs/real_world_image_sr.png

[lollo2]

Should I use this porting: https://github.com/HolyWu/vs-swinir ?

I do not have VapourSynth installed, so please allow some time for me to get familiar with it.

[PatchWorKs]

Should I use this porting: https://github.com/HolyWu/vs-swinir ?

I do not have VapourSynth installed, so please allow some time for me to get familiar with it.
You can, of course.

@Selur has already tested it, we're discussing it here:
here are a few examples:
https://i.ibb.co/Pg3nJBm/SwinIR.png (https://ibb.co/2dmQRXn)
used:
# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/DeinterlaceFilter/TIVTC/libtivtc.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/d2vSource/d2vsource.dll")
# source: 'E:\clips\VTS_02_1-Sample-Beginning.demuxed.m2v'
# current color space: YUV420P8, bit depth: 8, resolution: 720x480, fps: 29.97, color matrix: 470bg, yuv luminance scale: limited, scanorder: telecine
# Loading E:\clips\VTS_02_1-Sample-Beginning.demuxed.m2v using D2VSource
clip = core.d2v.Source(input="E:/Temp/m2v_5d36292e1f7f53fd6e26be51d50bbf8c_853323747.d2v")
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 29.97
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# Deinterlacing using TIVTC
clip = core.tivtc.TFM(clip=clip)
clip = core.tivtc.TDecimate(clip=clip)# new fps: 23.976
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
# DEBUG: vsTIVTC changed scanorder to: progressive
# cropping the video to 704x480
clip = core.std.CropRel(clip=clip, left=8, right=8, top=0, bottom=0)
from vsswinir import SwinIR
# adjusting color space from YUV420P8 to RGBS for VsSwinIR
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# resizing using SwinIR
clip = SwinIR(clip=clip, task="real_sr_large", scale=4, tile_x=352, tile_y=240, tile_pad=16, device_type="cuda", device_index=0) # 2816x1920
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1920, h=1474, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGB48 to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 23.976fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
# Output
clip.set_output()
https://i.ibb.co/dpykKQ0/Swin-IR-clip-task-classical-sr-scale-4.png (https://ibb.co/z2wPfSh)
https://i.ibb.co/k2D9LtV/Swin-IR-clip-task-lightweight-sr-scale-4.png (https://ibb.co/whQwnfZ)
https://i.ibb.co/51Qb71F/Swin-IR-clip-task-real-sr-scale-4.png (https://ibb.co/vcN0tcY)
https://i.ibb.co/pryQgfC/Swin-IR-clip-task-real-sr-large-scale-4.png (https://ibb.co/3Yzvwpt)

some more using RealSR_large:
https://i.ibb.co/YR6nGnf/swinir-real-sr-large-4.png (https://ibb.co/mDmZ7Zh)
https://i.ibb.co/M5TF29C/swinir-real-sr-large-4-2.png (https://ibb.co/rfPD7vw)
https://i.ibb.co/MSSQt2m/swinir-real-sr-large-4-4.png (https://ibb.co/C88DY1f)

Cu Selur

[kolak]

What is the difference in case of girl with the phone?
Different parameters?
Some look good, others not.
Overall bit too artificial look, enough to add some nice (real grain) and it's not bad.
What is the speed for eg. HD to UHD?

[PatchWorKs]

What is the difference in case of girl with the phone?
Different parameters?
Some look good, others not.
Overall bit too artificial look, enough to add some nice (real grain) and it's not bad.
Different models.

Try yourself (don't abuse): https://camo.githubusercontent.com/84f0493939e0c4de4e6dbe113251b4bfb5353e57134ffd9fcab6b8714514d4d1/68747470733a2f2f636f6c61622e72657365617263682e676f6f676c652e636f6d2f6173736574732f636f6c61622d62616467652e737667 (https://colab.research.google.com/github/Lin-Sinorodin/SwinIR_wrapper/blob/main/Demo.ipynb)

What is the speed for eg. HD to UHD?
Slow.
Maybe an optimized version will come.

Note that is an image (not video) upscaler yet.

[kolak]

Tried myself but had issue with CUDA.
At the end got scrip loading but black video.

I know it's very slow, but how slow. 1fps or 0.1fps?

[lollo2]

I did not succeed either, both trying to install VapourSynth + Vs-SwinIR or using your link.

If you wish to experiment, here are the images

Original video + Colormatrix("Rec.601->Rec.709") frame 1: https://drive.google.com/file/d/1VCkEBIx92_K-uw4JxibHurYQtga7tuAM/view?usp=sharing
Original video + Colormatrix("Rec.601->Rec.709") frame 2: https://drive.google.com/file/d/1YwsYDicB64-7svLVHPmlJKPRlLZ2BW-m/view?usp=sharing

For comparison

AviSynth denoise and upscale frame 1: https://drive.google.com/file/d/10ljqfv5oR-hbrE_bHJrLUNVTkOGxdhjj/view?usp=sharing
AviSynth denoise and upscale frame 2: https://drive.google.com/file/d/1Kz5PiSkeAadopXiBV6uO3UvnK66vO5IW/view?usp=sharing

Topaz VEAI "denoise" and upscale frame 1: https://drive.google.com/file/d/1xu0gpCH9gF9xFdv3QGG9tqN3WrJ_5mu8/view?usp=sharing
Topaz VEAI "denoise" and upscale frame 2:https://drive.google.com/file/d/12exmfDXDm5KN-mIavoxWOef1H7bKxpB4/view?usp=sharing

AviSynth denoise + Topaz VEAI "denoise" and upscale frame 1: https://drive.google.com/file/d/1d4Avx_10509gzWfsFM1mZRUDqpPUr7Ch/view?usp=sharing
AviSynth denoise + Topaz VEAI "denoise" and upscale frame 2: https://drive.google.com/file/d/1dleN_7VCh8HMNzW-7wIqLcfW6UWQC9IL/view?usp=sharing

[sixthofmay]

Has anyone gotten Vs-SwinIR working? I'd like to try it on a film that has only a VHS and low bitrate streaming release.

I followed some links and installed Miniconda but then lost the webpage link.

I'm experienced with Avisynth+, QTGMC and other deinterlacers, Virtualdub2, x264, TVEIA, Resolve Studio, etc. but know nothing about Vaporsynth. I had Covid19 and it's affected my ability to concentrate.

Dumb question but searching it was futile: Can Vaporsynth be installed at the same time as Avisynth+?

If anyone has Vs-SwinIR working, maybe you could write a step-by-step for Windows 10 that others could follow?

[Selur]

Can Vaporsynth be installed at the same time as Avisynth+?
Sure.

If anyone has Vs-SwinIR working, maybe you could write a step-by-step for Windows 10 that others could follow?
send me a pm and I can send you a link to:
a. my current Hybrid dev version (~720MB download, ~2.5GB install)
b. my current torchPy addon (~12GB download, ~17.5GB install)
last one is a portable Vapoursynth with torchPy and VSGAN, vsDPIR, vsRIFE,vsBasidVSR++, vsSwinIR and vsHINet in it.
First one is just my current dev version which can use the addon if it's extracted into the Hybrid/64bit-folder and thus replaces the Vapopursynth&Co that is shipped with Hybrid.

Cu Selur

[Emulgator]

Back to OP: Topaz VEAI 2.6.2 under Win10, i9-11900K 128GB, 3080RTX 16GB.
Downloaded all 186GB of models manually:
6432 files in C:\Users\<user>\AppData\Roaming\Topaz Labs LLC\Video Enhance AI\models
6 model families, each in 1..6 subversions with all their kernels, in 1..3 builds each.
If you don't preload everything VEAI will phone home for every model it sees fit and tries to get the next 30MB .tz file

Artemis: seems to be a one-stop shop for starters
Chronos: Good fps interpolation, see below
Dione: Deinterlacing, not mature yet I would say (knowing (Q)TGMC)
Gaia: one-in-all for CGI, I haven't tested yet.
Proteus: Fine tunable restoration for Pros, see below.
Theia: I haven't tested yet.

Used so far on Panasonic FT-3 encoder mud (1980x1080x25p on 25i)
Aiming to stay in 1920x1080, but with less camera artifacts.
Then maybe interpolate to 1920x1080x50p, maybe end in 1280x720x50p.

First decoding with DGDecode, then QTGMC for getting Chroma back onto Y, then into VEAI.
Proteus could repaint convincing seawater structure back, and resynthed details of hair and skin,
When overdone repeating patterns occurred, but not too bad !
Going back to source I started to hate the source more than the result.

Chronos 50p interpolation: knocks Mvtools2 out of the water, almost no morphing blotches.
In hard cases soft blends of details, but only visible while framestepping and pixelpeeping

Other source VHS on DV: Dione Deinterlacing: QTGMC still wins by far for now.
But with improving AI models almost everything seems within reach.

For me Proteus and Chronos are well worth it now and will stay in my daily toolbox.
Will not replace, but accompany Avisynth for the tasks where they shine.

Running stable here, Proteus V3 @ 4,5fps, Chronos V3 @ 6fps.
With such hard- and software a lot of projects will see a second AI edition now.

A few GUI unusual behaviours: Model Update management does list incomplete model sets only by filesize, not by files.
Hard to find out which is missing, you have to dump the complete set 3..6GB and try to download the complete set gain
Preview framecounter seems to be ready for input of framecount, but only renders the same amount 30 or 50.
One cannot save his preferences (why oh why do I have to input manually for every new clip)...

Reading up their forums it seemed to be much worse in the beginning
and from the responses it looks like only one developer responding, but still:
he answers, and there is definitely progress in GUI usability and I guess in number and usability of models too.

Output: Avoid .mov (ProRes 422HQ) for now, I got small blocks all over the sky.
If output avc in .mp4, give enough bits (I felt safe with CQ12)
and add grain (I used 1,8) otherwise you get blocks again.
Any fps model loses audio, even if duration stays.
Uncompressed .avi or VfW encoder interface: not yet implemented
.tiff frames ask for huge storage, but this is well known.

P.S. Just started redoing a paused film restoration from 720x576x25p DVD 5Mbps.
Proteus Dehalo: never expected that this can be achieved.
Clean transitions in 1280x720 and 1920x1080 while almost undisturbed by the source's blockfest.
I can happily trash my previous Dehaloing attempts now.

Just tried the unspeakable just to saturate the rendering system: Run 2 instances of VEAI in parallel. Works !

[lollo2]

Thanks for your report, interesting!

I am limited so far to experiments only on my S-VHS captures, but wish to see in the future how TopaVEAI behaves with 1920x1080 50p recording from my old Sony HDR-CX570E.

[takla]

I doubt prores causes such artefacts. the bitrate alone is basically lossless. those artefacts must be caused by the upscaling, or some bug inside the software.

I personally always pick prores as my output and never had any artefacts or other issues.

[Emulgator]

You may try it. Install it. Switch to Pro Res and its there. Use different output and it is gone.-> Not anymore, see below (Did I have seen apcn the first time ?)

(Where in the chain I did not consider, maybe the encoder implementation within VEAI or the decoding playback chain on given system.
Poor user will have it coming and going with that choice, often noticing when too late (looking at myself hit by various poor implementations of DV before finding the one that worked.)
So: just a suggestion to not step into any single source of pain while investing a good piece of effort, not voting against ProRes globally.)

P.S. Just tried it again today, running 2 instances of VEAI in parallel:
Can not repeat the fault. Mediainfo is showing apch, which should be good. Ok, so nevermind... I hope.

[takla]

Do you actually get more frames processed by running two apps at the same time (with your 3080)? For me it makes no difference since one instance already uses 100% of my gpu (amd radeon rx 5700).

[ReinerSchweinlin]

Do you actually get more frames processed by running two apps at the same time (with your 3080)? For me it makes no difference since one instance already uses 100% of my gpu (amd radeon rx 5700).

It depends on the combination of GPU/CPU/scale-factor/model/etc.... If it does not improve speed in your case - than thats it for you... In other usecases, I ran up to 6 instances and squeezed a little more speed out of it (the gain getting slower from instance to instance.... biggest speed bump with 2... )..
I have not found the "rule of thumb" yet in which situation what will happen - you simply have to try out on your card for yourself..

having some free ressources of course is an indicator... Having a lot of VRAM helps....

Also, this behaviour changed every time Topaz changed the engine slightly...

On thing you could tamper with is the "reduce load" setting and the VRAM. The first one lowers the thread count processed on the GPU and lowers the needed VRAM in some cases, the second one is obvious (if you reach a limit of tile size of the model variant, no change in VRAM will occur, no matter hat you set at the vram usage slider).

[Emulgator]

Single instance of Topaz VEAI:
The 11900K/3080 combo was hovering around 60% CPU/GPU with short peaks around 95% and dips around 40%.
2 instances of Topaz VEAI: Total load closer to 80..90% CPU/GPU, peaks around 100% and dips around 60%.
(Proteus numbers, IIRC, Chronos saturates 20% less.)
I did not try more than 2 instances, and each was served almost the same speed, well, like 80% as if they were singled.
So yes, a little improved saturation with that combo possible.

[ReinerSchweinlin]

Single instance of Topaz VEAI:
The 11900K/3080 combo was hovering around 60% CPU/GPU with short peaks around 95% and dips around 40%.
2 instances of Topaz VEAI: Total load closer to 80..90% CPU/GPU, peaks around 100% and dips around 60%.
(Proteus numbers, IIRC, Chronos saturates 20% less.)
I did not try more than 2 instances, and each was served almost the same speed, well, like 80% as if they were singled.
So yes, a little improved saturation with that combo possible.

Thats roughly what happened in my tests as well... Two instances got me around 60% to 90% overall speed increase over one single instance...

[takla]

Good to know!👍

[DMD]

Good morning.
As for the deinterlacing section, I noticed that all the presets (Dione) also operate a noise reduction and sharpness, is there any possibility to control this?
Thank you

[ReinerSchweinlin]

Good morning.
As for the deinterlacing section, I noticed that all the presets (Dione) also operate a noise reduction and sharpness, is there any possibility to control this?
Thank you

No, the Dione Models have no tuning parameters.

If you add Avisynth prior to VEAI and let Avisynth do the deinterlacing, you can choose whatever Model you want - andproteus gives you the choice of how much you want to add/reduce sharpness or noise, halo, etc...

[toot367]

Good morning.
As for the deinterlacing section, I noticed that all the presets (Dione) also operate a noise reduction and sharpness, is there any possibility to control this?
Thank you

I haven't seen v2.6.2.

FYI, QTGMC is no longer the best deinterlacer for standard definition. Dione Interlaced Robust v3 and Dione Interlaced Robust Dehalo v1 WITH the json mods to disable NR and sharpening provides much more detail for many SD videos. QTGMC still has value for 1080i though.

[poisondeathray]

FYI, QTGMC is no longer the best deinterlacer for standard definition. Dione Interlaced Robust v3 and Dione Interlaced Robust Dehalo v1 WITH the json mods to disable NR and sharpening provides much more detail for many SD videos. QTGMC still has value for 1080i though.


How is the bob flicker with the json mods ?

QTGMC's main reason for existence (and it's precursor, tempgaussmc_beta) is to reduce bob flicker, not "keep details". In fact, as a result of QTGMC's temporal smoothing, noise and details are lost - as expected - ie. it's a tradeoff . You can adjust QTGMC settings to retain more details at the expense of more flicker

[toot367]

How is the bob flicker with the json mods ?

Are you referring to Dione screwing up the field order so that the frame jumps up or down for a frame?

If so, it's not perfect. I had to fix about 10 jumps on a 92 minute VHS movie. I duplicated the frame in Windows Explorer and used FillDrops(thresh=0.4) to fix the duplicate judder. When it finds a duplicate, FillDrops interpolates the extra frame and replaces it.

It's a nifty function for VHS, because you can fix many glitches by duplicating the previous or next frame. I duped 312 glitched frames in the movie and FillDrops fixed them all. Note I had VEAI save to a png sequence specifically to work directly with the frames using Windows Explorer, Paint, Photoshop, and Irfanview. I got rid of the microphone sneaking into one scene too. There's probably a more efficient method.

I tried some script to find bad frames, but it was useless (FindBadFrames it was called). It generated a file with "bad" frames but none of them were bad. So I had to find them with several viewings. I wish there were an automated way.

With DVD video, I haven't had any issues.

[poisondeathray]

How is the bob flicker with the json mods ?


Are you referring to Dione screwing up the field order so that the frame jumps up or down for a frame?


No, just referring regular deinterlacing artifacts.

Such as aliasing, jaggies, when viewed in motion or over a few frame, it flickers.

The main reason QTGMC exists is to calm that flicker (at the expense of smoothing, detail loss)

[toot367]

Such as aliasing, jaggies, when viewed in motion or over a few frame, it flickers.

There are no jagges, artifacts, or flickering.

[Emulgator]

And here after some TB rendered: a small VEAI ingest/encode guide

VEAI 2.6.4 Do's and Don'ts

Besides of VEAI 2.6.4 solving most of its main tasks superior
one has to consider how to get the most out of it and not lose quality due to some unexpected behaviour.

First off: The recent Topaz VEAI 2.6.4 uses ffmpeg under the hood for decoding and encoding,

Input: I can happily feed .avs scripts
but
Output: Unfortunately all ffmpeg encoders have been left at defaults !

Audio: 2.0 Audio in .mp4 oder .mov will be transcoded to 96kbps aac.
The resulting 96kbps .aac I do consider unusable, hi-Freq morphing and mumbling at the same time.

Solution: Demux source Audio to 16bit .aiff using external ffmpeg, and remux into .mov after Topas VEAI using external ffmpeg !
Some NLEs, (Vegas 14 here) seems to be able to decode only 16bit .aiff

Video: The same situation.
All encoders are left at defaults (ffmpeg x264 core 157 r2970)
No CABAC, only CRF can be set.

CRF10 @ 1980x1080x50p: 157Mbps 1,524bpp NoCABAC 1Ref, GOP M=1, N=12 0,25s/f -> almost transparent
CRF12 @ 1980x1080x50p: 125Mbps 1,208bpp NoCABAC 1Ref, GOP M=1, N=12 0,22s/f -> blocks starting to appear
CRF14 @ 1980x1080x50p: 91,2Mbps 0,879bpp NoCABAC 1Ref, GOP M=1, N=12 0,22s/f -> fat blockiness
CRF16 @ 1980x1080x50p: 74,1Mbps 0,715bpp NoCABAC 1Ref, GOP M=1, N=12 0,22s/f -> unbearable blockiness
CRF20 @ 1280x720x29.97p -> 7Mbps -> Blockfest !
CRF20 @ 1280x720x59.94p -> 8,6Mbps -> Blockfest !

Go to regedit
Computer\HKEY_CURRENT_USER\SOFTWARE\Topaz Labs LLC\Video Enhance AI
This contains only the same settings you can reach via GUI.
So encoder parameters are really the ffmpeg hardcoded defaults...

Who wants to harvest quality has to use ProRes...

ProRes 422 10bit HQ @ 1980x1080x50p: 387Mbps 3,734bpp -> clean !

++++++++ 4K-Uprezzing ++++++++++
8-bit YUV444P8 Uncompressed mean 24bpp, 3840x2160x50p yield pixel rate 414Mpix/s
and need for 24bpp 10Gb/s. YUV422P8 would be 16bpp, -> 6,63Gb/s

ProRes 422 HQ 10bit @ 3840x2160x50p: 1545Mbps, 3,734bpp (23,5% of YUV422P8) 0,37s/f -> clean, faultfree und detailed picture !
BUT: Now a 43s short 147MB .MTS 140MB .264 explodes 55-fold to 8GB!
Well, a second thought comes up: Such huge files.
Can't we quick-transcode/remux ?

A RTX3080 can munch 4K-H264 or 4K-H265 at 33..34fps, then MP4 muxing
(MeGUI oder Simple Launcher, 2022-06-22 can mux H265 in .mp4)

NVENcC265 at 4K:
CQP10: 391/618Mbps, 0,942bpp, 25% .mov size: Water gradients usable
CQP12: 315/497Mbps, 0,759bpp, 20% .mov size: Water gradients stepping
CQP15: 203/388Mbps, 0,49bpp, 12,5% .mov size: Water gradients get horrible steps

NVEncC264 at 4K:
CQP10: 406/637Mbps 0,98bpp, (27% .mov size, 6,5% YUV422P8 uncompressed): Water gradients almost faultless, better as H.265.

Unfortunately Vegas 14 can't decode all these varieties, and still there is a visible quality loss.
So for me it is back to huge ProRes apch into Vegas 14.
1TB/h, well, so lets invest in even more storage X-|

P.S. Not NVEncC's fault, just tested Rigaya's 6.01 from within StaxRip 2.13 with improved tuning and it fares much better qualitywise,
I get almost transparent 4K50p quality out of 45Mbps, so I guess again: VEAI 2.6.4. left NVEncC at defaults...

[Emulgator]

BTW, Topaz are hiring now.
https://topazlabs.com/careers/
Dallas, TX. Anyone willing to help them out with a simple ffmpeg interface ?
I am not affiliated, BTW.

Or even better: proper deinterlacing as developed in this almighty forum ? ;_)

P.S. Seems that upcoming version 3 gets a better encoding preset system.

[miltonsoto658]

Back to OP: Topaz VEAI 2.6.2 under Win10, i9-11900K 128GB, 3080RTX 16GB.
Downloaded all 186GB of models manually:
6432 files in C:\Users\<user>\AppData\Roaming\Topaz Labs LLC\Video Enhance AI\models
6 model families, each in 1..6 subversions with all their kernels, in 1..3 builds each.
If you don't preload everything VEAI will phone home for every model it sees fit and tries to get the next 30MB .tz file

Artemis: seems to be a one-stop shop for starters
Chronos: Good fps interpolation, see below
Dione: Deinterlacing, not mature yet I would say (knowing (Q)TGMC)
Gaia: one-in-all for CGI, I haven't tested yet.
Proteus: Fine tunable restoration for Pros, see below.
Theia: I haven't tested yet.

Used so far on Panasonic FT-3 encoder mud (1980x1080x25p on 25i)
Aiming to stay in 1920x1080, but with less camera artifacts.
Then maybe interpolate to 1920x1080x50p, maybe end in 1280x720x50p.

First decoding with DGDecode, then QTGMC for getting Chroma back onto Y, then into VEAI.
Proteus could repaint convincing seawater structure back, and resynthed details of hair and skin,
When overdone repeating patterns occurred, but not too bad !
Going back to source I started to hate the source more than the result.

Chronos 50p interpolation: knocks Mvtools2 out of the water, almost no morphing blotches.
In hard cases soft blends of details, but only visible while framestepping and pixelpeeping

Other source VHS on DV: Dione Deinterlacing: QTGMC still wins by far for now.
But with improving AI models almost everything seems within reach.

For me Proteus and Chronos are well worth it now and will stay in my daily toolbox.
Will not replace, but accompany Avisynth for the tasks where they shine.

Running stable here, Proteus V3 @ 4,5fps, Chronos V3 @ 6fps.
With such hard- and software a lot of projects will see a second AI edition now.

A few GUI unusual behaviours: Model Update management does list incomplete model sets only by filesize, not by files.
Hard to find out which is missing, you have to dump the complete set 3..6GB and try to download the complete set gain
Preview framecounter seems to be ready for input of framecount, but only renders the same amount 30 or 50.
One cannot save his preferences (why oh why do I have to input manually for every new clip)...

Reading up their forums it seemed to be much worse in the beginning
and from the responses it looks like only one developer responding, but still:
he answers, and there is definitely progress in GUI usability and I guess in number and usability of models too.

Output: Avoid .mov (ProRes 422HQ) for now, I got small blocks all over the sky.
If output avc in .mp4, give enough bits (I felt safe with CQ12)
and add grain (I used 1,8) otherwise you get blocks again.
Any fps model loses audio, even if duration stays.
Uncompressed .avi or VfW encoder interface: not yet implemented
.tiff frames ask for huge storage, but this is well known.

P.S. Just started redoing a paused film restoration from 720x576x25p DVD 5Mbps.
Proteus Dehalo: never expected that this can be achieved.
Clean transitions in 1280x720 and 1920x1080 while almost undisturbed by the source's blockfest.
I can happily trash my previous Dehaloing attempts now.

Just tried the unspeakable just to saturate the rendering system: Run 2 instances of VEAI in parallel. Works !


Can i ask how did you download manually all the models?

[Emulgator]

Well, manually was not the correct term.
It was via GUI, repeatedly invoking and deleting incomplete downloads over the course of a few days.

[rgr]

25fps->50fps conversion + image enhancement (from youtube).

First Chronos, then Artemis Low Quality -- or vice versa?

[Emulgator]

Might be source dependent.
I go vice versa, first enhance/upscale then upfps.
Takes longer, but gives... better butter.

[James_b]

Emulgator, what do you think about a madVR-plugin? would be cool. Found some screenshots from huh'n https://forum.doom9.org/showthread.php?p=1967409#post1967409

[Emulgator]

ATM I see VEAI and other AI algos rather as a choice of offline reconstruction algos.
Once AI gets so cheap that it an be performanced realtime, maybe.
But I think that guessing at content (to such extent and in effect repainting it) is not what a renderer should do.
A few years later my today's opinion may well be overridden by development ;-)

To continue Topaz discussion:
I can not get any enlightening out of Version 3 so far up to 3.0.4:
Slower, clumsier behaviour,
(continues rendering costly preview threads under the hood, every time I have to untick something),
the two screens are not updated in realtime but stutter around (worked in 2.6.4),
still Deinterlacing can not do what I am inclined to believe it could,
"Ugly face syndrome" still unsolved
(while calcuating Proteus on them, if faces happen to be below a certain pixel size they degrade to ugly grimaces).

I haven't even tried to get the output options tested for better parameters.
And from this month on I would have to shell out the yearly price again to be updated...
So it is 2.6.4 for me still.

[ognirats]

There are no jagges, artifacts, or flickering.

Doubt

[James_b]

ATM I see VEAI and other AI algos rather as a choice of offline reconstruction algos.
Once AI gets so cheap that it an be performanced realtime, maybe.
But I think that guessing at content (to such extent and in effect repainting it) is not what a renderer should do.
A few years later my today's opinion may well be overridden by development ;-)


Hello Topaz master :)
I meant the other way around, mad vir in Topaz. super-xbr is pretty good. In Topaz one can not change the upscaler? as far as i concern. Only the clean up algos. Do you know what they use?

So back to the 2.6.4. And when speaking of 264, the x264-out have some what shallow dynamics, or is it only me? kicks of from 30.

And when playing some of my old avi i get 0-255 dynamics in many players, almost HDR (in dynamics). But the very good ProRes output from 16 black. Standard yes, but dynamics is not so much Enhance.

[Emulgator]

the x264-out have some what shallow dynamics, or is it only me? kicks of from 30.
Yes, depending on footage shadow values tended to brighten up a bit.
Could well have been a straight mismatch implemented while piping into ffmpeg x264.
Back then in a good number of cases I inserted a "lowering shadow values" gamma curve after Topaz (SmoothAdjust16),
but these days I am telling 2.6.4 to render into ProRes to lift further processing chain into 10bit precision.

super-xbr: I haven't tried seriously any other than VEAI after I got ridiculous and still watchable 450%..900% 4K upscales working, just for fun. But maybe I should someday...