Topaz Video Enhance AI

[Emulgator]

Back to OP: Topaz VEAI 2.6.2 under Win10, i9-11900K 128GB, 3080RTX 16GB.
Downloaded all 186GB of models manually:
6432 files in C:\Users\<user>\AppData\Roaming\Topaz Labs LLC\Video Enhance AI\models
6 model families, each in 1..6 subversions with all their kernels, in 1..3 builds each.
If you don't preload everything VEAI will phone home for every model it sees fit and tries to get the next 30MB .tz file

Artemis: seems to be a one-stop shop for starters
Chronos: Good fps interpolation, see below
Dione: Deinterlacing, not mature yet I would say (knowing (Q)TGMC)
Gaia: one-in-all for CGI, I haven't tested yet.
Proteus: Fine tunable restoration for Pros, see below.
Theia: I haven't tested yet.

Used so far on Panasonic FT-3 encoder mud (1980x1080x25p on 25i)
Aiming to stay in 1920x1080, but with less camera artifacts.
Then maybe interpolate to 1920x1080x50p, maybe end in 1280x720x50p.

First decoding with DGDecode, then QTGMC for getting Chroma back onto Y, then into VEAI.
Proteus could repaint convincing seawater structure back, and resynthed details of hair and skin,
When overdone repeating patterns occurred, but not too bad !
Going back to source I started to hate the source more than the result.

Chronos 50p interpolation: knocks Mvtools2 out of the water, almost no morphing blotches.
In hard cases soft blends of details, but only visible while framestepping and pixelpeeping

Other source VHS on DV: Dione Deinterlacing: QTGMC still wins by far for now.
But with improving AI models almost everything seems within reach.

For me Proteus and Chronos are well worth it now and will stay in my daily toolbox.
Will not replace, but accompany Avisynth for the tasks where they shine.

Running stable here, Proteus V3 @ 4,5fps, Chronos V3 @ 6fps.
With such hard- and software a lot of projects will see a second AI edition now.

A few GUI unusual behaviours: Model Update management does list incomplete model sets only by filesize, not by files.
Hard to find out which is missing, you have to dump the complete set 3..6GB and try to download the complete set gain
Preview framecounter seems to be ready for input of framecount, but only renders the same amount 30 or 50.
One cannot save his preferences (why oh why do I have to input manually for every new clip)...

Reading up their forums it seemed to be much worse in the beginning
and from the responses it looks like only one developer responding, but still:
he answers, and there is definitely progress in GUI usability and I guess in number and usability of models too.

Output: Avoid .mov (ProRes 422HQ) for now, I got small blocks all over the sky.
If output avc in .mp4, give enough bits (I felt safe with CQ12)
and add grain (I used 1,8) otherwise you get blocks again.
Any fps model loses audio, even if duration stays.
Uncompressed .avi or VfW encoder interface: not yet implemented
.tiff frames ask for huge storage, but this is well known.

P.S. Just started redoing a paused film restoration from 720x576x25p DVD 5Mbps.
Proteus Dehalo: never expected that this can be achieved.
Clean transitions in 1280x720 and 1920x1080 while almost undisturbed by the source's blockfest.
I can happily trash my previous Dehaloing attempts now.

Just tried the unspeakable just to saturate the rendering system: Run 2 instances of VEAI in parallel. Works !

[lollo2]

Thanks for your report, interesting!

I am limited so far to experiments only on my S-VHS captures, but wish to see in the future how TopaVEAI behaves with 1920x1080 50p recording from my old Sony HDR-CX570E.

[takla]

I doubt prores causes such artefacts. the bitrate alone is basically lossless. those artefacts must be caused by the upscaling, or some bug inside the software.

I personally always pick prores as my output and never had any artefacts or other issues.

[Emulgator]

You may try it. Install it. Switch to Pro Res and its there. Use different output and it is gone.-> Not anymore, see below (Did I have seen apcn the first time ?)

(Where in the chain I did not consider, maybe the encoder implementation within VEAI or the decoding playback chain on given system.
Poor user will have it coming and going with that choice, often noticing when too late (looking at myself hit by various poor implementations of DV before finding the one that worked.)
So: just a suggestion to not step into any single source of pain while investing a good piece of effort, not voting against ProRes globally.)

P.S. Just tried it again today, running 2 instances of VEAI in parallel:
Can not repeat the fault. Mediainfo is showing apch, which should be good. Ok, so nevermind... I hope.

[takla]

Do you actually get more frames processed by running two apps at the same time (with your 3080)? For me it makes no difference since one instance already uses 100% of my gpu (amd radeon rx 5700).

[ReinerSchweinlin]

Originally Posted by takla

Do you actually get more frames processed by running two apps at the same time (with your 3080)? For me it makes no difference since one instance already uses 100% of my gpu (amd radeon rx 5700).

It depends on the combination of GPU/CPU/scale-factor/model/etc.... If it does not improve speed in your case - than thats it for you... In other usecases, I ran up to 6 instances and squeezed a little more speed out of it (the gain getting slower from instance to instance.... biggest speed bump with 2... )..
I have not found the "rule of thumb" yet in which situation what will happen - you simply have to try out on your card for yourself..

having some free ressources of course is an indicator... Having a lot of VRAM helps....

Also, this behaviour changed every time Topaz changed the engine slightly...

On thing you could tamper with is the "reduce load" setting and the VRAM. The first one lowers the thread count processed on the GPU and lowers the needed VRAM in some cases, the second one is obvious (if you reach a limit of tile size of the model variant, no change in VRAM will occur, no matter hat you set at the vram usage slider).

[Emulgator]

Single instance of Topaz VEAI:
The 11900K/3080 combo was hovering around 60% CPU/GPU with short peaks around 95% and dips around 40%.
2 instances of Topaz VEAI: Total load closer to 80..90% CPU/GPU, peaks around 100% and dips around 60%.
(Proteus numbers, IIRC, Chronos saturates 20% less.)
I did not try more than 2 instances, and each was served almost the same speed, well, like 80% as if they were singled.
So yes, a little improved saturation with that combo possible.

[ReinerSchweinlin]

Originally Posted by Emulgator

Single instance of Topaz VEAI:
The 11900K/3080 combo was hovering around 60% CPU/GPU with short peaks around 95% and dips around 40%.
2 instances of Topaz VEAI: Total load closer to 80..90% CPU/GPU, peaks around 100% and dips around 60%.
(Proteus numbers, IIRC, Chronos saturates 20% less.)
I did not try more than 2 instances, and each was served almost the same speed, well, like 80% as if they were singled.
So yes, a little improved saturation with that combo possible.

Thats roughly what happened in my tests as well... Two instances got me around 60% to 90% overall speed increase over one single instance...

[takla]

Good to know!👍

[DMD]

Good morning.
As for the deinterlacing section, I noticed that all the presets (Dione) also operate a noise reduction and sharpness, is there any possibility to control this?
Thank you

[ReinerSchweinlin]

Originally Posted by DMD

Good morning.
As for the deinterlacing section, I noticed that all the presets (Dione) also operate a noise reduction and sharpness, is there any possibility to control this?
Thank you

No, the Dione Models have no tuning parameters.

If you add Avisynth prior to VEAI and let Avisynth do the deinterlacing, you can choose whatever Model you want - andproteus gives you the choice of how much you want to add/reduce sharpness or noise, halo, etc...

[toot367]

Originally Posted by DMD

Good morning.
As for the deinterlacing section, I noticed that all the presets (Dione) also operate a noise reduction and sharpness, is there any possibility to control this?
Thank you

I haven't seen v2.6.2.

FYI, QTGMC is no longer the best deinterlacer for standard definition. Dione Interlaced Robust v3 and Dione Interlaced Robust Dehalo v1 WITH the json mods to disable NR and sharpening provides much more detail for many SD videos. QTGMC still has value for 1080i though.

[poisondeathray]

Originally Posted by toot367

FYI, QTGMC is no longer the best deinterlacer for standard definition. Dione Interlaced Robust v3 and Dione Interlaced Robust Dehalo v1 WITH the json mods to disable NR and sharpening provides much more detail for many SD videos. QTGMC still has value for 1080i though.

How is the bob flicker with the json mods ?

QTGMC's main reason for existence (and it's precursor, tempgaussmc_beta) is to reduce bob flicker, not "keep details". In fact, as a result of QTGMC's temporal smoothing, noise and details are lost - as expected - ie. it's a tradeoff . You can adjust QTGMC settings to retain more details at the expense of more flicker

[toot367]

Originally Posted by poisondeathray

How is the bob flicker with the json mods ?

Are you referring to Dione screwing up the field order so that the frame jumps up or down for a frame?

If so, it's not perfect. I had to fix about 10 jumps on a 92 minute VHS movie. I duplicated the frame in Windows Explorer and used FillDrops(thresh=0.4) to fix the duplicate judder. When it finds a duplicate, FillDrops interpolates the extra frame and replaces it.

It's a nifty function for VHS, because you can fix many glitches by duplicating the previous or next frame. I duped 312 glitched frames in the movie and FillDrops fixed them all. Note I had VEAI save to a png sequence specifically to work directly with the frames using Windows Explorer, Paint, Photoshop, and Irfanview. I got rid of the microphone sneaking into one scene too. There's probably a more efficient method.

I tried some script to find bad frames, but it was useless (FindBadFrames it was called). It generated a file with "bad" frames but none of them were bad. So I had to find them with several viewings. I wish there were an automated way.

With DVD video, I haven't had any issues.

[poisondeathray]

Originally Posted by toot367

Are you referring to Dione screwing up the field order so that the frame jumps up or down for a frame?

No, just referring regular deinterlacing artifacts.

Such as aliasing, jaggies, when viewed in motion or over a few frame, it flickers.

The main reason QTGMC exists is to calm that flicker (at the expense of smoothing, detail loss)

[toot367]

Originally Posted by poisondeathray

Such as aliasing, jaggies, when viewed in motion or over a few frame, it flickers.

There are no jagges, artifacts, or flickering.

[Emulgator]

And here after some TB rendered: a small VEAI ingest/encode guide

VEAI 2.6.4 Do's and Don'ts

Besides of VEAI 2.6.4 solving most of its main tasks superior
one has to consider how to get the most out of it and not lose quality due to some unexpected behaviour.

First off: The recent Topaz VEAI 2.6.4 uses ffmpeg under the hood for decoding and encoding,

Input: I can happily feed .avs scripts
but
Output: Unfortunately all ffmpeg encoders have been left at defaults !

Audio: 2.0 Audio in .mp4 oder .mov will be transcoded to 96kbps aac.
The resulting 96kbps .aac I do consider unusable, hi-Freq morphing and mumbling at the same time.

Solution: Demux source Audio to 16bit .aiff using external ffmpeg, and remux into .mov after Topas VEAI using external ffmpeg !
Some NLEs, (Vegas 14 here) seems to be able to decode only 16bit .aiff

Video: The same situation.
All encoders are left at defaults (ffmpeg x264 core 157 r2970)
No CABAC, only CRF can be set.

CRF10 @ 1980x1080x50p: 157Mbps 1,524bpp NoCABAC 1Ref, GOP M=1, N=12 0,25s/f -> almost transparent
CRF12 @ 1980x1080x50p: 125Mbps 1,208bpp NoCABAC 1Ref, GOP M=1, N=12 0,22s/f -> blocks starting to appear
CRF14 @ 1980x1080x50p: 91,2Mbps 0,879bpp NoCABAC 1Ref, GOP M=1, N=12 0,22s/f -> fat blockiness
CRF16 @ 1980x1080x50p: 74,1Mbps 0,715bpp NoCABAC 1Ref, GOP M=1, N=12 0,22s/f -> unbearable blockiness
CRF20 @ 1280x720x29.97p -> 7Mbps -> Blockfest !
CRF20 @ 1280x720x59.94p -> 8,6Mbps -> Blockfest !

Go to regedit
Computer\HKEY_CURRENT_USER\SOFTWARE\Topaz Labs LLC\Video Enhance AI
This contains only the same settings you can reach via GUI.
So encoder parameters are really the ffmpeg hardcoded defaults...

Who wants to harvest quality has to use ProRes...

ProRes 422 10bit HQ @ 1980x1080x50p: 387Mbps 3,734bpp -> clean !

++++++++ 4K-Uprezzing ++++++++++
8-bit YUV444P8 Uncompressed mean 24bpp, 3840x2160x50p yield pixel rate 414Mpix/s
and need for 24bpp 10Gb/s. YUV422P8 would be 16bpp, -> 6,63Gb/s

ProRes 422 HQ 10bit @ 3840x2160x50p: 1545Mbps, 3,734bpp (23,5% of YUV422P8) 0,37s/f -> clean, faultfree und detailed picture !
BUT: Now a 43s short 147MB .MTS 140MB .264 explodes 55-fold to 8GB!
Well, a second thought comes up: Such huge files.
Can't we quick-transcode/remux ?

A RTX3080 can munch 4K-H264 or 4K-H265 at 33..34fps, then MP4 muxing
(MeGUI oder Simple Launcher, 2022-06-22 can mux H265 in .mp4)

NVENcC265 at 4K:
CQP10: 391/618Mbps, 0,942bpp, 25% .mov size: Water gradients usable
CQP12: 315/497Mbps, 0,759bpp, 20% .mov size: Water gradients stepping
CQP15: 203/388Mbps, 0,49bpp, 12,5% .mov size: Water gradients get horrible steps

NVEncC264 at 4K:
CQP10: 406/637Mbps 0,98bpp, (27% .mov size, 6,5% YUV422P8 uncompressed): Water gradients almost faultless, better as H.265.

Unfortunately Vegas 14 can't decode all these varieties, and still there is a visible quality loss.
So for me it is back to huge ProRes apch into Vegas 14.
1TB/h, well, so lets invest in even more storage X-|

P.S. Not NVEncC's fault, just tested Rigaya's 6.01 from within StaxRip 2.13 with improved tuning and it fares much better qualitywise,
I get almost transparent 4K50p quality out of 45Mbps, so I guess again: VEAI 2.6.4. left NVEncC at defaults...

[Emulgator]

BTW, Topaz are hiring now.
https://topazlabs.com/careers/
Dallas, TX. Anyone willing to help them out with a simple ffmpeg interface ?
I am not affiliated, BTW.

Or even better: proper deinterlacing as developed in this almighty forum ? ;_)

P.S. Seems that upcoming version 3 gets a better encoding preset system.

[miltonsoto658]

Originally Posted by Emulgator

Back to OP: Topaz VEAI 2.6.2 under Win10, i9-11900K 128GB, 3080RTX 16GB.
Downloaded all 186GB of models manually:
6432 files in C:\Users\<user>\AppData\Roaming\Topaz Labs LLC\Video Enhance AI\models
6 model families, each in 1..6 subversions with all their kernels, in 1..3 builds each.
If you don't preload everything VEAI will phone home for every model it sees fit and tries to get the next 30MB .tz file

Artemis: seems to be a one-stop shop for starters
Chronos: Good fps interpolation, see below
Dione: Deinterlacing, not mature yet I would say (knowing (Q)TGMC)
Gaia: one-in-all for CGI, I haven't tested yet.
Proteus: Fine tunable restoration for Pros, see below.
Theia: I haven't tested yet.

Used so far on Panasonic FT-3 encoder mud (1980x1080x25p on 25i)
Aiming to stay in 1920x1080, but with less camera artifacts.
Then maybe interpolate to 1920x1080x50p, maybe end in 1280x720x50p.

First decoding with DGDecode, then QTGMC for getting Chroma back onto Y, then into VEAI.
Proteus could repaint convincing seawater structure back, and resynthed details of hair and skin,
When overdone repeating patterns occurred, but not too bad !
Going back to source I started to hate the source more than the result.

Chronos 50p interpolation: knocks Mvtools2 out of the water, almost no morphing blotches.
In hard cases soft blends of details, but only visible while framestepping and pixelpeeping

Other source VHS on DV: Dione Deinterlacing: QTGMC still wins by far for now.
But with improving AI models almost everything seems within reach.

For me Proteus and Chronos are well worth it now and will stay in my daily toolbox.
Will not replace, but accompany Avisynth for the tasks where they shine.

Running stable here, Proteus V3 @ 4,5fps, Chronos V3 @ 6fps.
With such hard- and software a lot of projects will see a second AI edition now.

A few GUI unusual behaviours: Model Update management does list incomplete model sets only by filesize, not by files.
Hard to find out which is missing, you have to dump the complete set 3..6GB and try to download the complete set gain
Preview framecounter seems to be ready for input of framecount, but only renders the same amount 30 or 50.
One cannot save his preferences (why oh why do I have to input manually for every new clip)...

Reading up their forums it seemed to be much worse in the beginning
and from the responses it looks like only one developer responding, but still:
he answers, and there is definitely progress in GUI usability and I guess in number and usability of models too.

Output: Avoid .mov (ProRes 422HQ) for now, I got small blocks all over the sky.
If output avc in .mp4, give enough bits (I felt safe with CQ12)
and add grain (I used 1,8) otherwise you get blocks again.
Any fps model loses audio, even if duration stays.
Uncompressed .avi or VfW encoder interface: not yet implemented
.tiff frames ask for huge storage, but this is well known.

P.S. Just started redoing a paused film restoration from 720x576x25p DVD 5Mbps.
Proteus Dehalo: never expected that this can be achieved.
Clean transitions in 1280x720 and 1920x1080 while almost undisturbed by the source's blockfest.
I can happily trash my previous Dehaloing attempts now.

Just tried the unspeakable just to saturate the rendering system: Run 2 instances of VEAI in parallel. Works !

Can i ask how did you download manually all the models?

[Emulgator]

Well, manually was not the correct term.
It was via GUI, repeatedly invoking and deleting incomplete downloads over the course of a few days.