Processing .png files?

[asarian]

I am currently on a trial for a very cool A.I. upscaler, called Topaz Video Enhance AI (v. 2.3.0). Looks very promising, except... it can only lossless to either .tif of .png files. (sic!) How the heck can I feed a zillion .png files to x265?!

This is probably animation software thing, to use single .png files, but I have no idea how to feed it to x265. Can anyone give me a clue as to how to best output those to x265? Thanks.

[FranceBB]

Well, first you get the .tiff out from your source file with ffmpeg, then you use the AI Upscaler, then you use ImageSequence to index all the .tiff and get a video stream out of them and encode with x265 and lastly you use FFMpeg to encode the audio indexed in Avisynth and mux it together with the video you've just encoded:

@echo off
setlocal EnableDelayedExpansion

cd "%~p0"


mkdir audio
mkdir tiff

"ffmpeg.exe" -i %1 -map 0:a:0 -acodec copy -y audio/01.wav



"ffmpeg.exe" -y -i %1 -an -r 24 -pix_fmt rgb24 -vcodec tiff "tiff/%%06d.tiff"



"yourAIupscaler.exe" -i tiff -o tiff -r 24

"x265.exe" "AVS Script.avs" --preset medium "raw_video.hevc"


ffmpeg.exe -i "raw_video.h265" -i "AVS Script.avs" -map 0:0 -c:v copy -map 1:1 -c:a pcm_s24le -ar 48000 -f mxf "final_output.mxf"


rd /S /Q audio
rd /S /Q tiff

pause

and the AVS Script looks something like:

video=CoronaSequence("tiff*.tiff", sort=1)
audio=WAVSource("01.wav")
AudioDub(video, audio)

with "sort=1" you're essentially sorting all the tiff in a folder.
Take a look at this: http://avisynth.nl/index.php/ImageSequence


p.s please note having 2 sets of .tiff for a whole movie can be quite challenging in terms of storage. I know 'cause I had some troubles when I was encoding a DCP as it occupied over 1.2 TB worth of data.
p.p.s please note that both the .BAT and the .avs I posted before are more like a proof of concept to show you how it can be done rather than a ready-to-use copy-paste solution

[asarian]

^^ Thanks for the help! The A.I upscaler is the one who outputs to the tiff files, so all I really need to to get the tiff to combine into a single processable stream. :) Audio I extracted already earlier.

[asarian]

p.s please note having 2 sets of .tiff for a whole movie can be quite challenging in terms of storage. I know 'cause I had some troubles when I was encoding a DCP as it occupied over 1.2 TB worth of data.
p.p.s please note that both the .BAT and the .avs I posted before are more like a proof of concept to show you how it can be done rather than a ready-to-use copy-paste solution


Yeah, I get that. Fortunately, I only have a single stream of tiff files (generated by the upscaler) to contend with. :)

[benwaggoner]

Yeah, I get that. Fortunately, I only have a single stream of tiff files (generated by the upscaler) to contend with. :)
A lot of tools output TIFF as uncompressed by default. You can losslessly recompress them, though. Generally using Zip plus planar/"per channel" is the most efficient that I recall ffmpeg supporting. Generally cuts size by ballpark 50% for typical natural image content. Less if it's noisy, a lot more if it's simple.

[FranceBB]

^^ Thanks for the help! The A.I upscaler is the one who outputs to the tiff files, so all I really need to to get the tiff to combine into a single processable stream. :)

Well then just make sure that the tiff are all in the same folder with the same name and an increasing number and then use an Avisynth Script with CoronaSequence("*.tiff", sort=1) and play with the wildcards: it will read all the images in the folder and put them together in a nice uncompressed video stream, as Avisynth does, then all you need to do is to set the framerate in Avisynth and you're good to go. :)

[asarian]

Thx, guys. I have enough information now to proceed.

[SeeMoreDigital]

How the heck can I feed a zillion .png files to x265?! I use VirtualDub2...

[benwaggoner]

I use VirtualDub2...
Yeah, that's a simple, straightforward way. AVISynth should work as well.

For more complex projects, I'll do it in After Effects and then export a lossless mezzanine for x265. But that would be overkill for just 8-bit SDR.

[Blue_MiSfit]

I favor VapourSynth for this kind of work, mostly due to the fabulous fmtc plugin that gives you total control over all the color science, all in high bit depth. It does a solid studio approved job of color primaries conversions etc...

[benwaggoner]

I favor VapourSynth for this kind of work, mostly due to the fabulous fmtc plugin that gives you total control over all the color science, all in high bit depth. It does a solid studio approved job of color primaries conversions etc...
Yeah. I'm surprised VaporSynth doesn't get talked about more. It has a lot of advantages over both AVISynth and its derivatives, and ffmpeg.

[tonemapped]

I use the same application and use ffmpeg (only to x264) and a simple command. Unless I misread your post, it seems you simply want to have the sequential output from VEAI be encoded into x265?

[ReinerSchweinlin]

virtualdub2 was mentioned, thats easy and straight forward.
hybrid by Selur can also take image sequences as input and encode it into one [whatever codec you like] videostream.
I think staxrip can do it, too.
ffmpeg via commandline is an option, as mentioned..

Topaz Video Enhance can output 8 and 16 Bit TIFF uncompressed. Not using compression (LZE, ZIP, etc..) was a decision done in the beta tester forums because some workflows required uncompressed TIFF....

With 16 Bit TIFF, you can keep the most out of the colour information of VEAI. Since it re-intreprets the scene, it can - to some extend - reduce banding - so depending on the footage it can be worth going for TIFF16. If you are going for 8 Bit, I´d switch to PNG Output, which is the same output speed in VEAI - only rare cases occur where the CPU bottlenecks and the PNG Compression slows things down. In most cases, its the same speed and safes space.

If you have enough HD space - this is the way to go, since you can resume the AI-treatmend or split it or run multiple instances (I had no problem doing 6 instances at once - depending on your card and resolution - this can speed up the whole process...)

If HD space is a problem, you could still split the file while processing and output to Prores, then you have 10 Bit 422 as output, near losless, which is fine in most cases....

ah, one thing: A image sequence has no information on the framerate. So you have to set it manually. Beware of the deinterlace-models in VEAI, some of which output double the framerate.
In cases of variable frame rates in the source - you have to get inventive :) come back if thats the case, then more info and help is needed.

[asarian]

virtualdub2 was mentioned, thats easy and straight forward.
hybrid by Selur can also take image sequences as input and encode it into one [whatever codec you like] videostream.
I think staxrip can do it, too.
ffmpeg via commandline is an option, as mentioned..

Topaz Video Enhance can output 8 and 16 Bit TIFF uncompressed. Not using compression (LZE, ZIP, etc..) was a decision done in the beta tester forums because some workflows required uncompressed TIFF....

With 16 Bit TIFF, you can keep the most out of the colour information of VEAI. Since it re-intreprets the scene, it can - to some extend - reduce banding - so depending on the footage it can be worth going for TIFF16. If you are going for 8 Bit, I´d switch to PNG Output, which is the same output speed in VEAI - only rare cases occur where the CPU bottlenecks and the PNG Compression slows things down. In most cases, its the same speed and safes space.

If you have enough HD space - this is the way to go, since you can resume the AI-treatmend or split it or run multiple instances (I had no problem doing 6 instances at once - depending on your card and resolution - this can speed up the whole process...)

If HD space is a problem, you could still split the file while processing and output to Prores, then you have 10 Bit 422 as output, near losless, which is fine in most cases....

ah, one thing: A image sequence has no information on the framerate. So you have to set it manually. Beware of the deinterlace-models in VEAI, some of which output double the framerate.
In cases of variable frame rates in the source - you have to get inventive :) come back if thats the case, then more info and help is needed.


:thanks:

TIFF is a good format, but I don't have a spare 8T drive lying around at the moment. I eventually went with H264 output, with CRF=0. This forces 4:4:4 Predictive H264 encoding (lossless, akin to QP=0 in x264). That gave me a ca. 450G intermediate output file, which would fit on my SSDs. Not ideal yet, but what I had.

What do you mean splitting jobs, though? Like a 4K movie, presplit into 4 different quadrants? Not sure I understand that, as it would require 4 extra lossless input streams. Wish VEAI could accept .vpy (VapourSynth) input 'files' for that, then I could split in there.

[ReinerSchweinlin]

In case VEAI crashes, the single filte method easily gives you the option of resuming.

"Splitting jobs" is related to that.. You can either start up multiple instances of VEAI or start it on several PCs and then process only a part of the file - putting all output files in the same directory. I did this when I was testing a System with an Intel iGPU, an RTX GEN20 and a VEGA 64 in one system.... Opening 5 instances: 2 on the RTX, two on the VEGA and one on the iGPU (just for fun, wouldnt recommend that as a permanent solution, poor igpu.... probably will burn out..). This wa much faster than running one instance on all cards and it even works with only one card in the system. There seems to be some internal "unused ressources" in many usecases with VEAI - so opening a second or third instance and let it run in parallel actually speeds things up... Try it ( ´6 instances on an Radeon IIV was no problem...).

Of course you could do that with h264 output as well, its just a little more fiddling to join the files together..

[asarian]

In case VEAI crashes, the single filte method easily gives you the option of resuming.

"Splitting jobs" is related to that.. You can either start up multiple instances of VEAI or start it on several PCs and then process only a part of the file - putting all output files in the same directory. I did this when I was testing a System with an Intel iGPU, an RTX GEN20 and a VEGA 64 in one system.... Opening 5 instances: 2 on the RTX, two on the VEGA and one on the iGPU (just for fun, wouldnt recommend that as a permanent solution, poor igpu.... probably will burn out..). This wa much faster than running one instance on all cards and it even works with only one card in the system. There seems to be some internal "unused ressources" in many usecases with VEAI - so opening a second or third instance and let it run in parallel actually speeds things up... Try it ( ´6 instances on an Radeon IIV was no problem...).

Of course you could do that with h264 output as well, its just a little more fiddling to join the files together..


Multiple instances sounds great. But I must admit I don't really get it. I run with max video memory set, which, if I am to believe the documentation, occupies like 8G already (of my 11G max). How on earth will it start 5 instances of that?! Or even 2?

I don't have enough hard disk space for outputing 1 file per frame, making preprocessing of split main movie tougher. Would still prefer if VEAI could accept .vpy (VapourSynth) scripts as input. Would make my life so much easier. :) (Also handy for pre-cropping borders, QTGMC deinterlacing, etc).

[tonemapped]

Multiple instances sounds great. But I must admit I don't really get it. I run with max video memory set, which, if I am to believe the documentation, occupies like 8G already (of my 11G max). How on earth will it start 5 instances of that?! Or even 2?

I don't have enough hard disk space for outputing 1 file per frame, making preprocessing of split main movie tougher. Would still prefer if VEAI could accept .vpy (VapourSynth) scripts as input. Would make my life so much easier. :) (Also handy for pre-cropping borders, QTGMC deinterlacing, etc).

It's always best to feed pre-processed video into VEAI. I've manually gone through over 9,000 frames before (a music video) in order to fix any pre-processing issues not fixed by scripts.

[ReinerSchweinlin]

Multiple instances sounds great. But I must admit I don't really get it. I run with max video memory set, which, if I am to believe the documentation, occupies like 8G already (of my 11G max). How on earth will it start 5 instances of that?! Or even 2?

I don't have enough hard disk space for outputing 1 file per frame, making preprocessing of split main movie tougher. Would still prefer if VEAI could accept .vpy (VapourSynth) scripts as input. Would make my life so much easier. :) (Also handy for pre-cropping borders, QTGMC deinterlacing, etc).
Of course you have to manage your ressources. If you are using one of the very memory intense models and input very high resolutions, you might run into VRAM shortage. Some hints to approach this:

- the "reduce load" switch in the settings reduces the number of threads run in parallel on the GPU. It also lowers the VRAM usage to around one half.
- disabling all other programs and switcxhing everything else that needs a GPU to a small seperate GPU or an iGPU frees up VRAM.
- The tile size the model is using is determined by the "VRAM USAGE" slider in the settings. At the moment, around 512x512 is the maximum provided by the downloadserver from Topaz.

In a usecase where your GPU is already maxed out to 90% with one instance - there is not much to gain. Best to check if thats the case is to use GPU-Z or better Process explorer to see what exactly is utiliezed (encoder / decorder / shader / cuda cores /etc.."!....

So some testing and fiddling is the way to go:

try lowering the tile size or rdeuce the number of threads to free up some ressources - then run a second instance. See if the overall speed you get is better than before.

This "multiple instances" scenario mostly makes sense in cases of lower input resolutions - if you are feeding 1080p or above, vram mostly will be the limiting faktor.

11GB of VRAM - are you using a 1080TI ? Well, than Topaz chooses FP32 models, sinse the 1080TI has no dedicated FP16 units - therefore the VRAM amount needed is double of what would be neccessary if you would use somehting like a GEN20 or VEGA56 and above or recent intel iGPU (which all have higher FP16 capabilities than older models)....

In case of the 1080Ti - I am not sure if it is able to handle FP16 at least with the same speed as FP32 - then one trick would be: download the FP16 models manually, switch off the "download models" and go for it - freeing up VRAM...

With VEAI (and similar sopftware), the best is to get a more suitable GPU.... Prices are slowly dropping, with a little luck you can get a VEGA 56 or 64 for under 400 bucks at the moment - these cards really shine with VEAI and easily outperfrom your 1080TI (the common "knowledge", based on gaming benchmarks ist, that the VEGA64 is around the speed of a 1080 - not in VEAI, these outperfrom smaller GEN30 cards ...)

direct vapoursynth input is not yet implemented, but almost everything usefull can be done in avisynth, too. Which is - at least on windows - the way to go to feed VEAI.

You might have some luck with something like avisynth virtual file system, maybe someone has adpoted that to vapoursynth - But I´d simply go for avisynth - staxrip, ripbot, etc... all produce good starting points for croping, QTGMC, denoise, etc... which can be directly fed into VEAI...

[poisondeathray]

You might have some luck with something like avisynth virtual file system, maybe someone has adpoted that to vapoursynth

Works ok with vapoursynth (avfs is bundled with vapoursynth)


avfs script.vpy

[ReinerSchweinlin]

Works ok with vapoursynth (avfs is bundled with vapoursynth)


avfs script.vpy

Ah, nice :) Thank you!

[asarian]

Of course you have to manage your ressources. If you are using one of the very memory intense models and input very high resolutions, you might run into VRAM shortage. Some hints to approach this:

- the "reduce load" switch in the settings reduces the number of threads run in parallel on the GPU. It also lowers the VRAM usage to around one half.
- disabling all other programs and switcxhing everything else that needs a GPU to a small seperate GPU or an iGPU frees up VRAM.
- The tile size the model is using is determined by the "VRAM USAGE" slider in the settings. At the moment, around 512x512 is the maximum provided by the downloadserver from Topaz.

In a usecase where your GPU is already maxed out to 90% with one instance - there is not much to gain. Best to check if thats the case is to use GPU-Z or better Process explorer to see what exactly is utiliezed (encoder / decorder / shader / cuda cores /etc.."!....

So some testing and fiddling is the way to go:

try lowering the tile size or rdeuce the number of threads to free up some ressources - then run a second instance. See if the overall speed you get is better than before.

This "multiple instances" scenario mostly makes sense in cases of lower input resolutions - if you are feeding 1080p or above, vram mostly will be the limiting faktor.

11GB of VRAM - are you using a 1080TI ? Well, than Topaz chooses FP32 models, sinse the 1080TI has no dedicated FP16 units - therefore the VRAM amount needed is double of what would be neccessary if you would use somehting like a GEN20 or VEGA56 and above or recent intel iGPU (which all have higher FP16 capabilities than older models)....

In case of the 1080Ti - I am not sure if it is able to handle FP16 at least with the same speed as FP32 - then one trick would be: download the FP16 models manually, switch off the "download models" and go for it - freeing up VRAM...

With VEAI (and similar sopftware), the best is to get a more suitable GPU.... Prices are slowly dropping, with a little luck you can get a VEGA 56 or 64 for under 400 bucks at the moment - these cards really shine with VEAI and easily outperfrom your 1080TI (the common "knowledge", based on gaming benchmarks ist, that the VEGA64 is around the speed of a 1080 - not in VEAI, these outperfrom smaller GEN30 cards ...)

direct vapoursynth input is not yet implemented, but almost everything usefull can be done in avisynth, too. Which is - at least on windows - the way to go to feed VEAI.

You might have some luck with something like avisynth virtual file system, maybe someone has adpoted that to vapoursynth - But I´d simply go for avisynth - staxrip, ripbot, etc... all produce good starting points for croping, QTGMC, denoise, etc... which can be directly fed into VEAI...


My apologies for the tardiness of my reply. And kudos to your fine intellect, in determining I do, indeed, own a 1080 Ti (it's the quaint 11G config that gave it away, right?). And very good info on the graphics cards. (It's hard to believe less than 3 years ago my 1080 Ti was nearly top-of-the-line).

As for VRAM, processing a 1080p source now, and VEAI only uses ca. 3.7G VRAM. Still takes 1.5 days (with an i7 10700K). But yeah, GPU is clearly bottleneck here. CPU load is only at ca. 17%, but I guess that's due to me doing lossless output (nothing to compress, really).

Oh, and I will install avisynth again. Thx. Indeed, only preprocessing I'd need is perhaps QTGMC deinterlacing, and perhaps some cropping.

Guess the verdict is really in: I need a better GPU. :)

[Losko]

I'm afraid I'm a bit late for this as the poster user already has a working solution.
Nevertheless, a long time ago I found a tiny utility among daala sources ( https://github.com/xiph/daala ) which is called png2y4m and it does... what the name says.
It suited my needs a couple of times, very handy.

[ReinerSchweinlin]

Guess the verdict is really in: I need a better GPU. :)
:)
Or: Get a second one and run them in parallel :) But since the 1080TI is not strong in FP16, a VEGA64 might be the better deal for the job, it is much faster...