Deep Space Nine upscale project

[Stereodude]

Originally Posted by JoelHruska

Also, be advised, Topaz *does* introduce color-shifts on some content. I don't know which or why, but Weyoun's jacket is often color-shifted towards a darker brown.

Are you feeding it TIFF/PNG files or a video file and how are you making them?

Also, how are you getting the image files back into video?

We discussed this with zapp7 earlier in the thread. If don't right you shouldn't have a color shift (unless Topaz is broken or we don't understand what it's doing).

[JoelHruska]

Stereodude,

This was based on a video file, not PNG or TIFF output. I started to play with lossless at one point but had underestimated the storage requirements (Emissary and even PNG output don't mix unless you've planned on the storage requirement beforehand).

Both HQ-CG (Topaz 1.1.0) and Gaia-CG can cause a color shift, but they do it rather differently. HQ-CG had a *very* subtle lightening effect on DS9. I'm not even sure I dislike it, since one of the persistent complaints about the show is how dark it is. (If I was going to dislike HQ-CG it would be for detail removal, not the very small color shift).

Gaia-CG can deviate more widely than HQ-CG, but where HQ-CG had a very subtle effect on everything, Gaia-CG shows this less globally.

Also, while I haven't played with the lossless upscale option yet, I had planned to use a method like the one CaptRobau gave in his blog posts.

https://captrobau.blogspot.com/2019/...-topaz-ai.html

When I duplicated his workflow back when Gigapixel AI was the only tool available, I also duplicated his command and had no problems with it:

ffmpeg -r 29.970628 -f image2 -s 1440x1080 -i example%04d.jpg -i original-audio.wav -vcodec libx264 -crf 20 -pix_fmt yuv420p -acodec copy upscaled_video.mp4

Any specific changes you'd recommend?

[Stereodude]

Originally Posted by JoelHruska

Any specific changes you'd recommend?

1) Not using lossy compression until your final compression step.

2) I probably wouldn't convert to 8-bit H.264, but 10-bit HEVC instead.

3) I wouldn't load the images from a ffmpeg command line. HD video has rec.709 color space. DVD has rec.601. I'm guessing Topaz doesn't compensate for that. You need to make that conversion.

We discussed this with zapp7 earlier in the thread.

Convert to RGB in your AVIsynth script, and save the output to 16-bit tiff files. Run those through Topaz and then save the output as tiff files.

z_ConvertFormat(pixel_type="RGBP16", colorspace_op="170m:601:170m:f=>rgb:srgb:170m:f")

Load the 16-bit tiff files into AVIsynth (if using ImageSource, don't forget pixel_type="RGB48") and convert them to 4:3 yuv420 16-bit video with a rec.709 color space.

ConvertToPlanarRGB()
z_ConvertFormat(width=2880, height=2160, resample_filter="bicubic", pixel_type="YUV420P16", colorspace_op="rgb:srgb:170m:f=>709:709:709:f")

Then dither down to 10-bits with f3kdb

f3kdb(range=31, grainY=15, grainC=10, sample_mode=2, dither_algo=3, dynamic_grain=true, keep_tv_range=false, output_depth=10)

Feed that into x265 to compress it to 10-bit HEVC.

[zapp7]

I've given up on the 16-bit tiff files. I just don't have the requisite hard drive space as one episode in 16 bit tiff takes more than 2 TB.

I still do the conversion to RGBP16 before Topaz to preserve the full color range, but I save Topaz output as 8-bit .png instead and convert that to YUV420. I honestly couldn't see a difference between the 8-bit and 10-bit final versions.

[Stereodude]

Originally Posted by zapp7

I've given up on the 16-bit tiff files. I just don't have the requisite hard drive space as one episode in 16 bit tiff takes more than 2 TB.

I still do the conversion to RGBP16 before Topaz to preserve the full color range, but I save Topaz output as 8-bit .png instead and convert that to YUV420. I honestly couldn't see a difference between the 8-bit and 10-bit final versions.

FWIW, if I couldn't only have 1 that's 16-bits, I'd go 8 bit in and 16-bit out.

[zapp7]

Originally Posted by Stereodude

FWIW, if I couldn't only have 1 that's 16-bits, I'd go 8 bit in and 16-bit out.

The 16-bit in isn't an issue because the frame size is smaller. It's the 4K 16 bit files that are massive.

[Stereodude]

Originally Posted by zapp7

The 16-bit in isn't an issue because the frame size is smaller. It's the 4K 16 bit files that are massive.

Well, you don't have to do the full episode in one shot. You could do some number of frames and convert to a losslessly compressed intermediate file or go straight to HEVC in segments. But the Topaz folks should make it more versatile in terms of what it can open and export.

[zapp7]

Originally Posted by Stereodude

Well, you don't have to do the full episode in one shot. You could do some number of frames and convert to a losslessly compressed intermediate file or go straight to HEVC in segments. But the Topaz folks should make it more versatile in terms of what it can open and export.

Yeah... problem is I have two GPU's that I want to run in parallel and doing one episode at a time in chunks will take forever with 176 episodes.

[ReinerSchweinlin]

Originally Posted by zapp7

Yeah... problem is I have two GPU's that I want to run in parallel and doing one episode at a time in chunks will take forever with 176 episodes.

Yeah, takes a looot of time... Topaz VEAI has CLI support, so you could whip up a batch file to use the two GPUs with one instance..

In general: I guess what Joel is trying to accomplish is to come up with a workflow that can be replicated by anyone so the upscale and encoding could be distributed to many people. I could join in with a s mall batch as soon as things are pinned down.

[zapp7]

Originally Posted by ReinerSchweinlin

Yeah, takes a looot of time... Topaz VEAI has CLI support, so you could whip up a batch file to use the two GPUs with one instance..

In general: I guess what Joel is trying to accomplish is to come up with a workflow that can be replicated by anyone so the upscale and encoding could be distributed to many people. I could join in with a s mall batch as soon as things are pinned down.

I saw CLI support mentioned on the Topaz forums but I can't find any info on how to use it. Do you know where to find this? I would love to have a batch file to make life easier.

[Stereodude]

Originally Posted by zapp7

Yeah... problem is I have two GPU's that I want to run in parallel and doing one episode at a time in chunks will take forever with 176 episodes.

A single episode is what ~3.0TB in 3840x2160 uncompressed 16-bit TIFF files + ~150GB for the 720x480 uncompressed 16-bit TIFF files?

I guess you could buy a very large HDD for the project.

[scharfis_brain]

Is 3840x2160 really necessary?
IMHO 1440x1080 should be enough. Thus cutting space consumption by a factor of five...

[zapp7]

Originally Posted by scharfis_brain

Is 3840x2160 really necessary?
IMHO 1440x1080 should be enough. Thus cutting space consumption by a factor of five...

I've asked myself that same question. My gut says to future proof it as much as I can. In other words I may be disappointed that I didn't do 4K when it's so common 10 years from now.

[Stereodude]

Originally Posted by zapp7

I've asked myself that same question. My gut says to future proof it as much as I can. In other words I may be disappointed that I didn't do 4K when it's so common 10 years from now.

That's only if there's actually more detail in the 2160p upscale vs. a 1080p upscale. Has anyone compared that?

[johnmeyer]

Originally Posted by zapp7

I've asked myself that same question. My gut says to future proof it as much as I can. In other words I may be disappointed that I didn't do 4K when it's so common 10 years from now.

You do realize that you get zero improvement in quality by up-scaling and you are therefore completely wasting computer cycles, your time, and storage space forever.

Keep it at its original resolution, for heaven's sake. You are not "future proofing" anything.

[hello_hello]

Originally Posted by johnmeyer

You do realize that you get zero improvement in quality by up-scaling and you are therefore completely wasting computer cycles, your time, and storage space forever.

Keep it at its original resolution, for heaven's sake. You are not "future proofing" anything.

I assume you haven't looked at any of the upscaled samples?

I'm not saying they're to everyone's taste, but they generally look better than the SD original. At least for the CGI upscaling samples I've seen.

Whether you can do just as well by filtering without upscaling is another question, but if you do you're still at the mercy of the player/display upscaling in respect to how it'll look.

[Forteen88]

Originally Posted by hello_hello

...
Whether you can do just as well by filtering without upscaling is another question, but if you do you're still at the mercy of the player/display upscaling in respect to how it'll look.

By "10 years from now", the computers will be so fast that players
can have very high quality resize.

[ReinerSchweinlin]

Originally Posted by zapp7

I saw CLI support mentioned on the Topaz forums but I can't find any info on how to use it. Do you know where to find this? I would love to have a batch file to make life easier.

It was present in an early version and there was a help file about it, then they removed the feature end re-introduced it some versions ago - but the help file is missing now, I just looked up.. I will recover some of the old betas and search for it and post it here, give me some time, got work to do...

[ReinerSchweinlin]

Originally Posted by scharfis_brain

Is 3840x2160 really necessary?
IMHO 1440x1080 should be enough. Thus cutting space consumption by a factor of five...

Good point

While dealing with AI Upscalers, I learned one thing (I am refering to techniques that deal with single frames only, to keep it simple)

IF the AI starts to "see" somehting, i can re-interpret what it sees and replace the low resolution or noisy stuff with something new - preferably somehting looking similar or at least more pleasing to the eye..

This is very easy with cartoons which mostly have simple geometrical forms as lines, areas, fades, rectangles, etc.. - not to many textures and no "real life footage". So the Ai only has to "know" these types of image content and has an easy job replacing everything with something that looks good and very "realistik"...

As soon as real life footage is to be dealt with, the content becomes endlessly complex... If a Model is trained on landscape footage like mountains, sky, clouds, grass, trees, etc... It will pick up these kinds of objects and "repaints" them - but it will fail horribly on faces and other stuff.

It´s obvious that no model can cover every possible situation. Companies like TOPAZ use training sets which have a wild mix of "most common footage" in them to get a good coverage of all possible situations. But of course, there is a lot missing in the models. That´s the reason why a spezialized model can work so much better on specific footage than a generic one. The game-upscale community is training models on very specific games or techniques to take that into account.

So when throwing DS) into VEAI from Topaz - TOPAZ only can do as much as it knows - which isn´t tooo much in the case of DS9.... I exeprimented with some older DVDs, originally filmed a few decades ago with conventional film-cameras, containing mostly live footage (no special effects, no "mean" postprocessig, effects, filters, just "point the cam and film")... With this kind of material, it works really well (I simply love "police squad" in 1080p now..). But DS9 - not much luck.

(In fact, I would propose whipping up an open source "upscale project" with some nifty AI Upscalers and the possibility to train own models and then take something similar to DS9 available in true HD, downscale it to DVD or record to VHS - train on that and make a "DS9" model.... Startrek Movies, Next Generation, etc... Most of the stuff neede is out there, it´s the matter of having enough GPU power, time, knowledge to whip it all together....

ok, back to DS9 and "how much is enough"...

If the AI picks up: "ah, hair" - it replaces it with new hair, almost similar to the old one. If the model is done right, the alog is good, etc... one could now go up to whatever the model and framework limit - 8K - no problem (with most models)..

But - if the AI doesn`t know what to do - it simply upscales the original, bicubic, Lancoz, etc.... In this case, no more detail is produced.

VEAI now has frameworks which are able to recover details from multiple frames and add some deconvolution stuff into the workflow - so in some cases even if the model isn´t getting "hair", it is able to get some detail back - works good on still shots - in some cases I even got garbage on a pc-screen to reveal the actual numbers on the screen (SG1 NTSC DVD in this case). This mostly is refered to "super resolution". But it has ists limits. A quick turning head simply doesn´t have enough previous and following frames with the same info as the frame in question to recover anything out of the time domain... The Deconvolution sometimes is able to sharpen a little or make it at least visually more pleasing - but in my oppinion, in the case of VEAI - Topaz has done too much of it... This is very visible with compression artefacts (blocking) or noise or halos or ghosting etc... These get pronounced too much by the VEAI implementation..

For the moment, I personaly would stick to 1080p and keep the source for future usage.
In the case of VEAI: I would wait for them to sort things out before making a final upscale.

If I had the time and knowledge, I´d go the route if implemting an open source solution and train a DS9 Model

[ReinerSchweinlin]

Originally Posted by johnmeyer

You do realize that you get zero improvement in quality by up-scaling and you are therefore completely wasting computer cycles, your time, and storage space forever.

Keep it at its original resolution, for heaven's sake. You are not "future proofing" anything.

John, take a look at some examples on youtube. Some are garbage, but wiht others, you can really see "new" details.... Of course, this is "reinterpretation at higher resolution", not really "upscaling". And it can be wrong (I had some marvel busts in one video - the AI thought "oh, a head, lets pain hair into it"... But in many cases, the new details are quite pleasing to the eye...

So it really differs from tradtional upscaling (if it works, read my post above)