Hybrid: Input -> x264/x265/Xvid/VP8/VP9

[ReinerSchweinlin]

I just stumpled across this:
https://forum.doom9.org/showthread.php?t=176961
Could this be implemented in Hybrid? I know, I´d be needing a bigger GPU, but I have some personal rare footage I´d like to throw at it..

[Selur]

Hybrid rev 2019.12.01.1:
*fixed*

• video: x265 make sure 'fixed quantizer' zones use integer

*changed*

• muxing: Hybrid doesn't send 'und' to MP4Box but empty language strings

*added*

• muxing: option to overwrite not only container but additionally stream aspect ratio

-> downloads: http://www.selur.de/downloads

----
Last I checked the main problem with lots of these neural network solutions is finding the tools and the dependencies that need to be available to incorporate them into Hybrid.
Often the dependencies are huge (in file size) and some times they are by their license not allowed to be distributed in parts or in a whole through others.
-> so if you can list a tool with its dependencies and those dependencies can be shipped with Hybrid and are not humongous I could add that tool/filter to Hybrid, atm. there are no plans for this.

"With waifu2x enabled, range is altered automaticaly. "
Hybrid only sets the range detected by the analysis.


Cu Selur

[Havokdan]

I some time without converting files, but experiencing this version, I realized that does not carry the QSVEnc profiles, I create one and try to match a aac audio, but always carries the codec x264.

[Selur]

I don't really understand what you are speaking of.
In case QSVEnc isn't snown in case:
a. no Intel CPU is detected
b. 'QSVEncC --check-features' reports 'QSV unavailable.' or nor H.264 and H.265 features.
Usually this is due to conflict of QSVEncC with the CPU drivers.
-> What does 'QSVEncC --check-features' report on your system ?

Cu Selur

[Havokdan]

Originally Posted by Selur

I don't really understand what you are speaking of.
In case QSVEnc isn't snown in case:
a. no Intel CPU is detected
b. 'QSVEncC --check-features' reports 'QSV unavailable.' or nor H.264 and H.265 features.
Usually this is due to conflict of QSVEncC with the CPU drivers.
-> What does 'QSVEncC --check-features' report on your system ?

Cu Selur

I'm sorry, I'm using an online translator, I wanted to say this, I create a profile for QSVEnc, so I try to create an "AV-Combo" profile with him, however, when trying to load the profile, video codec that is loaded it is x264.

[ReinerSchweinlin]

Thanx for the updates!!

Originally Posted by Selur

----
Last I checked the main problem with lots of these neural network solutions is finding the tools and the dependencies that need to be available to incorporate them into Hybrid.
Often the dependencies are huge (in file size) and some times they are by their license not allowed to be distributed in parts or in a whole through others.
-> so if you can list a tool with its dependencies and those dependencies can be shipped with Hybrid and are not humongous I could add that tool/filter to Hybrid, atm. there are no plans for this.

Unfortunately, my programming skills are too basic to really get a grasp of everything needed to implement this filter, but as far as I understand it, this
https://github.com/imPRAGMA/VSGAN
simply seems to be a vapourynth port of ESRGAN.
Reading through the install-files, it looks straight forward to me - all tthat´s needed to add is the model which calculations are done with.
Some are found here:
https://upscale.wiki/wiki/Model_Database
which seem to work, according to some forum members...
I wouldn´t mind downloading these manualy

Originally Posted by Selur

"With waifu2x enabled, range is altered automaticaly. "
Hybrid only sets the range detected by the analysis.

thanx, I will investigate my source file further to see if there is weird colour setting in it that causes the range change...
Does Hybrid do a different analsys if wafu2x is enabled? If I leave eveyrthing unchanged and simply turn waifu2x on or off, I get different colour ranges, so my assumption is, that it has something to do with waifu2x, not hybrids interpretation of the source file analysis.

[Selur]

I wanted to say this, I create a profile for QSVEnc, so I try to create an "AV-Combo" profile with him, however, when trying to load the profile, video codec that is loaded it is x264.

Ah okay, will try to reproduce this tomorrow.
(Haven't used QSVEnc for ages since my encoding machine uses an AMD CPU and my other machines only have older Intel CPUs where GPU encoding wasn't that appealing,..)
-> will fix, happens not only with QSVEnc, so it's easier to reproduce and fix.

Does Hybrid do a different analsys if wafu2x is enabled? If I leave eveyrthing unchanged and simply turn waifu2x on or off, I get different colour ranges, so my assumption is, that it has something to do with waifu2x, not hybrids interpretation of the source file analysis.

No the analysis is right but unless a rgb <> yuv conversion is needed the color range is only used for flagging in the output.
In case a rgb <> yuv conversion is used the assumed color matrix will be used during that conversion.
Here's an example:

# Imports
import vapoursynth as vs
core = vs.get_core()
import ctypes
# Loading Support Files
Dllref = ctypes.windll.LoadLibrary("I:/Hybrid/64bit/vsfilters/ResizeFilter/Waifu2x/w2xc.dll")
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/ResizeFilter/Waifu2x/Waifu2x-w2xc.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2k.dll")
# Loading source using FFMS2K
clip = core.ffms2.Source(source="F:/TESTCL~1/files/test.avi",cachefile="E:/Temp/avi_078c37f69bb356e7b5fa040c71584c40_853323747.ffindex",format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Point(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# resizing clip to 1280x704
# adjusting color space from YUV420P8 to RGB48 for Waifu2x-w2xc
clip = core.resize.Bicubic(clip=clip, format=vs.RGB48, matrix_in_s="470bg", range_s="limited")
# current color sampling: RGB48
# adjusting bit depth to 32bit
clip = core.fmtc.bitdepth(clip=clip,bits=32)
# resizing using Waifu2x
clip = core.w2xc.Waifu2x(clip=clip, scale=2)
# current color sampling: RGB48
# adjusting output color from: RGB48 to YUV420P8 for x264Model (i420)
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# Output
clip.set_output()

Since you didn't share any details I have no way to reproduce your problem and can't really help.

Cu Selur

[ReinerSchweinlin]

Hi Selur,

thank you for your answers.
I tried some stuff and found it: After setting range conversion to "limited->full" in Filtering-vs-colour, the colors stay the same.

The script looks like this now:

# Imports
import vapoursynth as vs
core = vs.get_core()
import ctypes
# Loading Support Files
Dllref = ctypes.windll.LoadLibrary("C:/Program Files/Hybrid/64bit/vsfilters/ResizeFilter/Waifu2x/w2xc.dll")
# Loading Plugins
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/ResizeFilter/Waifu2x/Waifu2x-w2xc.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/Support/fmtconv.dll")
core.std.LoadPlugin(path="C:/Program Files/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# Loading G:\video\test.mkv using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/video/test.mkv", format="YUV420P8", cache=0, prefer_hw=3)
# making sure input color matrix is set as 470bg
clip = core.resize.Point(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 24
clip = core.std.AssumeFPS(clip, fpsnum=24, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
clip = core.resize.Point(clip, range_in_s="limited", range_s="full")
# Setting color range to PC (full) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=0)
# resizing clip to 1280x960
# adjusting color space from YUV420P8 to RGB48 for Waifu2x-w2xc
clip = core.resize.Bicubic(clip=clip, format=vs.RGB48, matrix_in_s="470bg", range_s="full")
# current color sampling: RGB48
# adjusting bit depth to 32bit
clip = core.fmtc.bitdepth(clip=clip,bits=32)
# resizing using Waifu2x
clip = core.w2xc.Waifu2x(clip=clip, scale=2, noise=0, block=1024)
# current color sampling: RGB48
# adjusting resizing
clip = core.fmtc.resample(clip=clip, w=1280, h=960, kernel="lanczos", interlaced=False, interlacedd=False)
# adjusting output color from: RGB48 to YUV420P8 for FFvHuffModel (i420)
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="full")
# Output
clip.set_output()

While testing some of the other Waifu2x Tools, I noticed quite a bit of speed increase when using RGB24Bit intermediate colour space without noticable quality change with old anime encoded in XVID (which are Yu12). Do you think this could speed up conversion in Hybrid, too? After turning off the noise reduction in Waifu2x and encoding to Huffyuf, upscaling a 640x480 file 2fold in size, I get 1fps on an gtx960... (i5 Kabylake).

when using wafu2x in Foto mode while keeping the resize factor to 1, it´s sometimes possible to clear up files with real life footage from noise, blocking, artefacts... In Hybrid, only upscaling is possible, as soon as the resizer is set to the input size, it leaves out waifu2x in the qeue. Could you implement "keep the resolution but still run it through waifu2x" please?

Ah, on the download page I still get the previous version, although its named different.

Another thing I noticed: While trying to distribute work onto cpu/gpus, I wanted to switch to qsv decoding. But no matter what I select in the config-input-decoding section, it´s always the nvidia card decoding.

Thanx for your help!

[Selur]

Ah, on the download page I still get the previous version, although its named different.

nope. just forgot to update the revision info properly.

While trying to distribute work onto cpu/gpus, I wanted to switch to qsv decoding. But no matter what I select in the config-input-decoding section, it´s always the nvidia card decoding.

I don't think Hybrid atm. does support using qsv for decoding with Vapoursynth.
It might be possible if I change the HW decoding support Hybrid uses with LibavSMASHSource, atm. Hybrid uses either "prefer_hw=0", or "prefer_hw=3", but I could extend this to :

                        - 0 : Use default software decoder.
                        - 1 : Use NVIDIA CUVID acceleration for supported codec, otherwise use default software decoder.
                        - 2 : Use Intel Quick Sync Video acceleration for supported codec, otherwise use default software decoder.
                        - 3 : Try hardware decoder in the order of CUVID->QSV. If none is available then use default software decoder.

-> will look into it.

[Selur]

Looked into https://github.com/imPRAGMA/VSGAN from what I gather Hybrid would need to install:
* PyTorch
* Numpy
* mvsfunc
* CUDA (NVIDIA GPU Support, optional but without this everything will be extremely slow)
* vsgan
-> unless there is a portable package I could ship with Hybrid or provide as a separate download I won't write support for this.
Will send you a PM in regard to LibavSMASHSource 'prefer_hw' and allowing waifu2x while no real resizing is done,...

While testing some of the other Waifu2x Tools, I noticed quite a bit of speed increase when using RGB24Bit intermediate colour space without noticable quality change with old anime encoded in XVID (which are Yu12). Do you think this could speed up conversion in Hybrid, too?

Yes, using RGB24 instead of RGB48 which Vapoursynth uses by default might speed-up stuff a bit. How much are we talking about here. (Not really sure it's worth the effort to support this.)

Cu Selur

[ReinerSchweinlin]

Originally Posted by Selur

Looked into https://github.com/imPRAGMA/VSGAN ... unless there is a portable package I could ship with Hybrid or provide as a separate download I won't write support for this.

ok, thanx for looking into it. Maybe the vapoursynthport will advance over time and I hope there will be a portable, simple solution available in the future. Postponed for the moment..

Will send you a PM in regard to LibavSMASHSource 'prefer_hw' and allowing waifu2x while no real resizing is done,...

Cool, thank you! Will try

Yes, using RGB24 instead of RGB48 which Vapoursynth uses by default might speed-up stuff a bit. How much are we talking about here. (Not really sure it's worth the effort to support this.)

My comparison is based video2x, which converts a videofile into intermediate PNG Files, throws all of them into waifu2x and then joines the image sequence back together into a videofile. Of course, the overhead here is much bigger than with hybrid which pipes all the steps into skript, so the speed difference might vary.

in Video2x, The speed difference on my system is between 50% and 80% speedup when using RGB24 instead of RGB48 (temp on SATA SSD)..

I could try to test this myself - my amateur skills give me the idea to tap into the skript of hybrid, play around with the settings and run it manualy to measure the speed difference. I´d have to do some digging, did some avisynth back in the days... Maybe I get it done - do you think this could be an approach to test it (so I can contribute a little work to the topic) ?

[Selur]

Maybe I get it done - do you think this could be an approach to test it (so I can contribute a little work to the topic) ?

You can see the script (Filtering->Vapoursynth->Vapoursynth Script View), copy the script into a new file and call vspip:

VSPipe.exe -p <PATH TO SCRIPT> NUL

and ist should output something like:

Frame: 115/5000 (2.62 fps)

(on a small test using RGB24 instead of RGB48 didn't give any speed gain; but that was only one test)

Cu Selur

[ReinerSchweinlin]

Thanx!

Report:
using RGB24 instead of RGB48 has almost no effect with my examples - maybe upscaling HD->4K would benefit more, but my GFX Card is too small and slow - with SD Sources I see no speed difference

Using Waifu2x with scale 1 ("enhance, no upscale") uses

# resizing using Waifu2x
clip = core.w2xc.Waifu2x(clip=clip, scale=2, block=1024)

In the VS Script. So it´s still upscaled and then resized to the original resolution. So it takes the same time as upscaling 2x.
Would it be possible to set

# resizing using Waifu2x
clip = core.w2xc.Waifu2x(clip=clip, scale=1, block=1024)
?

Thanx to your hint above on how to call vspipe manualy, I did a speedtest:

# resizing using Waifu2x
0,95fps <- clip = core.w2xc.Waifu2x(clip=clip, scale=2)
4,33fps <- clip = core.w2xc.Waifu2x(clip=clip, scale=1, block=1024)


Regarding HW-Decode with QSV: So far no luck, have to investigate more, seems decoding is still done in SW or Nvidia Card.

[Selur]

Looking into it.
---
HW decoding requires LWLibAVSource (or DGDecNV) to be used and for QSV prefer_hw should be set to 2, script wise Hybrid seems to properly allow to set the value, so if QSV isn't used working it might be a driver problem or a bug in LWLibAVSource.

[ReinerSchweinlin]

Hey. Waifu works fine now, many thanks for looking into this! The colour issue is gone and scale=1 works, too

Assigning all the GPU Power still is a little bumpy - when doing NVENC (RTX 2060) Encodes, I assign NLMEANS CL to the same card (CUDA) and decoding to QSV - which won`t work. In Handbrake, it works right away and with others, too, so drivers etc.. seem to be ok..
I have to do some more testing on this to give a clearer report - for the meantime, this is no problem, since the speed difference is small.

Christmas probably kicking in @all, so time will be spent elsewhere

[Selur]

Hybrid rev 2020.01.11.1:
*fixed*

• analyse: HDR parameter import rounding
• Synth: ignore timecodes for output if FrameInterpolation or Restore is
• cosmectice: AudioTab, AudioInfo on track change
• Synth: ImageSource handling
• video: qsv encoding mode
• tools: manual set tools could cause crashes
• Vapoursynt: Waifu2x limited range compression by using RGBS
• video: vsPipe Problem gefixed
• video: NVEnc only singal VUI in case VUI Signaling is enabled
• audio: handling of multiple trueHD streams with ffmpeg
• jobs: mkvmerge muxing timecodes parallel processing
• video: x265 support histogram scence change detection

*changed*

• audio: dts-es muxing
• Synth: allow to set LibWAV HW decoding preference
• Vapoursynth: allow Waifu2X even when resize resolution is the same as the crop resolution.

*added*

• Vapoursynth: added support for EdgeFixer
• Synth: 'SelectRangeEvery' for Avisynth and Vapoursynth (video only!)

-> downloads: http://www.selur.de/downloads

Cu Selur

[Havokdan]

I already made the request to the StaxRip, do the same for the Hybrid now automatically import the Hybrid is possible tags/fields nfo files (https://kodi.wiki/view/NFO_files) when the support such target format information?

[Selur]

Not totally sure I get that.
Do you want Hybrid to:
a. create kodi-NFO-files from the tags of a file, or
b. import tag information from an NFO file, or
c. do something different I totally missed?

Cu Selur

[Havokdan]

B

I want to import the nfo file information to the video file in a format that supports such information, the information in the nfo file that usually use is the standard kodi.

[Selur]

Hmm, looking at what tags Hybrid supports (for the main containers):

• Container tags:
  • Name
  • Genre
  • Comments
  • Copyright
  • Cover
• Stream tags:
  • Title
  • Language

and the tags Kodi allows (https://kodi.wiki/view/NFO_files/Movies#nfo_Tags)
I don't see anything for Copyright in the movie info and the Cover info would probably best match to the 'thumb' info, but to be useful Hybrid would need to download stuff (not likely to happen).

I see the following mapping:
Container->Name <=> movie->title
Container->Genre <=> movie->genre
Container->Comments <=> movie->plot
and
for audio streams:
Stream->Title <=> movie->fileinfo->streamdetails->audio->language
There is no language for video streams.

Since normally at least the audio language tags are usually properly flagged in the files itself, the only real gain from this would be to import the container tags.
(Importing the audio language could also be done, but probably not worth the trouble.)

-> don't really think that it importing the container tags is worth the effort, but if you find a few other users which think this is a nice feature I can probably write some code for it.

Cu Selur

Ps.: for mkv output adding a 'import for nfo' under 'Config->Output->MKV Tags' might be more useful, since there would be a bigger overlap of the kodi-NFO data and the mkv tags.