Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 20th February 2018, 05:10   #441  |  Link
Motenai Yoda
Registered User
 
Motenai Yoda's Avatar
 
Join Date: Jan 2010
Posts: 709
Quote:
Originally Posted by real.finder View Post
since the native 16 bit is slower than lsb, even with this https://forum.doom9.org/showthread.p...23#post1833823 it's little faster but still way slower than the lsb method

can mdegrain has "bits" parameter? so it can be used instead of lsb with avs+
IIRC lsb isn't 16bit processing but 8bit input -> 16bit output
also you can use ie convertbits(10,dither=1) before mdegrain stuff and convertbits(16) after.
also having a rounding option other than truncate/dithering options will be good.
__________________
powered by Google Translator

Last edited by Motenai Yoda; 20th February 2018 at 05:12.
Motenai Yoda is offline   Reply With Quote
Old 20th February 2018, 14:33   #442  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
Quote:
Originally Posted by Motenai Yoda View Post
IIRC lsb isn't 16bit processing but 8bit input -> 16bit output
also you can use ie convertbits(10,dither=1) before mdegrain stuff and convertbits(16) after.
also having a rounding option other than truncate/dithering options will be good.
yes in mvtools there are only lsb output, 8 input 16 lsb output

the bits parameter will do same thing with bits=16, and it's should be faster than lsb, and can use another bit setting like 12 for more speed, and can use 10 bit clip (for 10 bit source and output 16 bit or any other depending on the settings) or others, let see what pinterf say
__________________
See My Avisynth Stuff

Last edited by real.finder; 20th February 2018 at 14:36.
real.finder is offline   Reply With Quote
Old 20th February 2018, 18:22   #443  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
Quote:
Originally Posted by real.finder View Post
since the native 16 bit is slower than lsb, even with this https://forum.doom9.org/showthread.p...23#post1833823 it's little faster but still way slower than the lsb method

can mdegrain has "bits" parameter? so it can be used instead of lsb with avs+
Open doors.

Since I had found the reason of another mysterious crash, namely 64bit x264 exited when a specific Qtgmc + Tivtc script was used. The culprit was an ancient assembler mmx code used in MSuper. This code did not clear mmx state, as the original asm code did not issue emms, and the Microsoft x64 mode lacks _mm_empty() command (unlike the Intel compiler possibly used in the 2.6.0.5 era). The result was a crash, x264_64.exe probably was conflicted with the non-restored FPU/MMX states.

So after having inserted the missing 'emms' instruction it became rock solid. But I was so angry on MSuper's mmx heritage that I started to backport the sse2 intrinsic rewrite from VapourSynth mvtools. Then I added 10-16 bit sse2 versions. So MSuper is much faster now in 16bits (50+%).

As a side-effect MSuper now works in 32bit float (no SSE2 yet).

Having a 32 bit float MSuper, I was wondering what happens in a 32bit float MDegrain? (Primarily to see that the 16bit-mode color shift would be seen in 32bit mode, using the artifical example in the previous posts)

I made MDegrain able to use arbitrary bit-depth Super clip with vectors created in 8-16 bit mode.
For example we can have vectors created in 16 bit mode and use MDegrain2 on a 32bit float clip by simply passing a 32bit Super clip for MDegrain.

I have to clean up the code, there was quite a bit of work with implementing these features.

And I still need to run tests because 8 bit MDegrain with overlaps uses a strange rounding but since it exists in the C code and used in the asm assembler files also (copy paste?), I need to know whether it is intentional on a bug.
There is a 11 bit precision integer arithmetic that is converted back in two steps: lose first 6 bit (,sum up them) then lose 5 bits.

In 16 bits (and stacked) mode: final_pixel = (x + 1024) / 2048
In 8 bit: final_pixel = ((x + 256) >> 6) >> 5
I don't think the latter is correct. It results an 0.125 increase on average in the final pixel values.

Probably it should be: final_pixel = ((x + 32) >> 6) + 16) >> 5 ?
pinterf is offline   Reply With Quote
Old 27th February 2018, 16:20   #444  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
Download MvTools2 2.7.25 with depans

There were quite a big changes, so please test it.
Apart from the time-consuming mmx state bugfix there is a significant change in MDegrain worth to mention.

Source and Super clip format is now independent of the motion vector clip formats. (apart from the subsampling differences which should be the same - I'm lazy).

This means: one can Degrain even a 32bit float (new!) clip with e.g. 8 bit vectors. Pass a 8 bit clip to Super, then to Analyse and get the vectors.

Then use these vectors in MDegrain, while passing a 32 bit source and a 32 bit Super clip.

Code:
- 2.7.25 (20180227)
  Fix: x64: not-cleared mmx state in MSuper assembly code would cause crash later, e.g. in x264 encoding, depending on following filters.
  Fix: MSCDetection SC value parameter name to Ysc from Yth (must be an ancient typo), docs are OK, the fix is now mentioned in docs
  MSuper: import 8 bit sse2 interpolators from mvtools-vs. Extend them for 10-16bits (faster super clip). Some filters are still todo.
  MSuper: support 32bit float clips, which can be used later by MDegrains (but not for MAnalyse)
  MDegrains: allow degraining clip with different bit depth from vectors. Clip and Super must be the same bit depth
  MDegrains: consistently use limit and limitC, 255 do nothing, otherwise scale 0-254 value to the current bit-depth range
  Overlaps: more correct internal rounding for 8 bits:
            old: pixel = Sum( (tmp + 256) >> 6) >> 5
            new: pixel = (Sum( (tmp + 32) >> 6) + 16) >> 5
  Overlaps: round for 16bits
            old: pixel = Sum(tmp) >> 11
            new: pixel = (Sum(tmp) + 1024) >> 11
  Overlaps: 32bit float (but still use the original 11 bit window constants)
  Project: change from yasm to nasm.
pinterf is offline   Reply With Quote
Old 27th February 2018, 23:06   #445  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
thanks

and the "bits" parameter in MDegrains?
__________________
See My Avisynth Stuff
real.finder is offline   Reply With Quote
Old 27th February 2018, 23:55   #446  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
What is your speed difference of a e.g. MDegrain2 when passing 8/16/32bit clip and super, but all with 8 bit vectors?
pinterf is offline   Reply With Quote
Old 28th February 2018, 00:34   #447  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
Quote:
Originally Posted by pinterf View Post
What is your speed difference of a e.g. MDegrain2 when passing 8/16/32bit clip and super, but all with 8 bit vectors?
oh, this time native is fast, the one that make it slower last time was MScalevect()

but still lsb a bit faster than native

all 8 bit:- 13.93 fps

native 16 with 8 bit vectors:- 13.34 fps

8 bit with lsb:- 13.60 fps

native 32 with 8 bit vectors:- 11 fps

the test done with ColorBars(width=640, height=480, pixel_type="yv12").AddGrainC
__________________
See My Avisynth Stuff
real.finder is offline   Reply With Quote
Old 28th February 2018, 14:04   #448  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,542
Quote:
Originally Posted by real.finder View Post
but still lsb a bit faster than native
"Precision" is the same? Effectiveness in removing noise? Perceived quality output?

I am a old aged noob, I prefer to have smart guys opinions about matters I just play with.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 28th February 2018, 18:44   #449  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
Quote:
Originally Posted by tormento View Post
"Precision" is the same? Effectiveness in removing noise? Perceived quality output?

I am a old aged noob, I prefer to have smart guys opinions about matters I just play with.
they should be same in the case above

HBD or lsb should give no banding output mean it will more quality but has some unseen noise, so 8bit should removing more noise

but anyway let see what pinterf say
__________________
See My Avisynth Stuff
real.finder is offline   Reply With Quote
Old 28th February 2018, 21:41   #450  |  Link
burfadel
Registered User
 
Join Date: Aug 2006
Posts: 2,229
Wouldn't the effective block size in the analysis be different between lsb and native 16 bit? If so it will affect processing speed and possibly output quality.
burfadel is offline   Reply With Quote
Old 1st March 2018, 03:32   #451  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
Quote:
Originally Posted by burfadel View Post
Wouldn't the effective block size in the analysis be different between lsb and native 16 bit? If so it will affect processing speed and possibly output quality.
not always

https://forum.doom9.org/showpost.php...&postcount=424

and yes motion analysis in 16 is slower
__________________
See My Avisynth Stuff
real.finder is offline   Reply With Quote
Old 1st March 2018, 13:24   #452  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,542
Quote:
Originally Posted by real.finder View Post
they should be same in the case above
From DGDecNV manual:

fulldepth: true/false (default: false)
When fulldepth=true and the encoded video is HEVC 10-bit or 12-bit, then DGSource() delivers 16-bit data to Avisynth with the unused lower bits zeroed. The reported pixel format is CS_YUV420P16. If either of the two conditions are not met, then DGSource() delivers 8-bit YV12 or I420 data, as determined by the i420 parameter. When fulldepth=false and the video is HEVC 10-bit or 12-bit, then the video is dithered down to 8-bit for delivery. If you need a reduced color space (less than 16 bits) for your high-bit-depth processing, you can use ConvertBits() as needed after your DGSource() call.

How should I feed that 16 bit stream to SMDegrain? LSB is not an option in this case, I think. The funny part would be also to output a dithered 10 bit video to feed x265 in Main10 profile or 8 bit to feed x264. Without thinking, at this moment, to HDR.
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 1st March 2018, 14:02   #453  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
Quote:
Originally Posted by tormento View Post
From DGDecNV manual:

fulldepth: true/false (default: false)
When fulldepth=true and the encoded video is HEVC 10-bit or 12-bit, then DGSource() delivers 16-bit data to Avisynth with the unused lower bits zeroed. The reported pixel format is CS_YUV420P16. If either of the two conditions are not met, then DGSource() delivers 8-bit YV12 or I420 data, as determined by the i420 parameter. When fulldepth=false and the video is HEVC 10-bit or 12-bit, then the video is dithered down to 8-bit for delivery. If you need a reduced color space (less than 16 bits) for your high-bit-depth processing, you can use ConvertBits() as needed after your DGSource() call.

How should I feed that 16 bit stream to SMDegrain? LSB is not an option in this case, I think. The funny part would be also to output a dithered 10 bit video to feed x265 in Main10 profile or 8 bit to feed x264. Without thinking, at this moment, to HDR.
that for UHD BD or whatever original HBD source, in most cases your original source is 8 bit, and if it UHD then I don't think it's need any filtering or even re-encoding

anyway I don't know anything about DGDecNV, but ffms2 do it without any parameter and for avs+ it will output the HBD as it

and if you feed native HBD to SMDegrain it will work in most cases (if all filters that used are support HBD which I think in your cases all them are support HBD nowday)

and of course lsb is not option if your source is HBD, since mvtools has only lsb output
__________________
See My Avisynth Stuff

Last edited by real.finder; 1st March 2018 at 14:08.
real.finder is offline   Reply With Quote
Old 1st March 2018, 16:17   #454  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,542
Quote:
Originally Posted by real.finder View Post
and of course lsb is not option if your source is HBD, since mvtools has only lsb output
I have read html again and what I have understood is:

lsb= true makes 32 bit denoising

lsb_in = true is used to get the source as native 16 bit (right?)

lsb_out = true outputs 16 bit clip as output

mode = 0…9 convert from internal 32bit to 8 bit (would be nice to have 10 or 12 too).

So, if I have a 16 bit video input (i.e. 10 bit with all upper 0), I should use lsb=true+lsb_in=true+lsb_out=true (unless you develop a mode for 10 and 12 bit), right?
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 1st March 2018, 18:37   #455  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
How on Earth is 32bit denoising achieved?
pinterf is offline   Reply With Quote
Old 1st March 2018, 22:29   #456  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,587
Quote:
Originally Posted by tormento View Post
I have read html again and what I have understood is:

lsb= true makes 32 bit denoising

lsb_in = true is used to get the source as native 16 bit (right?)

lsb_out = true outputs 16 bit clip as output

mode = 0Â…9 convert from internal 32bit to 8 bit (would be nice to have 10 or 12 too).

So, if I have a 16 bit video input (i.e. 10 bit with all upper 0), I should use lsb=true+lsb_in=true+lsb_out=true (unless you develop a mode for 10 and 12 bit), right?
these (lsb things) in SMDegrain are hack for the lsb (which it's hack by itself!), they use some dither tools method for do this, cuz there are no real lsb_in for mvtools (MDegrain*), mvtools has only lsb parameter that use 8bit input and output 16 bit as lsb, and lsb is not native in first place! it's hack use 8bit clip with double height

and this dither tools method (that used to support lsb_in by dogway) will not be real 16 bit, even if it better than doing 8bit

in short, for SMDegrain lsb and lsb_out are real 16 bit and it's ok for use them, but they still hack, lsb_in in SMDegrain will not be real 16 bit

and for dither back to 8 bit modes as pinterf said, it's not 32 bit, it's just 16 bit stacked

and these mode come from dither tools again, not in or from mvtools!
__________________
See My Avisynth Stuff

Last edited by real.finder; 1st March 2018 at 22:38.
real.finder is offline   Reply With Quote
Old 1st March 2018, 22:48   #457  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
Next version will come with that "bits" or whatever parameter that returns native 16bit instead of the stacked one from 8 bit source. The mv bitdepth independent inputs could be applied to other filters as well, I think.
pinterf is offline   Reply With Quote
Old 2nd March 2018, 10:21   #458  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,542
Quote:
Originally Posted by pinterf View Post
The mv bitdepth independent inputs could be applied to other filters as well, I think.
So, if I understood well, at the moment a DGDecNV output from DGDecNV, that outputs a Ultra HD BD from 10 bits to 16bits with upper 0, is not natively useful when applying filters in a avisynth script with mvtools or dither, correct? We need dither (RGTools) update too, right?
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Old 2nd March 2018, 10:35   #459  |  Link
pinterf
Registered User
 
Join Date: Jan 2014
Posts: 2,309
Quote:
Originally Posted by tormento View Post
So, if I understood well, at the moment a DGDecNV output from DGDecNV, that outputs a Ultra HD BD from 10 bits to 16bits with upper 0, is not natively useful when applying filters in a avisynth script with mvtools or dither, correct? We need dither (RGTools) update too, right?
I meant that probably the same process - like with MDegrain was done - could be applied on MFlowXXX (using motion vectors from 8 bit clip (which is faster than the 10+bit path) and use them directly with 10+ bits clips)
pinterf is offline   Reply With Quote
Old 2nd March 2018, 12:16   #460  |  Link
tormento
Acid fr0g
 
tormento's Avatar
 
Join Date: May 2002
Location: Italy
Posts: 2,542
Quote:
Originally Posted by pinterf View Post
I meant that probably the same process - like with MDegrain was done - could be applied on MFlowXXX (using motion vectors from 8 bit clip (which is faster than the 10+bit path) and use them directly with 10+ bits clips)
Understood. I know x265 Main10 is still a bit new and "experimental", especially with HDR. Ultra HD BDs are out and I would like to play a bit with them
__________________
@turment on Telegram
tormento is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 18:28.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.