Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 15th May 2022, 08:01   #101  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
Quote:
Originally Posted by DTL View Post
AVX mean it do not have fast enough integer operations with increased size register file. Also operations are limited to SSE2 128bit integer per op, AVX2 allow 256bit integer ops that is virtually twice faster. It is better to upgrade to AVX2 CPU at least in 202x years. Intel promises AVX1024 in the mid of 202x already.
This.

You can get a very decent second hand AVX2 system on the cheap thanks to AMD. The 5xxx series Ryzens can be used on very old mobos like several X370 or B350 based ones after a BIOS update. The previous generation is also not bad at all, I'm very happy with the 3900X. I'm contemplating on upgrading to 5900X since people seem to be switching to 5800X3D and there's a supply of used CPUs.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline   Reply With Quote
Old 15th May 2022, 10:02   #102  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,041
Quote:
Originally Posted by takla View Post
niche cases like TR=>10 with pel=2 or 4 at 4K.
'Big' tr-values really more naturally for 4K and more because with the same amount of light passed via input aperture of camera the samples at 4K tesselation of image receive 4 times less light in compare with 1080. Also if fill-factor of sensors is not great - even less.
So for 4K we need about 4 times more tr to have same relative shot noise in compare with 1080.

Also the new motion vectors low pass processing become more effective with not very small tr values. At least >5..6. So the number of position samples may be covered with impulse kernel of about 10 size. Also if I understand right to make good low-pass filter with cut-off of about 0.1 of the max (Nyquist) band we also need to process not very few samples (may be about 10).

The value noise of samples really cause 'position noise' because the position of image area (patch/block) defined by values of its samples only. So noised values of completely static block start to be translated in the position noise with current motion-search algoriphms (and may be with any motion search algo). After receiving not-zeroed coordinates for completely static block the blending engine start to motion-compensate with errors-noised vectors and it will cause blurring of block. For slow moving blocks it also cause not-perfect motion compensation and blurring. Also blending with shifted blocks with non-zero average value cause some shifting of block in the output frame so performed conversion of value noise in the position noise and MPEG encoder need to find and encode this error-movement too.
The effect of false noise-caused movement may become more visible with decreasing of block size. So the MVLPF processing may be more effective at smaller block sizes (like 8x8 or even 4x4).
So low-pass filtering of motion vectors after search for current pair of src+ref frames in big enough time domain using the nature of real images not have too fast and random moving of blocks can decrease these negative effects. But to perform filtering good enough it need to gather big enough number of motion vectors values of the block in consecutive frames.

The need of creating AVS-interface (script-accessible of samples data for read and write) to the MV data passed from MAnalyse to MDegrain really high. It will allow script-writers to create more advanced algoriphms of intermediate motion search data processing before MDegrain blending in the script form. And after slow experiments the best methods may be encoded into compiled functions of MDegrain or some new compiled filters of mvtools-pack for fast execution.
I think of creating motion-data clip in the RGB16 format with mapping of 16bit (dx,dy,SAD) data to R,G,B channels. But real current motion-data 'frame' have also some header of about 256 bytes inside with analysis data. So need to find if this header can be transfered as metadata unchanged and not in 'active' part of frame.

Last edited by DTL; 15th May 2022 at 10:20.
DTL is offline   Reply With Quote
Old 20th May 2022, 19:38   #103  |  Link
takla
Registered User
 
Join Date: May 2018
Posts: 182
Quote:
Originally Posted by DTL View Post
'Big' tr-values really more naturally for 4K and more because with the same amount of light passed via input aperture of camera the samples at 4K tesselation of image receive 4 times less light in compare with 1080. Also if fill-factor of sensors is not great - even less.
So for 4K we need about 4 times more tr to have same relative shot noise in compare with 1080.
Oh wow. I didn't think about this but it makes perfect sense.
takla is offline   Reply With Quote
Old 27th May 2022, 05:05   #104  |  Link
takla
Registered User
 
Join Date: May 2018
Posts: 182
AVX512 is coming you said @DTL?
takla is offline   Reply With Quote
Old 27th May 2022, 11:16   #105  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,041
AVX1024 is coming in mid of 202x if this civilization will not die too fast. I hope it will have even larger register file size to perform more operations in fastest available memory in chip. AMD chips with at least AVX512 support is also good.

From https://clickthis.blog/en/sluhi-o-pr...e-k-2025-godu/


Currently expected the most of performance boost from AVX2/AVX512 operations strategic redesign of MVtools are internal sub-sample shift and scale for both MAnalyse (onCPU processing) and MDegrain (onCPU processing) for pel 2 and 4. It will greatly reduce memory requirement for pre-calculated super clip and it looks very important for new massive multicore chips and current cache AVS design. Also as core become more and more faster in compare with main host memory it should finally make internal scale/shift processing faster in compare with pre-calculated upscaled planes for pel 2 and 4.

I also interested when intel hardware will provide DX12-ME API. It looks intel UHD Graphics 750 still do not have either drivers designed or required hardware features to expose this interface from MPEG encoder.

"I didn't think about this but it makes perfect sense."

Same is about HFR feature of new video systems - the max possible frame accumulation time with HFR is lower so number of accumulated photons per frame per object's view also lower.

So MDegrainN works as 'secondary level video camera' allowing to extend photon's flux accumulation time over the frame's exposure time in 'first level time sampling video camera'. It is only about real optical video cameras (not about digitally synth images). The more image data carriers (photons) accumulated per object's view - the more precision got. The object's view less distorted (noised) by natural photon's shot noise. So the possible tr value is in best case unlimited. Practically may be limited to typical cutscene duration that is several seconds minimum and with even old 24/25 fps it is about many 10'ths or 100'ths.

So the host RAM-based solution allow much higher tr values in compare with full onAccelerator processing (the max tr multiplied to threads number is very limited by accelerator's memory size to hold all required frames). Only sending current+ref pairs to accelerator to ME-process as it work now. Actually it now upload to accelerator current frame once and only send new refs per one output frame of MDegrainN.

Last edited by DTL; 27th May 2022 at 13:09.
DTL is offline   Reply With Quote
Old 27th May 2022, 17:03   #106  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,980
Methinks you take on monster task DTL, mucho respecto, grassy ass
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 27th May 2022, 17:15   #107  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,041
The more 'monster task' is to add more transforms support in MVtools - like rotate, scale, lighting (additive and multiplied levels change) and may be more. MVtools with only translate transform support are still at the very beginning of development.

Last edited by DTL; 27th May 2022 at 17:20.
DTL is offline   Reply With Quote
Old 28th May 2022, 04:44   #108  |  Link
TDS
Formally known as .......
 
TDS's Avatar
 
Join Date: Sep 2021
Location: Down Under.
Posts: 965
Quote:
Originally Posted by DTL View Post
Hi DTL, I had an error with this build with SMDegrain. (In RipBot264)

I can't tell you what the error was other than it was something to do with Line 645, of SMDegrain 3.5.0d

https://github.com/Dogway/Avisynth-S...5.0d.avsi#L645

Or maybe this isn't meant to be used with SMDegrain
__________________
Long term RipBot264 user.

RipBot264 modded builds..
TDS is offline   Reply With Quote
Old 28th May 2022, 05:31   #109  |  Link
takla
Registered User
 
Join Date: May 2018
Posts: 182
Quote:
Originally Posted by DTL View Post
Same is about HFR feature of new video systems - the max possible frame accumulation time with HFR is lower so number of accumulated photons per frame per object's view also lower.
Yep. But the higher frame rate also allows your brain to process more informations. This basically cancels that downside. (When comparred to the same video with a lower frame rate, anyway)

Last edited by takla; 28th May 2022 at 05:35.
takla is offline   Reply With Quote
Old 28th May 2022, 07:02   #110  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,041
"higher frame rate also allows your brain to process more informations. "

Nowdays before viewing we typically need to very highly compress moving picture data to broadcast or stream via IP or torrents. So MDegrain is mostly part of MPEG compression to make the MPEG output rate and filesize as low as possible with given quality level. I now can make rips from HD1080i sources with x264 crf=18 at medium output rate down to about 4000 kBit/s for low-movement documentaries.

"Line 645"

It is just end of large 'messages output'. Need real AVS error message to see what may be wrong. As pinterf ask it should be compatible with all old scripts when no new params are used (internal defaults should be equal to 2.7.45 release). But to use new features it is required to manually edit scripts like SMDegrain to add new params in no-default values.

Last edited by DTL; 28th May 2022 at 07:08.
DTL is offline   Reply With Quote
Old 28th May 2022, 07:50   #111  |  Link
TDS
Formally known as .......
 
TDS's Avatar
 
Join Date: Sep 2021
Location: Down Under.
Posts: 965
Quote:
Originally Posted by DTL View Post
"Line 645"

It is just end of large 'messages output'. Need real AVS error message to see what may be wrong. As pinterf ask it should be compatible with all old scripts when no new params are used (internal defaults should be equal to 2.7.45 release). But to use new features it is required to manually edit scripts like SMDegrain to add new params in no-default values.
That "message output" IS SMDegrain v3.5.0d !!!

I don't understand what you are saying here, it's a LOT over my head...I would think that Dogway would need to rectify this, so as to use your Mvtools2 build(s).
__________________
Long term RipBot264 user.

RipBot264 modded builds..
TDS is offline   Reply With Quote
Old 28th May 2022, 10:15   #112  |  Link
kedautinh12
Registered User
 
Join Date: Jan 2018
Posts: 2,153
You need show more detail about error. That not only said line 645. What is error in line 645?? You need sceenshot when error show with ripbot
kedautinh12 is offline   Reply With Quote
Old 28th May 2022, 10:15   #113  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,041
If your error is with command-line AVSmeter: What is the error text ? Where this 645-line data come from ?

The 645 line at github is 2 }} - this either end of "SHOW PANEL" block or end of all SMDegrain() function. Anyway it is not command or filter or function call line so error may be anywhere above this line (or in any function call from above block).

Last edited by DTL; 28th May 2022 at 10:20.
DTL is offline   Reply With Quote
Old 28th May 2022, 12:39   #114  |  Link
TDS
Formally known as .......
 
TDS's Avatar
 
Join Date: Sep 2021
Location: Down Under.
Posts: 965
Quote:
Originally Posted by kedautinh12 View Post
You need show more detail about error. That not only said line 645. What is error in line 645?? You need sceenshot when error show with ripbot
I can't give anymore info, as all AVSMeter displays is the path where SMDegrain v3.5.0d is, and that Line 645 is where the error is, and AVSMeter stops, at that point.

I don't know why it chooses that line which is simply }}, and I guess that's where Dogway needs to have a look.

That's ALL I can do...sorry.
__________________
Long term RipBot264 user.

RipBot264 modded builds..
TDS is offline   Reply With Quote
Old 30th May 2022, 05:54   #115  |  Link
takla
Registered User
 
Join Date: May 2018
Posts: 182
@DTL newest release gives me this error which goes away if I delete line 5. Is dithering no longer valid?
takla is offline   Reply With Quote
Old 30th May 2022, 08:48   #116  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,041
Oh - I need to check it. Dithering in AVS ConvertBits() from 16 to 8 - is not in MDegrain. But may be output of MDegrain somehow cause this error. If you feed MDegrain with 16bit it should output 16bit too (with right params or defaults ?) and it can not not perform dithering internally. The DX12-ME mode MAnalyse only can use 8bit YV12 now so you need to make conversion in the script EZdenoise internally if you use 16bit input.

Where is the version of EZdenoise that you use in that script ?

Last edited by DTL; 30th May 2022 at 08:54.
DTL is offline   Reply With Quote
Old 30th May 2022, 18:20   #117  |  Link
takla
Registered User
 
Join Date: May 2018
Posts: 182
@DTL fixed it like this:

Code:
function EZdenoise(clip Input, int "thSAD", int "thSADC", int "TR", int "BLKSize", int "Overlap", int "Pel", bool "Chroma")
{
thSAD = default(thSAD, 150)
thSADC = default(thSADC, thSAD)
TR = default(TR, 3)
BLKSize = default(BLKSize, 8)
Overlap = default(Overlap, 4)
Pel = default(Pel, 1)
Chroma = default(Chroma, false)

Super = Input.ConvertBits(16).MSuper(Pel=Pel, Chroma=Chroma)
Multi_Vector = Super.ConvertBits(8).MAnalyse(Multi=True, Delta=TR, BLKSize=BLKSize, Overlap=Overlap, Chroma=Chroma)

Input.ConvertBits(16).MDegrainN(Super, Multi_Vector, TR, thSAD=thSAD, thSAD2=thSAD/2, thSADC=thSADC, thSADC2=thSADC/2)
}

Last edited by takla; 30th May 2022 at 20:23.
takla is offline   Reply With Quote
Old 30th May 2022, 20:27   #118  |  Link
DTL
Registered User
 
Join Date: Jul 2018
Posts: 1,041
I catch this crash with 16bit source: It happens only in release build in MAnalyse in SAD 16bit SSE function. Need some time to find what is wrong. I not touch that part from beginning.

Current workaround may be to feed 8bit to MAnalyse and 16bit to MDegrain.

This looks like working:
Code:
function EZdenoise(clip Input, int "thSAD", int "thSADC", int "TR", int "BLKSize", int "Overlap", int "Pel", bool "Chroma")
{
thSAD = default(thSAD, 150)
thSADC = default(thSADC, thSAD)
TR = default(TR, 3)
BLKSize = default(BLKSize, 8)
Overlap = default(Overlap, 4)
Pel = default(Pel, 1)
Chroma = default(Chroma, false)

Super = Input.MSuper(Pel=Pel, Chroma=Chroma)
Super8 = Input.ConvertBits(8).MSuper(Pel=Pel, Chroma=Chroma)
Multi_Vector = Super8.MAnalyse(Multi=True, Delta=TR, BLKSize=BLKSize, Overlap=Overlap, Chroma=Chroma)

Input.MDegrainN(Super, Multi_Vector, TR, thSAD=thSAD, thSAD2=thSAD/2, thSADC=thSADC, thSADC2=thSADC/2)
}
DTL is offline   Reply With Quote
Old 30th May 2022, 20:38   #119  |  Link
takla
Registered User
 
Join Date: May 2018
Posts: 182
@DTL new issues
mvtools2 from printerf MvTools2 2.7.45 and your newest release is no longer bit identical with same settings (post 117)

MvTools2 2.7.45 pinterf
time=21.380s
501.153 KB
ProRes

mvtools2 DTL
time=19.116s
500.510 KB
ProRes

The last build that was bit identical was https://github.com/DTL2020/mvtools/r.../r.2.7.46-a.05

Last edited by takla; 30th May 2022 at 22:43.
takla is offline   Reply With Quote
Old 30th May 2022, 21:50   #120  |  Link
ChaosKing
Registered User
 
Join Date: Dec 2005
Location: Germany
Posts: 1,795
Quote:
Originally Posted by takla View Post
@DTL new issues
mvtools2 from printerf MvTools2 2.7.45 and your newest release is no longer bit identical with same settings (post 117)
Maybe because some bugs were fixed!?
__________________
AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth
VapourSynth Portable FATPACK || VapourSynth Database
ChaosKing is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 16:18.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.