Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
1st March 2011, 15:31 | #382 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
Additional oddities stem from the fact that (all!) plugins *should* be compiled with the corresponding libraries of the targetted Avisynth version. A plugin for vanilla 2.5.8 should be compiled differently for 2.5.8.MT should be comiled differently for 2.5.7.MT should be compiled differently for 2.6 should be compiled differently for 2.6.MT.
If not, be prepared to experience problems. Has it already been noted that Avisynth MT is an unutterable, giant mess?
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
1st March 2011, 21:10 | #383 | Link |
Registered User
Join Date: Aug 2005
Location: UK
Posts: 35
|
This statement about resize horizontal first for speed-up intrigues me ... Can I ask a couple of stupid questions in terms of sequence then ?
1. Should you perform the horizontal resize before QTGMC ? 2. For point 1 should it be new horizontal x cropped height ? I appreciate these are not totally specific to QTGMC, but it seems people are talking about speed-ups and I'm just not seeing this when I test it. Of course I can perform lots of different tests to come up with the answer although I'm guessing given the posts in this thread so far there is a "known" way to achieve this... As an example if I have source of 544x576 and have crop of 2,2,-2,-4 this gives me an effective image size of 538x572. If I'm working to 640xwhatever (accepting this is upscaling), should I therefore have something like:- Crop(2,2,-4,-2) Spline36Resize(640,572) QTGMC( Preset="Faster", EdiThreads=4 ).SelectEven() Spline36Resize(640,352) Last edited by dirk362; 1st March 2011 at 21:18. |
1st March 2011, 21:38 | #384 | Link |
Registered User
Join Date: Jul 2010
Posts: 448
|
Ah, no... that trick is only a speed-up if you're downscaling. QTGMC's speed is directly proportional to the number of pixels in the material it is given. So give it the lowest possible resolution. If you're downscaling, doing the horizontal resize first gives QTGMC less to work on. You can't do the vertical rescale before QTGMC because you'd destroy the interlacing.
However, if you're upscaling you should QTGMC the original smaller resolution. |
1st March 2011, 22:01 | #385 | Link | |
Registered User
Join Date: Aug 2005
Location: UK
Posts: 35
|
Quote:
But it now makes sense, so thanks for imparting knowledge. If we didn't ask we wouldn't learn |
|
4th March 2011, 10:09 | #386 | Link |
partially-informed layman
Join Date: Jan 2002
Location: Bangkok, Thailand
Posts: 314
|
I have written a tutorial for preparing video for the web using the following workflow:
I was going to use TDeint+nnedi3 as the recommended deinterlacing method until I discovered QTGMC. -Vit- and Didée (not to mention tritical et al), thanks for this brilliant script. The tutorial is here and we're discussing it on the Sony Vegas Pro forum here. If anyone has any feedback, please let me know. John Meyer tried it an made some interesting findings regarding multi-threading QTGMC, which can be read about 30 posts down that thread. I've asked him to report his findings here too. |
4th March 2011, 11:18 | #387 | Link | |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
Quote:
The key is resolution. With SD input, getting full CPU load is no problem at all. However, with 1080i it's a problem indeed. CPU usage is much lower then. For the exact reasons - don't ask me. It must be related to all the memory shuffling that's going on under the hood. Somewhere there is a borderline, and 1080i is definetly located on the other side.
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
|
4th March 2011, 13:43 | #388 | Link |
Registered User
Join Date: Jul 2010
Posts: 448
|
Ideal thread counts differ by hardware, by settings and by source resolution. Also by what you're encoding with/to. On default settings I run SD at 13 threads for best speed, full HD at 8 threads. I have seen people indicate anywhere from 2 to 14 threads as their ideal. With HD material 4 threads is a good start, and it can be tricky to get stability and balance memory with more. But it is certainly not a hard limit - don't make the mistake of thinking that threads are tied to physical cores (it's possible for a developer to specify this, but it's not the norm).
Last edited by -Vit-; 4th March 2011 at 14:27. |
4th March 2011, 14:14 | #389 | Link |
Registered User
Join Date: Jul 2010
Posts: 448
|
Regarding multithreading HD material, you need to work SetMemoryMax properly. But I haven't seen it explained in detail anywhere, so here goes:
SetMemoryMax effectively sets two memory limits: - The number you provide is the maximum memory used for the avisynth cache* - (2Gb - the number you provide) is the maximum memory reserved for everything else used by your script (e.g. avisynth itself, NNEDI3, MVTools) - that's on a 32-bit workflow - With 64-bit avisynth, it's more like (available physical memory - number you provide) reserved for plugins etc. That means as you increase SetMemoryMax, you are reducing the amount of memory available for the plugins like NNEDI3. Plugins working on HD or complex settings will need considerable memory. Even more so when multi-threading because each thread needs plugin memory. So if you set memory max too high -> script failure. [BTW. That's a reason to reduce EdiThreads, it may not have an impact on the speed, but it might free up memory for more main threads] The nature of a cache means that it always will fill up whatever space it is given (except in trivial examples). But it only actually needs to cache those frames that will be reused. Once those are in the cache, making it larger will have no positive effect at all, but it will reduce memory available for plugins. That means there's a sweet spot, where you've reserved just enough to cache the needed frames and no more. So you do not simply set the value larger and larger for higher resolutions. Go above the sweet spot and you may exhaust memory with no benefit. Go below the sweet spot and it will slow down because frames are not found in the cache (or more usually it will crash - there must be bugs in the avisynth caching). I find the sweet spot for HD around 700-800 but YMMV. At some point you hit a limit, where: (memory needed for the avisynth cache) + threads x (memory needed for plugins) > your available memory. The formula depends on the factors I noted in the last post - it sets a limit on the number of threads you can run. It might not be enough threads to max out your CPU. * The memory max figure is actually just a target. Internally, if avisynth desperately wants to cache something it can exceed the value you set. That's one reason why the behaviour is so hard to predict. Last edited by -Vit-; 4th March 2011 at 15:47. |
5th March 2011, 19:20 | #390 | Link |
Registered User
Join Date: Jul 2010
Posts: 448
|
The latest version, v3.20, is in the first post. There are three areas of improvement:
Documentation Most documentation has been stripped from the script itself and put in a separate HTML file. In Avisynth livery. All the new features below are explained in detail in there. Noise processing / Denoising I've improved the noise processing in several ways and made it simpler to use: - Supports chroma noise processing / denoising (set ChromaNoise=true, it defaults to false) - Supports dfttest as an alternative to fft3dfilter. I added this when I saw what strange results fft3dfilter produces on chroma denoising. I might post about that experience elsewhere. dfttest seems to identify noise vs detail more accurately - Can provide a motion compensated clip to the denoisers, which further improves noise vs detail detection - New "EZ" modes of operation which allow denoising or grain retention with just two settings. You set a value to indicate the strength of effect and a preset to determine quality of processing (presets from "Slower" to "Faster" only - used to make choices for the other settings: denoiser, motion compensation etc.). Examples: Code:
# Automatic denoising, value is denoising strength, preset is quality/speed tradeoff QTGMC( Preset="Slower", EZDenoise=2.5, NoisePreset="Slow" ) # Automatic grain retention - set amount of grain to retain, can be > 1.0 QTGMC( Preset="Slower", EZKeepGrain=1.0, NoisePreset="Faster" ) QTGMC now sets various uniquely named globals to give the calling script access to its motion vectors and more. Saves time recalculating vectors. Refer to the documentation (the "External Linkage" section) for complete details. Here are some examples: Code:
# Use QTGMC motion vectors for our own smoothing QTGMC( Preset="Medium", ForceTR=2, SubPel=2 ) # Ensure we get temporal radius of 2 and match Pel setting with next line super = MSuper( pel=2, levels=1 ) MDegrain2( super, QTGMC_bVec1,QTGMC_fVec1, QTGMC_bVec2,QTGMC_fVec2 ) # Use QTGMC-created motion vectors # QTGMC can reuse its own vectors. Here repairing progressive material with serious deinterlacing artefacts t = QTGMC( Preset="Slower", InputType=2 ) b = QTGMC( Preset="Slower", InputType=3, PrevGlobals="Reuse" ) # Reuse motion vectors from first call for a good speed-up Repair( t, b, 1 ) ____ Full Version Changelog: * Motion vectors made available to calling script (see External Linkage), added related GlobalNames, PrevGlobals and ForceTR settings * Added simplified noise processing settings: EZDenoise, EZKeepGrain and NoisePreset * Support for dfttest as denoiser for noise processing - added settings Denoiser and DftThreads * Added ChromaNoise setting to optionally enable chroma noise processing * Added DenoiseMC setting to use motion-compensated clip for denoising, improves detection of noise vs detail * Replaced fft3dfilter-specific BT setting with NoiseTR setting to set temporal radius for denoiser. * Added StabilizeNoise setting to "stabilize" & "deshimmer" noise restored in noise bypass (was part of "Generate" mode) * Dropped NoiseRemove setting, now always removes all noise specified by given Sigma (enough controls already!) * MotionBlur renamed to ShutterBlur, MBlurLimit renamed to SBlurLimit, DetailRestore renamed to GrainRestore (old names gave wrong impression) * Bug fixes: stabilization of "Generate" noise mode, manually enabling SLMode with SourceMatch * Documentation converted to HTML Last edited by -Vit-; 5th March 2011 at 21:43. |
7th March 2011, 11:20 | #394 | Link |
Registered User
Join Date: Apr 2009
Posts: 478
|
Many thanks -Vit- for the continual development of this awesome script!
I have been wondering one thing... AFAIK, there isn't a 64-bit version of Dfttest right? Yet when I modified my 64-bit script to use Dfttest as a denoiser, to my surprise it worked. So I was wondering, might there be a bug in the script that isn't calling Dfttest properly? Here's my QTGMC settings, for reference. QTGMC(Preset="Slower",ChromaNoise=True,denoiser="dfttest",DenoiseMC=True) |
7th March 2011, 16:47 | #396 | Link |
Registered User
Join Date: Apr 2009
Location: Martin, Slovakia
Posts: 79
|
Here is dfttest x64: http://forum.doom9.org/showthread.php?t=152800 but only an older version
for chroma denoising support. I have couple questions:
|
7th March 2011, 22:34 | #397 | Link |
Registered User
Join Date: Jul 2010
Posts: 448
|
nhope: Thanks for the updates. I've added them to the OP
aegisofrime: Are you sure you don't actually have the 64-bit dfttest? Those settings will certainly select it. To test if dfttest is really being used, temporarily remove its dll from your plugins folder and try to run your script. If it comes up with a "can't find dfttest" error then you really were using dfttest. pokazene_maslo: 1. NoiseDeint is only relevant if you are have either GrainRestore > 0 or NoiseRestore > 0, i.e. if you retaining grain rather than denoising. It is set automatically by NoisePreset. [This reminds me that I didn't mention the obvious: EZDenoise and EZKeepGrain are mutually exclusive] 2. Yes, that is intentional. Firstly, dfttest doesn't support interlaced input. Secondly, it's impractical to motion-compensate interlaced input for the denoisers (DenoiseMC=true). So I use a point resize on the fields to get something I can feed to the denoisers that matches the motion vectors. The point resize will not blur the noise and the denoisers seem to be able to handle the line doubling. After the denoise I drop the doubled lines and reweave the result to get noise to match the original interlace. Or put more simply, I implement "interlaced=true" manually in the script. Last edited by -Vit-; 8th March 2011 at 14:41. |
8th March 2011, 03:57 | #399 | Link |
Registered User
Join Date: Jul 2010
Posts: 448
|
aegisofrime: Just looked again at your QTGMC line. Although you're selecting dfttest you're not actually doing any denoising, that's why it isn't showing any error. I think I should make the documentation clearer and maybe rename the NoiseBypass setting to NoiseProcess
|
8th March 2011, 04:06 | #400 | Link | |
Registered User
Join Date: Apr 2009
Posts: 478
|
Quote:
|
|
|
|