Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Closed Thread
 
Thread Tools Search this Thread Display Modes
Old 27th January 2011, 18:22   #181  |  Link
osgZach
Registered User
 
Join Date: Feb 2009
Location: USA
Posts: 676
Silly question if you don't mind. If I feed it a progressive clip, is it going to make a "this is combed" decision and skip frames, or is it going to process every single frame in the clip?

There is a lot of material to read through, so I'm sorry if this was asked/answered before. I'm generally interested in using it to clean up IVTC'd material where the stream result is progressive obviously, but sporadic combing will occur. I could see using this for Clip2 operations, but if it doesn't do "needless" processing where there is no combing, it may be easier to just throw it on at the end of the script in some cases.


edit: ok, its runs about 15 fps using "QTGMC( Preset="Medium", InputType=1 )" which is realistically fine I guess.. It seems to be pretty much identical to the source. So ignoring the fact I am deliberately slowing my script down more than I -need- to. If I wanted to use TFM and set PP to 0 (don't hint, don't attempt to deinterlace) and then just tack QTGMC on at the end, to catch and fix any combing (surprisingly I couldn't find any real combing but I used the most intensive match mode so thats probably why.) It would be a valid and correct usage, right? I just hate tweaking thresholds and all that stuff so I'd take a time increase vs having to tweak per file all the time.

I'm sure I could adjust a few things to make it go faster and not ruin the quality, too.

Last edited by osgZach; 27th January 2011 at 20:10.
osgZach is offline  
Old 27th January 2011, 21:22   #182  |  Link
-Vit-
Registered User
 
Join Date: Jul 2010
Posts: 448
The script makes no attempt to detect combing. It runs a temporal binomial smooth on every frame to remove bob-shimmer when deinterlacing. This same process is also effective in remove deinterlacing residue from progressive video. But it isn't selective on which frames to process. I've seen others run a QTGMC in your situation, but it's not a workflow I need so I can't really comment further.
-Vit- is offline  
Old 27th January 2011, 22:42   #183  |  Link
osgZach
Registered User
 
Join Date: Feb 2009
Location: USA
Posts: 676
Fair enough. If anyone else has been using it for this purpose, I'd be interested to hear their take as well.
osgZach is offline  
Old 27th January 2011, 23:02   #184  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,389
What kind of "residual" combing? Low-value combing (from mpeg compression, because the matched fields come from different coded frames)? Or fully interlaced frames because of lone fields that couldn't be matched?

Because - when I hear (or see) "residual combing", my first thought is "vinverse". Not "QTGMC". ...
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline  
Old 27th January 2011, 23:45   #185  |  Link
osgZach
Registered User
 
Join Date: Feb 2009
Location: USA
Posts: 676
Oh jeezus Didee, you scared me for a second. I thought I said "residual combing" somewhere and used the wrong term.. Bad Didee!

To Clarify: I am simply talking about combing left in frames either from doing no post-processing whatsoever (TFM PP=0), or combing that was not picked up after field matching for whatever reason (Mic below threshold, etc).. I'm not a "tweaker" at heart, I prefer "run it once and its done" operations, I guess. So I was just inquiring if it were a proper use or not. McBob/TGMC are just too slow for me, but QTGMC seems to be fast enough I don't mind waiting as much.

IIRC AnimeIVTC uses Vinverse doesn't it? I have always been curious about it but its options have always been a little intimidating (no offense). And seems like "more tweaking" for this "non-tweaker at heart"
osgZach is offline  
Old 28th January 2011, 19:16   #186  |  Link
maki
Registered User
 
Join Date: Nov 2009
Posts: 12
What settings should I use to decomb and deinterlace this progressive content. I also want to remove noise/dirt artifacts that appear around the lips, that's pretty much the only place where they appear.

I noticed that deinterlace/jaggy lines and noise only appear when I downsize the video to 720p.

I am new to the whole decoming and deinterlace in a video. And I don't really understand the difference between jagginess and deinterlace. Can someone explain it to me. I've read about it but I don't really get it.

Also I applied this plugin to the video and it seems that coming appears in every frame:

Quote:
http://img15.imageshack.us/i/piczb.png/

AVCSource("C:\00007.track_4113.dga")
FieldDeinterlace(full=true, map=true)
LanczosResize(1280,720) # Lanczos (Sharp)
Trim(29560, 31846)
#deinterlace
#crop
#denoise
ConvertToYV12()
Can someone give me some nice settings to fix all of the above.
maki is offline  
Old 28th January 2011, 23:40   #187  |  Link
Fullmetal Encoder
Registered User
 
Join Date: Jan 2011
Posts: 107
Has anyone had any success getting the Very Slow preset, with or without any variations, to work with any speed/stability using 3.0 on a Core i7+ system? At this point I've run 44 different tests in which I've altered SetMTMode and SetMemoryMax and every test produces either less than 1fps output (usually much less) and/or crashes. Varying EdiThreads doesn't seem to have any effect. The basic script I am using is:

Code:
SetMemoryMax(700)
SetMTMode(5,4)
Import("C:\QTGMC3.0.avsi")
LoadCPlugin("F:\ffmpegsource-2.14-mt-avs64\ffms2.dll")
FFVideoSource("D:\X.mkv")
SetMTMode(2)
Trim(1753, 3542)
QTGMC(Preset="Very Slow", MatchPreset="Medium", SourceMatch=3, Lossless=2, MatchEnhance=0.6, Sharpness=0.25, NoiseBypass=2, NoiseRemove=1.0, DetailRestore=0.5, NoiseRestore=0.1, Sigma=3, NoiseDeint="Generate")
The MKV I'm using as a source is an interlaced 1080P clip. I realize the QTGMC settings I am using are rather aggressive but they are an example that I believe -Vit- posted previously. What I am trying to accomplish is to determine a set of very high quality, nearly lossless settings which are still functional, to then tweak the Avisynth settings to see how high I can push output performance, and then to do some quality comparison tests on various clips to determine whether it's worth it to stick with those settings or reduce quality in exchange for speed. I am using JoshyD's (04/16/2010) Avisynth on a Core i7 860 with Windows 7 Professional 64-bit and 8GB of RAM.

Using similar QTGMC (slow, very slow) settings which Avisynth settings give you the best performance? What kind of system are you running? So far, all I can conclude is that it's either impossible to run the above QTGMC settings on any modern system or that there is something seriously wrong with my set-up.
Fullmetal Encoder is offline  
Old 29th January 2011, 00:45   #188  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,389
Way too complex for 1080 video. Are you even halfway aware of the amount & kind of operations that QTGMC is doing with these settings?

Running single-threaded, SetMemoryMax(1200) gives me 0.25 fps. SetMemoryMax(1400) gives 1.1 fps, however Vdub now has peaked at 1.85 GB. SetMemoryMax(1440) loads, but crashes upon seeking. Of course, such memory load doesn't allow for e.g. direct x264 encoding. I don't get the script to run using 4 Threads via SetMTmode(2), even setmmax(500) throws kernelbase after a few frames.

Way too complex for 1080 video. And breaking a fly on a wheel, anyway.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline  
Old 29th January 2011, 01:41   #189  |  Link
-Vit-
Registered User
 
Join Date: Jul 2010
Posts: 448
I designed the settings to make a steady gradient from "as fast as possible" to "unnecessarily accurate". The slowest settings are all about ultimate precision and completely ignore speed as a factor. Also, none of the presets take any account of resolution, so you need to. Settings like yours are visually great for big upscales on SD, but unless you have a huge display, you simply won't see the difference at the very slowest settings at HD resolutions.

If you must get it to run, a 64-bit workflow is required (check) and with regards threading work upwards from 1 main thread + 1 EdiThread. I find having EdiThreads lower than main threads helps on my system (I use a 12Gb i7 930 and a similar 950). Render lossless, do your x264 in a second pass.

Regarding tweaks other than the presets, you can get a speed boost from MatchEdi2="Bob". Also NoiseDeint="DoubleWeave" will be faster and might make little difference, depending on your source. Make sure your SetMemoryMax is tweaked - wrong setting there can slow things down. Check the presets table in the script for other ideas.

From my experience, the stability problems come from three points:
(a) Memory issues: 32-bit avisynth running out of memory (overeager / buggy caching?), the black art of SetMemoryMax, leaky plugins - thankfully these issues are less noticable on 64-bit
(b) Buggy plugins: many plugins inherit threading issues from avisynth (see below), but some have other bugs too. At this very moment I'm rewriting AddGrainC from scratch because I think it is causing crashes for me, I can almost guarantee a crash if I try very complex HD scripts with NoiseDeint="Generate" (uses AddGrain), and I don't get them without. This may be your problem. Try setting NoiseDeint="DoubleWeave" or NoiseBypass=0.
(c) Avisynth multithreading. In early experiments with QTGMC I had serious problems with crashes on 1080p even at medium settings. Traced the problem to the avisynth<->plugin threading (synchronization problems with ref counts usually). SEt has fixed some of these in his MT avisynth builds, but they affect "avisynth.h" which needs to be compiled into all the QTGMC plugins. I've done that with all the 32-bit versions and also added a hideous hack of my own, and it helped a lot. However, I still need to resolve a few issues with the 64-bit versions and test a bit more before I could post them (all VS2010 and IC2011 - some conversion issues need checking).

So even if you can get a speed you can live with, it's certain that as your resolution and script complexity increases, so does your chance of avisynth instability. I have had to tone down scripts from the level I wanted because of this in the past.

Last edited by -Vit-; 29th January 2011 at 02:08.
-Vit- is offline  
Old 29th January 2011, 01:56   #190  |  Link
Fullmetal Encoder
Registered User
 
Join Date: Jan 2011
Posts: 107
Quote:
Originally Posted by Didée View Post
Way too complex for 1080 video. Are you even halfway aware of the amount & kind of operations that QTGMC is doing with these settings?

Running single-threaded, SetMemoryMax(1200) gives me 0.25 fps. SetMemoryMax(1400) gives 1.1 fps, however Vdub now has peaked at 1.85 GB. SetMemoryMax(1440) loads, but crashes upon seeking. Of course, such memory load doesn't allow for e.g. direct x264 encoding. I don't get the script to run using 4 Threads via SetMTmode(2), even setmmax(500) throws kernelbase after a few frames.

Way too complex for 1080 video. And breaking a fly on a wheel, anyway.
Ok, thanks for the feedback. Yes, I know it is quite complicated and while I understand some of the basics of what it's doing I am not aware of its intricacies, especially when it comes to its use of mvtools and masktools, so probably, no Just wanted to get some input from others before proceeding with any other testing in case it would end up being a waste of time. And throughout my testing I have started to get the impression that what I was attempting to do was really to kill a mosquito with a bazooka.

The more I read up on this plugin though the more impressed I am with what it's able to do so thanks to -Vit- for his work on it and to you Didée for the underlying TGMC.
Fullmetal Encoder is offline  
Old 29th January 2011, 02:15   #191  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,389
The "most important" application IMHO is on SD content, anyway. When SD content is cheaply bob/deinterlaced and viewed on a big screen, then you'll see the downsides from bad deinterlacing in all glory, because the upscale makes everything more objectable.
On 1080i, this is so much less of concern, because no upscale is taking place. Even the "simple" deinterlacers result in an acceptable viewing experience. If I personally use Q/TGMC on 1080i, then it surely is with pretty fast settings. Even the most simplistic settings give like, say, 80% of what Q/TGMC can achieve. And those 80% are already way ahead of what other deinterlacers can achieve. See this older sample from 1 year back: old TGMC with most-fast and most-basic settings, versus YadifMod with NNEDI2 interpolation. TGMC was much faster, still it let Yadifmod/NNEDI2 stand with egg in their face.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline  
Old 29th January 2011, 16:08   #192  |  Link
TheSkiller
Registered User
 
Join Date: Dec 2007
Location: Germany
Posts: 632
Thanks for the new version, just tried it - looks great with the new SourceMatch on my MiniDV stuff.
But with another type of source that requires rep0=0 there is something odd I noticed: whenever I use any of the lossless modes, especially without SourceMatch (SourceMatch=0) I pretty much always have combing left in the output. With version 2.51 this was not the case (version 2.51's fake lossless "Lossless=1" was free of remaining combs for me).

I also find the fake lossless mode to have a lot more remaining combing than the true lossless mode Lossless=1 (which doesn't make sense), even with the default rep0=4.


Edit: Sorry, forgot to add: I'm using rep0=0 because otherwise static parts (Text) laying on top of the motion would have an ugly up and down bobbing occasionally on this source (is there a better way to fix this?). The described problem with combing never occurs with rep>0, and never with QTGMC 2.51.

Edit: cleaned my post, got messed up somehow.

Code:
QTGMC(2,2,1, 0,0,4, Lossless=2, SourceMatch=0) #lots of combs (but steady text)

Last edited by TheSkiller; 29th January 2011 at 19:07.
TheSkiller is offline  
Old 29th January 2011, 19:31   #193  |  Link
STC-Fan
MeGUI / AviSynth user
 
Join Date: Jul 2004
Location: England
Posts: 31
Excellent results

Tried this out today with one of my PAL VHS captures - really impressed with it. Using QTGMC with just the "Slow" preset has produced a massive improvement when compared with my current filter choice in AviSynth, which is MDegrain2i2 + Cnr2k + Yadif.

My only problem now is that I get a Visual C++ runtime error along the lines of "This application has requested the Runtime to terminate it in an unusual way", which happens after a short while when I try to do a 2-pass encode with x264 in MeGUI, which I am guessing is down to the program running out of memory.

I've only tried SetMemoryMax() with a value of 512 so far, and that hasn't resolved the problem. Presumably, I will need to increase the value to 1024 or more, and close Firefox when the encode is running along with any other programs.

Before it crashed, I was getting around 5.6 FPS on the first pass, which is fairly reasonable. I'm a bit surprised that even with SetMTMode(2) that I can't get more on a Core 2 Quad Q6600 - I suppose I could try a faster mode in QTGMC, or get myself a computer with a bit more power; I'm thinking along the lines of something from Cray, to be honest :P

I put up some comparison screenshots and sample scripts - the best thing is the reduction of jagged edges on the graphics, which the old trio of filters couldn't really do much about:

Original frame | AviSynth script

MDegrain2i2 + Cnr2k + Yadif | AviSynth script

QGTMC 3.0, Slow preset | AviSynth script

Overall, though, it's a very good script, this - I think that QTGMC3 could even be better than Cillit Bang / Easy-Off BAM
__________________
When all else fails... "DOT CRAWL. RAINBOWS. CHROMA BLEEDING NOISE. They're a challenge for some household video cleaners - but not for CILLIT BANG. BANG! - and the VHS head switching noise is gone!"
STC-Fan is offline  
Old 29th January 2011, 20:02   #194  |  Link
-Vit-
Registered User
 
Join Date: Jul 2010
Posts: 448
Quote:
Originally Posted by TheSkiller View Post
...there is something odd I noticed: whenever I use any of the lossless modes, especially without SourceMatch (SourceMatch=0) I pretty much always have combing left in the output.
Some of what you say doesn't surprise me - it's a very intricate script now and I may have overlooked some combinations of settings. However, I don't get more combing with "fake" lossless than "true", the only difference between those modes is when they occur in the script, before or after the final smooth. Still, I'll have a look at the rep modes and see what I find...

Quote:
Originally Posted by STC-Fan View Post
Visual C++ runtime error
I listed the range of things that causes stability issues a couple of posts above. I also posted some notes on performance and MT use. One other important thing I note in the OP is using the right version of AviSynth. I can only get extended stability with SEt's 2.6MT. Make sure all your other plugins are up to date too. A SetMemoryMax of 512 should be OK for Preset="Slow" alone (provided your overall RAM is not low), but maybe tweaking it might help. Sometimes it needs to be lowered.

If you do get it stable, try tweaking the thread-count as described in that overview I mentioned. You may get better speeds with more or less threads.
-Vit- is offline  
Old 3rd February 2011, 13:57   #195  |  Link
matfra
Registered User
 
Join Date: Jul 2009
Posts: 111
I have a basic question for you.
MT("""
QTGMC(tr0=0,tr1=1,tr2=0,EdiMode="nnedi3",rep0=4,rep1=0,rep2=4,sharpness=1.00)
""",threads=4, overlap=4)

Is it the best way to run this script with MT ?
Also what is SetMemoryMax(). Does it really speed up the encoding ?
Merci !
matfra is offline  
Old 3rd February 2011, 14:00   #196  |  Link
matfra
Registered User
 
Join Date: Jul 2009
Posts: 111
Im wondering if its better to apply some denoising and sharpening in with QTGMC, or if its better to apply a simple Deinterlaced, then denoise and sharpen after.

What do you recommand guys ?
matfra is offline  
Old 3rd February 2011, 14:03   #197  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,718
Quote:
Originally Posted by -Vit- View Post
I listed the range of things that causes stability issues a couple of posts above. I also posted some notes on performance and MT use. One other important thing I note in the OP is using the right version of AviSynth. I can only get extended stability with SEt's 2.6MT. Make sure all your other plugins are up to date too.
For some reason, I've had much better stability using tsp's last MT build of Avisynth 2.5.7. For two weeks I played around with SEt's two builds but was never able to finish an x264 encode or encode to lossless intemediate file without Avisynth crashing. (For what it's worth, I had a script with Deblock_QED + MCTemporalDenoise in it).
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is offline  
Old 3rd February 2011, 18:00   #198  |  Link
-Vit-
Registered User
 
Join Date: Jul 2010
Posts: 448
@matfra

You should not use MT("...") with QTGMC, you should use SetMTMode(...). SetMemoryMax is not a speed-up generally - it sets a balance of memory for your script to work - you might not need it.
Here is a template script, adjust it as you need:
Code:
SetMemoryMax(700)   # This line may not be necessary - depends on your settings, number of threads and available memory. Try 400,600,700,800,1000,1200...
SetMTMode( 5, 4 )     # Use 4 threads, adjust to suit your machine
WhateverSource(...)   # FFVideoSource, DGDecode_mpeg2source, AviSource, whatever your source requires
SetMTMode(2)

QTGMC( ... , EdiThreads=2 ) # Set number of threads for NNEDI3 (more or less than setting above), add any other settings you require
QTGMC provides some sharpening and denoising control, but only you can decide if it is good enough, or if other processing is needed. In other words, try it both ways and decide for yourself which is better.
-Vit- is offline  
Old 3rd February 2011, 18:28   #199  |  Link
ajp_anton
Registered User
 
ajp_anton's Avatar
 
Join Date: Aug 2006
Location: Stockholm/Helsinki
Posts: 805
Is it a good idea to merge this with MCTemporalDenoise into one single giant function so that MCTD can re-use the motion vectors from QTGMC?
ajp_anton is offline  
Old 3rd February 2011, 18:31   #200  |  Link
STC-Fan
MeGUI / AviSynth user
 
Join Date: Jul 2004
Location: England
Posts: 31
Quote:
Originally Posted by -Vit- View Post
I listed the range of things that causes stability issues a couple of posts above. I also posted some notes on performance and MT use. One other important thing I note in the OP is using the right version of AviSynth. I can only get extended stability with SEt's 2.6MT
Thanks for that - I've actually been able to resolve the biggest issue (slow speed encoding leading to a crash) by reverting back to using MT mode 5 before AVISource, then using MT mode 2 for QTGMC and other plugins. In fact, I'm now getting nearly 12 FPS for first and second passes - not 16 or 20 like with MDegrain2i2, but the payoff in restored detail and better correction of interlacing artifacts etc. is well worth it

As for the memory running out - on the last script I ran for a 6-minute video, I increased SetMemoryMax to 768 and this has eliminated crashes for the time being. However, I have recently had BSOD crashes with the MEMORY_MANAGEMENT error on Vista (SP2), so I might need to run the memory checker and possibly replace the RAM I have.

The other filter I sometimes use is fizick's DeFreq for removing RF (diagonal line) noise, so I'll check and see if that will work harmoniously with QTGMC.

EDIT: poisondeathray - my source video is all TFF, I have tested your suggestion and the 50 FPS output from QTGMC is now nice and smooth, thanks for the tip! The equivalent settings I used for Yadif were mode=1 (double framerate) and order=1 (top field first).

I've also tested the output on some more noisy VHS clips which need DeFreq (placed before QTGMC) and they looks very nice as well. With those clips (thankfully only a few) there is still a small amount of background noise crawl and feint horizontal lines, although in the final output from x264 much of this isn't really visible anyway. Probably I could mess around a bit with QTGMC's parameters in detail, beyond the standard Slow preset, but I'm more than satisfied with the results at the moment

UPDATE: I've had a look at some stronger denoising to go with the detail restoration / deinterlacing from QTGMC. I've found that for my VHS content, I can get better results by using the MDegrain2i2 function before QTGMC(Preset="Slow") - normally I would just use one or the other. With both of these in one script, they work extremely well together, though it does hit encoding time to x264 somewhat (down to about 6.5 FPS instead of 12 FPS with just QTGMC @ Slow). Despite this, the results are definitely worth it. I have uploaded some standard size screenshots, 4x magnified sections of those screenshots and my most recent AviSynth scripts here:

QTGMC + MDegrain2i2, QTGMC alone and MDegrain2i2 alone
__________________
When all else fails... "DOT CRAWL. RAINBOWS. CHROMA BLEEDING NOISE. They're a challenge for some household video cleaners - but not for CILLIT BANG. BANG! - and the VHS head switching noise is gone!"

Last edited by STC-Fan; 6th February 2011 at 17:34. Reason: Tested suggestion from poisondeathray, other filter testing, fixed URL
STC-Fan is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 10:28.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.