Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Closed Thread
 
Thread Tools Search this Thread Display Modes
Old 25th January 2011, 09:55   #161  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,394
Wow. That surely was lots of work. Respect!

Any thoughts about making the searchclip/processing better suited for mixed progressive/interlaced sources?

* * * * *

BTW, now that the script is over 1000 lines and comes closer to 100 kB script size, I can't resist to post this - just for fun:

Code:
i  = last
b  = i.bob(.0,.5)
t  = b.temporalsoften(1,255,255,25,2).merge(b,0.25)
s1 = t.blur(1.4).blur(1.4).merge(t,0.25).msuper()
s2 = b.msuper(levels=1)
b.mdegrain1(s2,s1.manalyse(isb=true),s1.manalyse(isb=false)).merge(b,0.25)
sharpen(.2,.8).repair(b,1)
The hard core of TGMC. Isn't it cute?
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline  
Old 25th January 2011, 14:28   #162  |  Link
-Vit-
Registered User
 
Join Date: Jul 2010
Posts: 448
Quote:
Originally Posted by Didée View Post
Wow. That surely was lots of work.
This version only started as a little experiment a few months ago, but then it kinda snowballed... 1000 lines and half of them are comments! And still to add YUY2 and chroma speed-ups

Quote:
Originally Posted by Didée View Post
Any thoughts about making the searchclip/processing better suited for mixed progressive/interlaced sources?
This does interest me as I have sources like that. Now I assume you mean without using some IsCombed variant, otherwise it's trivial. But then for each frame how do you know whether to bob-interpolate or not? There are [effectively] two frame rates in such sources. There may be a trick to it - I'll give it some thought.

Quote:
Originally Posted by Didée View Post
The hard core of TGMC. Isn't it cute?
Shall I add it...? Preset="Cute"
____

BTW: Just noticed that I've broken the "Draft" preset. That's not too important so I'll wait to see if there are any other issues before posting a fix.

Last edited by -Vit-; 25th January 2011 at 19:39.
-Vit- is offline  
Old 25th January 2011, 15:54   #163  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,138
It would also be interesting to know whether there are good settings to use with fieldblended sources to get the good frames (output from SRestore, that is) as detailed as possible. The detection clip most likely needs to be treated in a different way from QTGMC.

Thanks for the new version
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is online now  
Old 25th January 2011, 16:04   #164  |  Link
elguaxo
Registered User
 
elguaxo's Avatar
 
Join Date: Jun 2006
Posts: 260
Quote:
Originally Posted by boulder View Post
it would also be interesting to know whether there are good settings to use with fieldblended sources to get the good frames (output from srestore, that is) as detailed as possible. The detection clip most likely needs to be treated in a different way from qtgmc.
+1
__________________
Doom10
elguaxo is offline  
Old 25th January 2011, 19:09   #165  |  Link
-Vit-
Registered User
 
Join Date: Jul 2010
Posts: 448
Quote:
Originally Posted by Boulder View Post
It would also be interesting to know whether there are good settings to use with fieldblended sources to get the good frames (output from SRestore, that is) as detailed as possible. The detection clip most likely needs to be treated in a different way from QTGMC.
I've had similar questions by PM. Unfortunately I don't work with fieldblended sources, and have barely used SRestore so I haven't looked at this issue. Maybe I'll get a chance to look at this, but I'm afraid it isn't a priority at the moment.

____

From some questions coming in I realize that despite that mass of detail earlier, I didn't properly address the practical issues of speed and memory. The new features require more memory and CPU time. They require you know how to run demanding scripts in avisynth. The defaults are tweaked for a particular balance of performance and quality, but you can tweak in either direction

Speed
Source-match performs 1 or 2 extra interpolations. So there are up to three interpolations per frame. The interpolators are determined by:
EdiMode="XXX" : the main interpolation, always occurs - nothing new here
MatchEdi="XXX" : Basic source match - for all source-match modes. Defaults to same as main interpolation, change if using a slow choice like "EEDI3+NNEDI3" as your main
MatchEdi2="XXX" : Refined source match - SourceMatch=2,3. Defaults to high speed NNEDI3. Can use "Bob" here with little penalty

Using "Bob" with SourceMatch=2,3 is a good performance boost for those modes:
Code:
QTGMC( Preset="Slower", SourceMatch=3, MatchEdi2="Bob" )
When using NNEDIx or EEDIx for source-match the interpolation speed can be tweaked. These default to much faster than the main preset:
MatchPreset="XXX" - typically "Fast" or quicker still
MatchPreset2="XXX" - typically "Very Fast" or quicker
____

Memory + Multi-Threading
Multiple NNEDI3s and complex QTGMC settings like noise bypass use memory. If you run out of memory you often get a crash. Use SetMemoryMax(XXX) at the start of the script to help. You might need higher values, or maybe lower, it depends on the source resolution, settings, SetMTMode use etc. - try 400,600,700,800,1000,1200...

Using multi-threading (SetMTMode) will help speed things up, but if you're on 32-bit avisynth you have a good chance of running out of memory if you go for very high settings, even on SD material, certainly on HD material. Again use SetMemoryMax, limit the number of threads in SetMTMode, and also use the EdiThreads setting of QTGMC - this last point is very important for source-match. 64-bit avisynth makes things easier - you still need to consider the same points, but you will have more latitude (depending on your machine).

An example multi-threading tweak to limit threads and allow a bit more memory, this is just an example, each case is different:
Code:
SetMemoryMax(700)
SetMTMode(5,4)       # 4 threads for avisynth
WhateverSource(...)
SetMTMode(2)

# Three interpolations when using SourceMatch=2,3. Two defaulting to NNEDI3 here, plus a Bob. Set 2 threads for *each* NNEDI3
QTGMC( Preset="Slower", SourceMatch=3, Lossless=2, MatchEdi2="Bob", EdiThreads=2 )

Last edited by -Vit-; 25th January 2011 at 19:36.
-Vit- is offline  
Old 25th January 2011, 22:04   #166  |  Link
pokazene_maslo
Registered User
 
Join Date: Apr 2009
Location: Martin, Slovakia
Posts: 79
I'm trying to figure out how to use "ShowNoise". Whatever parameters I use, I get only plain grey picture, no detail, no noise. What am I doing wrong?
Code:
# Set DAR in encoder to 31 : 17. The following line is for automatic signalling
global MeGUI_darx = 31
global MeGUI_dary = 17
LoadPlugin("D:\Install\meGUIx64\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("D:\Download\VTS_01_1.d2v", info=3)
LoadPlugin("D:\Install\meGUIx64\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, interlaced=true, threads=0)

QTGMC( Preset="Slow", ShowNoise=8.0, sigma=2)
pokazene_maslo is offline  
Old 26th January 2011, 00:44   #167  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,138
Quote:
Originally Posted by -Vit- View Post
I've had similar questions by PM. Unfortunately I don't work with fieldblended sources, and have barely used SRestore so I haven't looked at this issue. Maybe I'll get a chance to look at this, but I'm afraid it isn't a priority at the moment.
Would you like to have a sample to look at? The problem with fieldblended material IMO is that the motion is not "real motion". The underlying source is basically always progressive.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...
Boulder is online now  
Old 26th January 2011, 01:46   #168  |  Link
-Vit-
Registered User
 
Join Date: Jul 2010
Posts: 448
Quote:
Originally Posted by pokazene_maslo View Post
I'm trying to figure out how to use "ShowNoise".
You need to enable noise bypass for this to work:
Code:
# I suggest you explicitly set NoiseDeint - it defaults to "DoubleWeave", "Generate" is best, you can also set "Bob"
QTGMC( Preset="Slow", NoiseBypass=1, ShowNoise=8.0, sigma=2, NoiseDeint="Generate" )
I suppose it should get switched on automatically if you set ShowNoise. I'll add that to the 3.01 version, which I'm using to fix all the inevitable minor issues like this.

____
Quote:
Originally Posted by Boulder View Post
Would you like to have a sample to look at?
It would be good to have a typical example of the problem from someone who deals with it. Still I don't expect to be able to look at this issue for some time...

Last edited by -Vit-; 26th January 2011 at 01:50.
-Vit- is offline  
Old 26th January 2011, 04:30   #169  |  Link
pokazene_maslo
Registered User
 
Join Date: Apr 2009
Location: Martin, Slovakia
Posts: 79
Heh I was assuming that it can set up all needed parameters automatically. Thanks.

Now I have another problem. I was trying to find appropriate sigma but I have found that I need to denoise also chroma planes to get good results. How it is with "plane=4" support? Are there needed any other changes in your script to get "plane=4" support?
pokazene_maslo is offline  
Old 26th January 2011, 04:54   #170  |  Link
-Vit-
Registered User
 
Join Date: Jul 2010
Posts: 448
Quote:
Originally Posted by pokazene_maslo View Post
Heh I was assuming that it can set up all needed parameters automatically. Thanks.

Now I have another problem. I was trying to find appropriate sigma but I have found that I need to denoise also chroma planes to get good results. How it is with "plane=4" support? Are there needed any other changes in your script to get "plane=4" support?
Sorry, it currently only denoises luma. That's temporary. There are quite a few changes needed to add chroma denoising, but it's the next version that will be focusing on the chroma planes. Maybe you should denoise as a separate step.

Last edited by -Vit-; 26th January 2011 at 04:58.
-Vit- is offline  
Old 26th January 2011, 17:00   #171  |  Link
dansrfe
Registered User
 
Join Date: Jan 2009
Posts: 1,212
n00b question. Ok so I'm slightly confused. Does QTGMC work with fieldblended PAL material (with sresotore traling it) and if not then what other sources can it potentially fix?
dansrfe is offline  
Old 26th January 2011, 19:31   #172  |  Link
txporter
Registered User
 
Join Date: Nov 2009
Posts: 110
Quote:
Originally Posted by dansrfe View Post
n00b question. Ok so I'm slightly confused. Does QTGMC work with fieldblended PAL material (with sresotore traling it) and if not then what other sources can it potentially fix?
It can be used on fieldblended PAL material, but srestore needs to have another clip to detect the blended frames.

I use something like this for converting NTSC Torchwood/Dr Who/Robin Hood/etc DVDs:
Code:
setmtmode(5,0)
loadcplugin("C:\program files\avisynth 2.5\plugins\yadif.dll")
o = MPEG2Source("Torchwood.S1E01-clip.d2v",cpu=3)
setmtmode(2)
y = o.yadif(mode=1,order=1)
qtgmc = o.QTGMC(Preset="Ultra Fast",rep2=4,SLmode=1,SLrad=1)
qtgmc.Srestore(frate=25,dclip=y)
txporter is offline  
Old 26th January 2011, 20:05   #173  |  Link
kutjong
Registered User
 
kutjong's Avatar
 
Join Date: Nov 2007
Posts: 114
I'm currently trying to repair some material that was sloppily converted from interlaced to progressive using what seems to be weave deinterlacing (lots of combing artifacts).

So far, I'm using this:

QTGMC( Preset="Slower", InputType=2, ProgSADMask = 10.0, SourceMatch=3, Sharpness=0.4, TR2=3 )

This makes substantial repairs, but there are still some combing artifacts in few frames.

Are there any other variables that I should experiment with?
I've read the documentation but embarrasingly most of it is just way over my head.
kutjong is offline  
Old 26th January 2011, 21:20   #174  |  Link
-Vit-
Registered User
 
Join Date: Jul 2010
Posts: 448
Quote:
Originally Posted by kutjong View Post
I'm currently trying to repair some material that was sloppily converted from interlaced to progressive using what seems to be weave deinterlacing (lots of combing artifacts).

So far, I'm using this:

QTGMC( Preset="Slower", InputType=2, ProgSADMask = 10.0, SourceMatch=3, Sharpness=0.4, TR2=3 )

This makes substantial repairs, but there are still some combing artifacts in few frames.

Are there any other variables that I should experiment with?
I've read the documentation but embarrasingly most of it is just way over my head.
That's a good start. Simplest thing to do is lower ProgSADMask, potentially to 0.0.

InputType=2 tries to rebuild a shimmer-free result from the even lines (fields), InputType=3 does the same using the odd lines. Each loses detail from the other lines. The following script runs both modes. It means some redundant processing, but it does improve detail and reduce artefacts:
Code:
c = WhateverSource("Your.src")

# Two versions of fixed source (use TR2 and ProgSADMask also if you wish)
t = c.QTGMC( Preset="Slower", InputType=2, SourceMatch=3, Sharpness=0.4 )
b = c.QTGMC( Preset="Slower", InputType=3, SourceMatch=3, Sharpness=0.4 )

# Now merge the results in some way. Choose ONLY ONE of the lines below and delete the other
Merge( t, b )                 # Smooth result, may need more sharpening
t.Repair( Merge( t, b ), 1 )  # Sharper result, could try 2 at end of this line
There are almost certainly even better ways to merge the results, this was just a quick idea...

One of the minor fixes I'm sitting on at the moment in version 3.01 is allowing Lossless modes for InputType=2,3. That helps with detail, but I guess that's not your problem.
____

You can do some serious script abuse by chaining together progressive QTGMCs. Generally it's not a good idea because the repeated motion compensations are both redundant and drift in accuracy which causes smearing. You also get gradual over-sharpening. [Although sensible uses like the one above makes me think a toolbox approach might be beneficial]

So another approach for your situation, that I don't recommend but you might want to try, is to run a QTGMC( InputType=1 ) on the result of your QTGMC( InputType=2 ). It will certainly clean up the artefacts, but it may cause smearing / oversharpening:
Code:
QTGMC( Preset="Slower", InputType=2, SourceMatch=3, Sharpness=0.4 ) # Plus any other settings
QTGMC( Preset="Slower", InputType=1, Sharpness=0.1 )                # Don't use source-match here

Last edited by -Vit-; 27th January 2011 at 00:23.
-Vit- is offline  
Old 27th January 2011, 01:43   #175  |  Link
AnXioZ
Registered User
 
Join Date: Jan 2011
Posts: 6
Excuse me for the n00b question but, do I loose any vertical resolution if I decide to keep the framerate of the interlaced source? In example, I have a 1080i60 (60 fields/s | 59.94 frames/s playback) bluray source, and I want to encode it to 1080p60 (59.94 frames/s). In my experiments I haven't seen any noticeable decrease in quality after deinterlacing, but I just wanted to know of the side-effects.
Thank you in advance.
AnXioZ is offline  
Old 27th January 2011, 01:51   #176  |  Link
osgZach
Registered User
 
Join Date: Feb 2009
Location: USA
Posts: 680
one field is thrown away, and the data from the other is used /interpolated to keep the same resolution, if I recall correctly. That's how it usually works. What you see is what you get basically... So if you are previewing and choose same rate deinterlacing, and you see the same image size as the source, then that is what you get
osgZach is offline  
Old 27th January 2011, 02:39   #177  |  Link
-Vit-
Registered User
 
Join Date: Jul 2010
Posts: 448
No, you won't really lose any resolution. Deinterlacing effectively increases vertical resolution.

Put simply: your 1080i60 source is really a 60fps stream of 540 pixel high images (fields). Each pair of fields is interlaced (weaved) into a single 1080 pixel high frame, resulting in a 30fps stream of interlaced frames. Deinterlacing separates the fields again and doubles their height to give a 60fps stream of true 1080 pixel high frames. Compare with the starting point: the vertical resolution is increased.

The new pixels are created by intelligent interpolation of other pixels in the same field and pixels in neighboring fields. No fields are thrown away during this process. If you decide to drop half the frames after deinterlacing to keep the original 30fps rate, that's fine - no quality is lost on those frames you keep.

The original fields in your source are sometimes altered a little in the deinterlaced output to ensure that the result is smooth and shimmer-free. But there should be very little detail loss (especially with the latest features). In fact, the newest developments mean the doubled height output can make a good estimate of lines that are not actually stored in the source. That's what the Source_Comparison example was showing in my earlier post. This compares a source recorded with 720 pixels of vertical resolution against a QTGMC deinterlacing of 360 pixel high fields (i.e. I deliberately threw away half the pixels and got QTGMC to try and work out what they were)

Last edited by -Vit-; 27th January 2011 at 02:49.
-Vit- is offline  
Old 27th January 2011, 02:56   #178  |  Link
AnXioZ
Registered User
 
Join Date: Jan 2011
Posts: 6
Thank you for the input guys. Much appreciated.

Quote:
Originally Posted by -Vit- View Post
No, you won't really lose any resolution. Deinterlacing effectively increases vertical resolution.

Put simply: your 1080i60 source is really a 60fps stream of 540 pixel high images (fields). Each pair of fields is interlaced (weaved) into a single 1080 pixel high frame, resulting in a 30fps stream of interlaced frames. Deinterlacing separates the fields again and doubles their height to give a 60fps stream of true 1080 pixel high frames. Compare with the starting point: the vertical resolution is increased.

The new pixels are created by intelligent interpolation of other pixels in the same field and pixels in neighboring fields. No fields are thrown away during this process. If you decide to drop half the frames after deinterlacing to keep the original 30fps rate, that's fine - no quality is lost on those frames you keep.

The original fields in your source are sometimes altered a little in the deinterlaced output to ensure that the result is smooth and shimmer-free. But there should be very little detail loss (especially with the latest features). In fact, the newest developments mean the doubled height output can make a good estimate of lines that are not actually stored in the source. That's what the Source_Comparison example was showing in my earlier post. This compares a source recorded with 720 pixels of vertical resolution against a QTGMC deinterlacing of 360 pixel high fields (i.e. I deliberately threw away half the pixels and got QTGMC to try and work out what they were)
Thank you for the great expiation -Vit-, and thank you for the great work!
AnXioZ is offline  
Old 27th January 2011, 03:28   #179  |  Link
AnXioZ
Registered User
 
Join Date: Jan 2011
Posts: 6
Hey -Vit-, I just installed the new 3.0 version. I am still playing around with it, but one thing I noticed was that when I set SourceMatch=3 the CPU utilization drops significantly. Without it I'm getting ~98% utilization, with SourceMatch I'm getting ~70%. Am I doing something wrong or is that expected with the new enchantments?
AnXioZ is offline  
Old 27th January 2011, 04:00   #180  |  Link
-Vit-
Registered User
 
Join Date: Jul 2010
Posts: 448
CPU under-utilization often means that there are cores idle on a multi-core machine. E.g. One core is busy running a key step in the script, another core is trying to run a task that needs the result - if there are no other tasks to do then the second core idles (it's rather more intricate than that, but you get the idea). Run more threads, give your cores more to do. Run multi-threaded using SetMTMode, or increase the thread count if you're using it already. I posted some tips on that a few posts ago in this thread. Also NNEDI3 runs two or three times with source-match and NNEDI3 defaults to running multi-threaded. You can tweak its thread usage with EdiThreads=x. Try tweaking x, it might help (or not).

Last edited by -Vit-; 28th January 2011 at 20:53.
-Vit- is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 11:18.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.