QTGMC for film based?

[Stereodude]

I hadn't tried MCTD before, nor for that matter MCTDMod, so I gave them both a spin (I wasn't sure which you were referring to). For each test encode they both resulted in almost identical file sizes. Is there a significant difference between the two I'm not seeing yet?I was referring to MCTemporalDenoise. When you say "two" are you referring to MCTemporalDenoise and QTGMC or MCTemporalDenoise and MCTemporalDenoiseMod?

FWIW, I wasn't referring to file size, but visual appearance.
For the noisy source I used the videos I linked to in an earlier post. One of them is the definition of horrible quality, the other 720p and not low quality as such, just noisy.
I ran MCTD using the default settings. Comparing it to QTGMC's EzDenoise function (progressive mode), "better at noise reduction" seems to be eluding me so far. I might need to start out aiming for "as good" to begin with.....If you're bored, start with the Blu-Ray of Moneyball and compare QTGMC to MCTemporalDenoise.

My typical setting is MCTemporalDenoise(settings="low", radius=3), if the video is really noisy like the aforementioned Moneyball I will use MCTemporalDenoise(settings="medium"). If I want something in between "low" and "medium" I use this:
MCTemporalDenoise(settings="low", radius=3, sigma=6, SHmode=3, useEEDI2=false, \
quant1=30, quant2=60, ECrad=4, ECmode="Removegrain(3,-1)", maxr=3, TTstr=2, \
GFthr=1.6, thSAD=400, thSAD2=400, thSCD1=400)

Don't get me wrong. I still use QTGMC. I use it on nearly all content that started out in interlaced 1080i mpeg-2 files. Even things that I IVTC or that were progressive encoded as interlaced. Interlaced MPEG-2 artifacts survive otherwise and setting QTGMC to any of the progressive settings like InputType=1 or InputType=2 doesn't get the job done in my experience. So, I generally use this combination:

QTGMC( Preset="Medium", InputType=0, Sharpness=0.2, ShowSettings=false ).SelectEven()
MCTemporalDenoise(settings="low", radius=3)

[Atlantis]

Or if you wouldn't mind uploading a small sample of the original video which results in combing after TIVTC, I'd be happy to try it myself.
Here it is. I call it the episode from hell. :devil: Try a normal tfm on it.

https://www.mediafire.com/?btumea3vep38crh

[Atlantis]

Strange.

I am using tdecimate(hybrid=1) which works great for the opening titles which are detected combed.

However I have noticed in the show where a scene change is a ghost. Half of previous scene and half of the next scene ghosted together. This behavior is normal when the 2 scenes are detected combed because of hybrid=1 option.

However these 2 shots blended together are reported Clean Frame by tfm and they are not combed. When I use tfm only, the transition is clean. When I add the tdecimate(hybrid=1), the transition is a half-half ghost frame. Why?

Shouldn't hybrid=1 affect only frames detected as combed?

[Stereodude]

Here it is. I call it the episode from hell. :devil: Try a normal tfm on it.

https://www.mediafire.com/?btumea3vep38crhI'll try it when I get home from work.

[Stereodude]

Shouldn't hybrid=1 affect only frames detected as combed?Not according to the documentation. It's blend decimation of 30p sections into 24p. If tdecimate thinks it's a video section (perhaps caused by the lack of a repeated frame in a sequence of 5 frames) it's going to do something with it.
Here it is. I call it the episode from hell. :devil: Try a normal tfm on it.This is pretty easily handled by this:
QTGMC( Preset="Medium", InputType=0, Sharpness=0.2, ProgSADMask=10, ShowSettings=false )
SRestore(frate=23.976)

[Atlantis]

This is pretty easily handled by this
Yes that is kind of what I did. tfm fails there.

What's the difference between SRestore and tdecimate? I used tdecimate.

[Atlantis]

Not according to the documentation. It's blend decimation of 30p sections into 24p. If tdecimate thinks it's a video section (perhaps caused by the lack of a repeated frame in a sequence of 5 frames) it's going to do something with it.
I always thought that tfm tells tdecimate what is combed and what is not.

When tdecimate gets the footage it's already deinterlaced and if tdecimate decides alone on its own if something is video or not, that's not good I think. I have to stop using hybrid=1 in that case despite of not using it makes the opening credits jumpy.

Any documentation on how tdecimate marks a section as video 30p?

[Stereodude]

What's the difference between SRestore and tdecimate? I used tdecimate.SRestore is a deblender. tdecimate looks for duplicated frames.

[Atlantis]

This is pretty easily handled by this:
QTGMC( Preset="Medium", InputType=0, Sharpness=0.2, ProgSADMask=10, ShowSettings=false )
SRestore(frate=23.976)
According to the manual ProgSADMask only applies to InputType=2,3.

[Stereodude]

According to the manual ProgSADMask only applies to InputType=2,3.Well you can delete it if it bothers you having an unused parameter there.

[Atlantis]

I finally did it! Found the "perfect" solution. What I was looking for all along. Something that you would set and forget. This is the script I tested and use now.

myvideo = "video.d2v"
DGDecode_mpeg2source(myvideo)

deint = QTGMC( Preset="Slower" ).SelectEven()
tfm(d2v=myvideo, clip2=deint)

QTGMC( Preset="Slower", InputType=2, ProgSADMask=12.0 )

tdecimate()

This way I don't need to set an overfile and go frame by frame finding bad frames.

By the way, I tried your script but it's too soft and you lose lots of details. The script above retains more details.

[Stereodude]

By soft I think you mean not artificially sharpened, but to each their own...

[Atlantis]

No, you are wrong. By soft I don't mean not artificially sharpened. By soft I mean you lose lots of detail with your script. Using QTGMC like that on the first step makes you lose lots of stuff. Can never come close to using tfm.

[Atlantis]

In this picture attached, on the top is your script and the bottom is mine.

[poisondeathray]

Did you give vinverse a shot ? It's more gentle for residual combing than QTGMC in progressive mode, not to mention much faster

And if you still wanted to use QTGMC in prog mode, wouldn't it make more sense to put it after TDecimate ? (Not only faster because fewer frames, but more accurate because no duplicates every 5th)

Did you want to do something about those rainbows too ?

[Atlantis]

Thanks a lot poisondeathray. No I didn't know vinverse and tried it. Not only it's faster, it retains more details. I revised my script to


myvideo = "video.d2v"
DGDecode_mpeg2source(myvideo)

deint = QTGMC( Preset="Slower" ).SelectEven()
tfm(d2v=myvideo, clip2=deint)
tdecimate()

vinverse()

[hello_hello]

I had a bit of a play with your sample but didn't come up with any particular form of cleverness. I'm definitely no NTSC expert but I'm not sure your sample is a sample of "normal".

It's the first time I've played around with TIVTC settings to any extent. I thought/hoped it'd be a simple case of lowering the detection level in order for the frames in question to be detected as combed. Which it seemed to be initially. The default MI threshold is 80. It's easy enough to use the "MI=?" setting to change it, but that's when things I don't understand began to happen.

Most of the frames where I could see combing in places had quite low MI values according to TFM. 15 or less in most cases. When using the display function so TFM would show me where in the frame it was detecting combing, it was never where I could see it. So I could see something which at least looked like combing but TFM wasn't detecting it as such. That, I don't understand.

From there I tried specifying a very low MI value, thinking that'd cause the frame to be de-interlaced even if TFM and I weren't in agreement as to where it needed it. A PP value less than 5 combined with clip2 appeared to be the requirement for de-interlacing the entire frame, and while it seemed to be "working" it didn't appear to be de-interlacing properly. The areas which looked combed tended to look worse after de-interlacing. I tried QTGMC, TDient and Yadif. It didn't matter.

I then reduced the MI threshold low enough to de-interlace all the frames which appeared to have combing, which in the end was pretty low because TFM was reporting MI values of 1 and 2 for many of them. The only way that'd seem logical at the moment is if somehow what appears to be combing, isn't. The lower I dropped the MI threshold the more frames were de-interlaced, but that seemed be causing a large amount of blurring much of the time. Maybe I was having a slow day, but it took me a while to realise frames were being blended instead of being IVTC'd as I'd expected. That I'm yet to get my head around.... the reason TFM would consider a de-interlaced frame needs to be blended rather than applying IVTC.

Finally I thought maybe I could fix the frame blending problem first and understand the "why" second....
I played with every parameter I could find which looked like it might contribute to stopping the unwanted frame blending but nothing I did made a difference. Except something like hybrid=0, which obviously wouldn't be a good solution when the video in question is hybrid. I even tried disabling hints so TFM wouldn't be telling TDecimate what to do. I can't remember if it had no effect or a negative one, but it definitely didn't fix anything. I'd remember that.

So after playing around, all I seemed to have achieved was to make many of the areas with combing look worse by de-interlacing the frames in question, which in turn caused frames to be blended rather than duplicates removed. Does any of the above sound like it might be considered normal, or is there "oddness" to the video in question I haven't found yet?

I wondered as I was typing.... I was de-interlacing using a script like this (PP and MI parameters etc not included).

deint=QTGMC().SelectEven()
tfm(clip2=deint).tdecimate(hybrid=1)

As it's been mentioned, I'm wondering if instead of the above this might produce a more satisfactory result? I guess I'll have to try it when I get a chance.

deint=vinverse()
tfm(clip2=deint).tdecimate(hybrid=1)

[hello_hello]

Stereodude,
I thought I should let you know I'm not ignoring your post #54. Other stuff got in the way. There's a good chance I won't be experimenting with noise filters again for a day or two. I'll do the replying when I come back to it.
Cheers.

[Atlantis]

I had a bit of a play with your sample but didn't come up with any particular form of cleverness. I'm definitely no NTSC expert but I'm not sure your sample is a sample of "normal".
Oh yes, it's not normal. That's why I call it the episode from hell. Of all the 24 episodes in season 2, it's only this episode that it's this bad.

Also after much testing I'm considering dropping tdecimate(). I will keep the video at 29.97. No matter how hard we try, we can't get a film based framerate. Maybe the show was edited on video, I don't know but even after all that I see some little jumps from time to time. So I will use this

myvideo = "video.d2v"
DGDecode_mpeg2source(myvideo)

deint = QTGMC( Preset="Slower" ).SelectEven()
tfm(d2v=myvideo, clip2=deint)

vinverse()

[poisondeathray]

Also after much testing I'm considering dropping tdecimate(). I will keep the video at 29.97. No matter how hard we try, we can't get a film based framerate. Maybe the show was edited on video, I don't know but even after all that I see some little jumps from time to time.




Are you referring to the main program (not the video titles) ?

Is TDecimate dropping the wrong frame in the cycle ? , or is it a TFM matching issue in the first place ?

Did you tweaking the tdecimate parameters ?

[Atlantis]

Are you referring to the main program (not the video titles) ?

Yes I'm referring to the main program that is recorded on film. I was prepared to accept jumpy video titles if all the rest was ok. But no still I get jumps, they are very close to a scene change. Near the end or beginning of a scene change (Like 2-3 frames to or after a cut). I did a closer examination as you asked and tdecimate() drops a frame that is not duplicate. That's why it looks jumpy.

So always close to an edit. I'm not an expert, I don't know if it has to drop a non-duplicate frame because of an edit cut? Maybe in case it has to drop a non-duplicate frame, it would be better if it drops the very last or very first non-duplicate frame. In that case we wouldn't see the jump.

No I didn't tweak the tdecimate parameters, I don't know what to tweak. Anyway to tell it to drop only the frames adjacent to a cut?

[poisondeathray]

I mentioned this earlier - the only way to get smooth titles and main program, is to use VFR (30p sections run at 30p, 23.976p sections run at 23.976p)

It's hard to say what's going on without examining it. You can look at both TFM and TDecimate's debug views (display=true) to see what decisions they are making

Both have different modes and settings which might be more appropriate . If you upload a section where it makes the wrong choice with your current settings, someone might be a le to suggest better settings to use

IMO, 30p for 24p is almost bad as a jumpy random drops

[Atlantis]

IMO, 30p for 24p is almost bad as a jumpy random drops
Could you please explain why? I get a perfect smooth video with no jumps at 30p. I get little jumps with 24p. Please explain why it's as bad? 30p looks correct where 24p has faults.

[poisondeathray]

Could you please explain why? I get a perfect smooth video with no jumps at 30p. I get little jumps with 24p. Please explain why it's as bad? 30p looks correct where 24p has faults.


If it's "film" - by definition it will be 24p .

30p means every 5th-6th frame will be duplicates, so you will have a cyclical stutter (That's the whole point of decimating after field matching)

If it looks correct at 30p, then perhaps it was shot on video (or at least that section you're referring to was shot on video, not film) . This is common on some DVD's where you have CGI effects (Frazier doesn't have any, but the common reference is Star Trek TNG DVD's). Still the only way to get 24p sections to play at 24, 30p sections to play at 30p is VFR

[Stereodude]

IMO, 30p for 24p is almost bad as a jumpy random dropsWell, you can Hybrid=3 to blend the 24p to 30p. IMHO that's better than judder.

[poisondeathray]

Well, you can Hybrid=3 to blend the 24p to 30p. IMHO that's better than judder.



Blends are the smoother but more blurry, but certainly another option

On a 60Hz display there will always be a 3:2 judder . But most NTSC-land born people are used to the judder. 24p native players on a 120Hz display have no judder, each frame is "evenly spaced" in time, or have the same display time on screen

My opinion is you should try to go back to what the original film content was, 24p, 1:1 frames for the encoding or backup. The telecine here was only added to make it compatible with DVD. If it was released on BD for example, it would have been native 24p. You have the option of applying blends or motion interpolate or various other methods on playback as post processing to smooth it out if you wanted to

[Stereodude]

On a 60Hz display there will always be a 3:2 judder . But most NTSC-land born people are used to the judder.This is where madVR's smooth motion is great.

[hello_hello]

I played around a bit more, and if nothing else at least I have a slightly better understanding of TIVTC.
I still think there's something not quite right though as some of the areas which appear combed still look a little odd after they're de-interlaced to me.

Because TDecimate uses info from TFM when deciding how it's going to do things. I found the only way I could play around successfully was to use TFM by itself in a script, then when everything was being IVTC'D and de-interlaced correctly, add TDecimate to the script and adjust it.

Anyway.... give something like these a go (different ways of doing much the same thing I tried):

deint=QTGMC().SelectEven()
tfm(cthresh=3, MI=15, order=1, clip2=deint)
tdecimate(vidThresh=6, hybrid=1)

deint=QTGMC().SelectEven()
tfm(cthresh=-1, order=1, clip2=deint)
tdecimate(vidThresh=6, hybrid=1)

deint1=Trim(0,189).QTGMC().SelectEven()
deint2=Trim(190,0).QTGMC().SelectEven()
Trim(0,189).tfm(order=1, clip2=deint1).tdecimate(hybrid=1)\
++Trim(190,0).tfm(cthresh=3, MI=15, order=1, clip2=deint2).tdecimate()

If you add "Display=true" to either TFM or TDecimate you can see some of the parameters which can help if you need to adjust them. For TFM it shows you which frames are being treated as "clean" and which ones are being de-interlaced. For TDecimate you can see if frames are considered video or film, and which frames are being dropped or blended.
I found QTGMC way too frustrating while fiddling, so I removed "clip2=deint" until I found which settings worked, adding "clip2=deint" again and checked nothing had changed.

Keeping in mind my understand of this is still fairly vague.....

Each frame is given a value (metric) displayed as an MIC value. You can specify "MI" in the script. The default is 80. If a frame's MIC value is equal to or higher than the MI value it's considered "clean". If you look through frames and determine the lowest MIC for a frame, then specify a lower MI value in the script, it should mean (I think) every frame gets de-interlaced. The trouble is, many frames which appeared to contain combing had MICs of one or zero. When you lower the value for "cthresh" (the default is "9") it increases the MIC value for each frame so you can then set an MI value which de-interlaces every frame.
For your "impossible to tfm" sample, cthresh=3 meant the lowest MIC frame value I could see was 17 (unless a frame was completely black), hence using cthresh=3 and MI=15 in the first script. If you wanted to be sure, you could probably output an overrides file to check more accurately.

Alternatively if you use "cthresh=-1" (second script), every pixel is detected as combed, therefore you don't need to mess around with MI values because everything gets de-interlaced anyway. I'm still not clear on the differences, but I'd assume "cthresh=-1" would mean every frame is de-interlaced completely, whereas without it, adaptive de-interlacing is used? When I encoded the first two scripts, the file size for the second one was a little smaller.

From there I added TDecimate and tried to find a way to stop it blending frames. I don't know why I couldn't do it yesterday, but it worked today. For your sample it was fairly easy to find a value for "vidThresh". I used "Display=true" for TDecimate while previewing the opening graphic. For it to be treated as video and frame blended, "cthresh" needed to be lower than around 7. For the actual video, above 4 seemed to prevent any frame blending, hence vidThresh=6 in the script. If you don't care about the opening graphic, you could just use TDecimate() and it'll drop frames for 23.976, but the graphic does look smoother when it's frame blended.

After messing around for a while it occurred to me to divide the encode into sections. Hence the third script which divides your sample into "opening graphic" and "everything else" with Trim(). Then different settings can be used for each section (it required doing the same for the de-interlacing so "clip2" would work).
Nothing special was needed for the opening graphic. The defaults plus Hybrid=1 worked fine. For the rest of it nothing but the defaults seem to be needed for TDecimate. Without Hybrid=1 it just gets decimated.

I've no idea whether the ways I worked out how to do it are necessarily the best, but they seem to work.

[Atlantis]

I have noticed an interesting side-effect with vinverse().

vinverse() does make the image a little worse when there is edge enhancement present.

It seems they introduced edge enhancements on later seasons of Frasier. When using vinverse those jaggy lines become even worse.

Anything for this? How to avoid vinverse making it worse? Or something else that doesn't affect edge enhancement?

[poisondeathray]

I have noticed an interesting side-effect with vinverse().

vinverse() does make the image a little worse when there is edge enhancement present.

It seems they introduced edge enhancements on later seasons of Frasier. When using vinverse those jaggy lines become even worse.

Anything for this? How to avoid vinverse making it worse? Or something else that doesn't affect edge enhancement?



Don't use vinverse on those episodes? :)

You can reduce the contrasharpen strength in vinverse (default sstr=2.7), or deal with the underlying problem . e.g If there are "jaggy lines" , and vinverse is enhancing those - consider using some antialiasing function first

[DeathAngelBR]

The best results I've had so far with a really shitty annoying VFR source is precisely TFM + QTGMC.

1st pass (analysis)

deint = AssumeTFF().QTGMC( Preset="slow", SourceMatch=2, Sharpness=0.4, TR2=1, noiseprocess=0 ).selecteven()
tfmtxt = "tfm_QTGMC.txt"
decitxt = "deci_QTGMC.txt"
tfm(d2v=_d2v_,order=1,slow=2,pp=4,clip2=deint,output=tfmtxt).tdecimate(mode=4,output=decitxt)

Then I open the output file written by tfm and look for all frames marked as combed, and manually check for any residual combing around those frames. This is when the override file comes in.

2nd pass (encode)

tfm(d2v=_d2v_,order=1,slow=2,pp=4,clip2=deint,ovr=ovrtxt,input=tfmtxt).tdecimate(mode=5,hybrid=2,tfmin=tfmtxt,input=decitxt,mkvout=mkvtxt)
deint = AssumeTFF().QTGMC( Preset="slow", SourceMatch=2, Sharpness=0.4, TR2=1, noiseprocess=0 ).selecteven()
tfmtxt = "tfm_QTGMC.txt"
decitxt = "deci_QTGMC.txt"
ovrtxt = "override.txt"
mkvtxt = "mkvout_QTGMC.txt"
tfm(d2v=_d2v_,order=1,slow=2,pp=4,clip2=deint,ovr=ovrtxt,input=tfmtxt).tdecimate(mode=5,hybrid=2,tfmin=tfmtxt,input=decitxt,mkvout=mkvtxt)

Instead of lowering MI and cthresh values which possibly results in deinterlacing frames that aren't really combed, you should try the override feature.

(I *had* to use pp=4 because the default pp=6 and others output really awful frames in particular scenes.)

[hello_hello]

Using the overrides function is possibly the best way, but more time consuming. I'm not sure I understand how pp=4 and clip2=deint work together in your second pass script as I'm still fairly new to playing with TFM settings.
For the clip in question I'm not sure frames with combing and frames marked as combed are necessarily grouped together as such.

I tried several methods of de-interlacing when using a script which causes every frame to be de-interlaced, and with Atlantis's sample the only way to get away with it was to use QTGMC. The side-effects were pretty minimal. If anything, it sharpened by a very small amount, although that can have a slight negative effect on straight lines. Every other de-interlacing method looked terrible.

For example:

tfm(cthresh=-1, order=1, pp=5)
tdecimate(vidThresh=6, hybrid=1)

http://i.imgur.com/rE8Evmn.jpg

tfm(cthresh=-1, order=1, pp=6) # pp=4 looks much the same
tdecimate(vidThresh=6, hybrid=1)

http://i.imgur.com/EW46lvV.jpg

deint=QTGMC().SelectEven()
tfm(cthresh=-1, order=1, clip2=deint)
tdecimate(vidThresh=6, hybrid=1)

http://i.imgur.com/LhDImO2.jpg

tfm(cthresh=-1, order=1, pp=1) # no deinterlacing
tdecimate(vidThresh=6, hybrid=1)

http://i.imgur.com/BlERJaV.jpg

Full disclosure:
I saved each image using jpg's maximum quality. When I downloaded each image from my post (to make sure I'd placed them in the correct order by comparing file sizes) I realised the hosting service had re-compressed the last image. It must have just exceeding some sort of file size limit. Instead of 100% quality, it's 90%.

[DeathAngelBR]

pp=4 and clip2=deint work precisely the same way in both passes. The only difference is that using a tfm input file speeds things up a bit in the 2nd pass.

This is the problem I had with default pp=6:

http://i.imgur.com/zQLsg1i.png

[hello_hello]

I'm probably not asking the question the right way.

The TFM help file says:
If pp => 5 then TFM will build the mask as usual and only pixels in moving areas will be taken from the frames in clip2.
If PP < 5 then frames will be taken as is from clip2.

Would I be correct in assuming the upshot of that is, QTGMC deinterlaced the whole frame in your second pic, whereas for the first one, the parts of the frame considered static would have been left untouched?

If that's correct then I've still just got to get my head around how it works in conjunction with whether a frame is detected as combed or not.

It seems obvious if no combing is detected then no de-interlacing takes place.
As an example (because this is what seemed to be happening when I was playing with Atlantis's sample)

If I used "Display=true" so TFM would show me where it was detecting combing, it might show it's detecting it in a small section at the bottom right of a frame, but I could see combing elsewhere.
If I used cthresh=-1 then everything is detected as combed.
What I'm trying to understand is whether one of the above scenarios could cause the frame to be de-interlaced differently, assuming the same pp value is used each time, or are combing detection and motion adaptive de-interlacing independent functions?

I'm assuming for your example above, the adaptive deinterlacing was effectively "getting it wrong" and it's not because TFM was telling the deinterlacer only to de-interlace the parts of the frame where it detected combing?

When playing with Atlantis's sample, it seemed "pp=4,clip2=deint" and "pp=6,clip2=deint" both fixed all the combing in a frame (assuming combing was detected) and I couldn't really see a difference between them, but encoding the sample resulted in slightly different file sizes, so I realised the video was being de-interlaced differently. I just wasn't sure when it came to fiddling with MI and cthresh values. Obviously they effect the number of frames which are de-interlaced, but I wasn't sure whether they'd also effect how those frames were deinterlaced.

[hello_hello]

Another question I also had, and I'll use the second pic from post #85 as an example.....

De-interlacing that way had a noticeable (negative) effect. The frame in question had no combing so required no de-interlacing, but what would happen if it did, while many frames preceding it and following it had no combing? If I did it "correctly" and configured TFM in such a way that it was the only deinterlaced frame, when playing the video normally that frame would probably stand out.

Obviously you wouldn't choose to deinterlace that way, but it did make me wonder if sometimes de-interlacing every frame might produce a more "consistent" result than you'd get from only de-interlacing combed frames. I guess it might come down the number of frames which need de-interlacing and the effect of doing so, but it did occur to me that maybe there's a point where deinterlacing every frame might look better than only deinterlacing the frames which "require it"?

[Atlantis]

I am at the end. Was doing season 11 and surprisingly my same script that worked for 10 seasons, gave me errors and couldn't open it.

I did a closer look and saw that my d2v files are marked as Frame_Rate=23976, 98.67% FILM.

So I went through frame by frame and saw that it is a very clean transfer and needs no processing except the opening titles.

So I did the first 10 seasons at 29.97fps with tfm/QTGMC and vinverse. I did the last season at 23.976 fps just with vinverse for the opening titles.

Previously I spent several weeks to do a season. It's impossible to go frame by frame or use overfiles. I was so happy with my script and things I learned here that I went back and deleted all the previous work I had done (maybe months of work). I started from scratch and I did all the 11 seasons in only one week.

[Atlantis]

Could you please explain to me what's this idea of DGIndex saying 98.67% FILM for example?

Does it do an internal tfm to check if the source is film? Why only my season 11 is recognized as 98.67% FILM and not the previous 10 seasons?

[hello_hello]

Keeping in mind I live in PAL-land so could be wrong.....
NTSC "film" can have either hard or soft pulldown. Soft pulldown means the video is 23.976fps, but it contains pulldown flags so the player adds the pulldown on playback and the output is 29.970fps. If that's the case, by ignoring the pulldown flags, you get the original progressive 23.976fps "film".

I'd guess your season 11 DVDs are authored differently. They use "soft pulldown", whereas the rest had it encoded. DGIndex knows 98.67% of the video uses soft pulldown. The other 1.33% is probably the opening/closing credits. It could either use hard pulldown or it's actually 29.970fps interlaced. I think for your DVDs it's probably the latter.

MeGUI has an option to use "force film". It's enabled by default and set to 95%. If 95% or more of the video is considered "film" (soft pulldown), DGIndex will treat it all as film and output 23.976fps. Parts which are actually interlaced mightn't look perfectly smooth etc. If less than 95% is "film", or with force film disabled, DGIndex will honour the puldown flags and output 29.970fps which then needs to be IVTC'd in the usual way. Maybe the way you've used the d2v option with TFM doesn't work for progressive video. Hence the errors. I'm not sure.

If you change the force film option in MeGUI's settings (I usually have it disabled myself) and it'd change the output (frame rate) as a result, you'll need to re-index the video to create a new index file.

One of the scripts I posted earlier didn't require the use of overrides files or playing with various TFM settings. Adding cthresh=-1 simply causes every frame to be de-interlaced after field matching. I think it looks a bit better than just plain de-interlacing and it de-interlaces the movement where there's combing better. There's not a huge difference but at least the output is 23.976fps.

[Atlantis]

I am using tfm with the output="output.txt" option. Something strange happens that I don't understand. If I do a 2 pass encode, (the same avs file) at the end I get an empty output file.

It's like at the second pass it creates an empty file but never executes the tfm command to fill the file.

Do you have any idea why this happens at pass 2? How the second pass works that its like it doesn't execute tfm?

[poisondeathray]

I am using tfm with the output="output.txt" option. Something strange happens that I don't understand. If I do a 2 pass encode, (the same avs file) at the end I get an empty output file.

It's like at the second pass it creates an empty file but never executes the tfm command to fill the file.

Do you have any idea why this happens at pass 2? How the second pass works that its like it doesn't execute tfm?

You have to close the avs script in whatever application you're using for pass 1 before the text file is written. If it stays open, nothing will be written

The 2nd pass needs to specify the input option, not output option. (e.g. input="output.txt" option, not output="output.txt")

[Atlantis]

I am not sure what you are saying. Yes I know about input option. I am using MeGUI 2 pass.

I don't want to use input in the second pass because then I have to have 2 avs files which makes it more complicated. I just want an output file at the end to check for combed frames.

Each time tfm runs, it replaces the old output file.

If you mean the same pass 1 output file is still open in pass 2, it should still contain all the info from pass 1 and not be empty I think.

[poisondeathray]

ok I see, just to look at the output.txt file, not to use

It's probably a megui bug , because it works OK with CLI

What is supposed to happen if you use the same avs, is output.txt will be written in the 1st pass, and overwritten in the 2nd pass (and it does contain the data when using CLI)

[StainlessS]

MeGUI two pass has nothing to do with it, thats for MeGUI to set bitrate on 2nd pass after scanning to see how complex each frame is on 1st pass.

You need to create the output.txt by eg loading script into VirtualDub and playing all the way through [an AssumeFPS(250.0)] at end of script will speed it up).

You then close VirtualDub and the output.txt file should exist.

Once the output.txt file exists, then you can use a 2nd script using input="output.txt" (and remove output="output.txt)
as and arg, and give that to MeGUI.

So you need two scripts with slightly different args [and without the AssumeFPS(250.0) in 2nd script].

Re-read what PoisonDeathRay wrote.

If you only want to see the output.txt file, then omit the MeGUI step altogether.

[poisondeathray]

I think the purpose of using megui is he wants to "kill 2 birds with 1 stone" . He's not using the TFM metrics as input, just wants to look at the matches after the fact, after encoding (I don't know why you would do that... because if something is wrong you would have to re-encode it seems like a waste of time.... but anyways....)

Another way you can just look at the output.txt quickly is file=>run video analysis pass in vdub. Or even avsmeter. I think Megui has this too with the "queue analysis pass" button

[Atlantis]

Sorry StainlessS, what you wrote didn't help at all. It seems you didn't understand. Please re-read. I already know about what you said.

I don't want to open the avs file in VirtualDub, that is one extra step.

I want to do my encoding and have the output file at the end too.

When I do my encoding in Megui, at the end of Pass 1 I do get the proper output file filled with the info. When pass 2 starts, the old output file is replaced with an empty one and never filled. It is this behavior the question and nothing else. You can try it yourself.

[Atlantis]

(I don't know why you would do that... because if something is wrong you would have to re-encode it seems like a waste of time.... but anyways....)
Yes I understand. I just want to understand why I get an empty file at pass 2. It's to better understand how avisynth/tfm/megui works.

[StainlessS]

Thanks PDR, I always forget about VDUB queue video analysis, I usually use AssumeFPS(250.0) as I want to see any
metrics on last frame (assuming script displays metrics), I guess the queue option is faster than Run Video Analysis Pass
as it does not even display video.
Thanks again.

[StainlessS]

Sorry for any misunderstanding Atlantis.

Did some exploring, Queued 2 Pass, with postponed muxing stage

Using only this script


RT_DebugF("Opening AVS File @ %s",RT_LocalTimeString(false))
VideoFileName ="D:\TFM\NTSC_Top_FILM.d2v"
MPEG2Source(VideoFileName).AssumeTFF
tfm(d2v=VideoFileName,output="output.txt")
Return Last


Results from RT_DebugF in DebugView Window

START OF PASS 1

00000001 0.00000000 [1048] AvsTimer 0.7
00000002 0.90207607 [1048] RemoveDirt 0.4
00000003 0.90755969 [1048] RemoveDirt 0.9
00000004 1.38205838 [1048] SSETools 0.1
00000005 2.14236116 [1048] RT_DebugF: Opening AVS File @ 2014-02-21 00:15:34.937
00000006 2.51615357 [1048] AvsTimer 0.7
00000007 3.41980290 [1048] RemoveDirt 0.4
00000008 3.42528105 [1048] RemoveDirt 0.9
00000009 3.89493632 [1048] SSETools 0.1
00000010 4.64975119 [1048] RT_DebugF: Opening AVS File @ 2014-02-21 00:15:37.453
00000011 5.03148890 [1048] AvsTimer 0.7
00000012 5.93346691 [1048] RemoveDirt 0.4
00000013 5.93898058 [1048] RemoveDirt 0.9
00000014 6.41274118 [1048] SSETools 0.1
00000015 7.16697073 [1048] RT_DebugF: Opening AVS File @ 2014-02-21 00:15:39.968
00000016 15.05930328 [1048] AvsTimer 0.7
00000017 15.96577930 [1048] RemoveDirt 0.4
00000018 15.97129250 [1048] RemoveDirt 0.9
00000019 16.44216728 [1048] SSETools 0.1
00000020 17.19692039 [1048] RT_DebugF: Opening AVS File @ 2014-02-21 00:15:50.000
00000021 17.76827812 [4004] AvsTimer 0.7
00000022 18.67107010 [4004] RemoveDirt 0.4
00000023 18.67652130 [4004] RemoveDirt 0.9
00000024 19.15471268 [4004] SSETools 0.1
00000025 19.92819977 [4004] RT_DebugF: Opening AVS File @ 2014-02-21 00:15:52.734

HERE TEXT FILE IS POPULATED @ AVS Closure

START OF PASS 2

00000026 107.40196228 [1048] AvsTimer 0.7
00000027 108.30832672 [1048] RemoveDirt 0.4
00000028 108.31529999 [1048] RemoveDirt 0.9
00000029 108.78475952 [1048] SSETools 0.1
00000030 109.53949738 [1048] RT_DebugF: Opening AVS File @ 2014-02-21 00:17:22.343
00000031 110.07365417 [2212] AvsTimer 0.7
00000032 110.97916412 [2212] RemoveDirt 0.4
00000033 110.98462677 [2212] RemoveDirt 0.9
00000034 111.45916748 [2212] SSETools 0.1
00000035 112.23111725 [2212] RT_DebugF: Opening AVS File @ 2014-02-21 00:17:25.031

HERE TEXT FILE IS POPULATED @ AVS Closure

MAYBE Re-SCANNING to get some info for MeGUI log file ???

00000036 195.30229187 [1048] AvsTimer 0.7
00000037 196.21223450 [1048] RemoveDirt 0.4
00000038 196.21777344 [1048] RemoveDirt 0.9
00000039 196.69239807 [1048] SSETools 0.1
00000040 197.44729614 [1048] RT_DebugF: Opening AVS File @ 2014-02-21 00:18:50.250

HERE TEXT FILE IS RE-CREATED/INITIALIZED DUE TO REOPENING OF AVS BUT NOT POPLATED

TEXT FILE contains only
#TFM v1.0.4 by tritical
field = top
crc32 = dbef609c


As you can see, AvsTimer through to SSETools is where plugins are loaded from Plugins Dir.

It opens avs file 5 times at start of 1st pass,
twice at start of 2nd pass,
and again after 2nd pass is completed (where output.txt is again initialized, ie empty, after closure it contains only the header)

Maybe Zathor can shed some light.

EDIT: Problem stems from eg tfm initializing (zero bytes) at opening, whereas many such plugins only create/re-create and populate output file at the end when script closed (and quite often only created/populated if every frame was scanned).

EDIT: I have 200+ plugins in my plugins Dir, the shown plugs are the only ones that output debug text during filter constructor .

EDIT: Loading avs into VirtualDub then closing without playing creates something like this (before closure it is zero bytes in size)

#TFM v1.0.4 by tritical
field = top
crc32 = dbef609c
0 c - [10]

where only the first frame has an entry, so looks like MeGUI only opens the AVS without requesting a single frame.

[Atlantis]

Thanks for the explanation.

After having perfect encodes at 23.976 fps for my season 11, I went back and looked closer at season 10. It seems each season has different transfers.

Only season 11 gave me 97% film and season 10 gives me 100% video when indexed but with closer inspection I have found out that season 10 also has a very clean transfer. When I use tfm with output option, I don't find a single detected combed frame in it. So I tried tdecimate and I get a very clean smooth video at 23.976 fps with no frame blending or further processing needed (no vinverse needed). The smooth video includes all, the main program, the beginning titles which are the most problematic and the end titles. It's actually better than season 11. Season 11 was detected at 97% film and MeGUI marked them at 23.976 but the opening titles looked interlaced in parts.

So I am going back one by one the seasons and redoing them! Inspect them closely. That's where tfm output comes handy. If I don't see a combed frame in the output file I think it would be a clean transfer and I can decimate and no need for post processing. If I see lots of combed frames, I have to investigate further. I'm guessing if lots of combed frames, I have to stay with my old script and output at 29.97 fps. The first seasons are not clean transfers for sure.

[Atlantis]

I was testing QTGMC in progressive mode for replacing vinverse where vinverse made the picture worse because of edge enhancements.

Surprisingly QTGMC in progressive mode is not good at all. I'm wondering if other people also feel this way. QTGMC in progressive mode produces bad motion artifacts.

Any other filter to replace vinverse?

What should I do when there is edge enhancement present?