Deep Space Nine upscale project [Archive] - Page 7

hello_hello

20th May 2020, 04:05

Wrong. Thank you for once again confirming everything I've said about your lack of reading skills.

You can keep accusing me of a lack of reading skills but it only makes you look more childish.
You implied.... no actually stated... that QTGMC's lossless mode doesn't field match. Of course it was a criticism.

Of course, the so-called "lossless" mode is nowhere near lossless, since it still doesn't bother attempting to restore the original progressive frames where they exist or perform any kind of field-matching

There is no consensus. Thank you for once again confirming everything I've said about your lack of reading skills.

You flogged the "poor reading skills" line to death a long time ago.
It doesn't take any reading skills to know that in almost every thread where you ask a question, someone complains about your attitude.

You've not mentioned any of the screenshots or samples I uploaded. Why not discuss them instead of just bickering? Or at least discuss the topic in addition to bickering. Where's your encodes showing how your method does it better? I supplied links for the source files that others supplied, and you provided links to your own source samples. Show us how they look using your method and explain why it's better instead of just pontificating.

manono

20th May 2020, 04:14

Geez but I'm getting sick of this. I'll happily close this thread and any other where the male testosterone level gets too high. For the last time, keep it civil. Don't gimme any of the "But she started it" crap, either. Some of you people behave like children.

Katie Boundary

20th May 2020, 04:57

I specifically asked for a sample of this

And for an "interlaced content" sample . (Not text, not fades)

Retrograde field behavior has nothing to do with text or fades.

poisondeathray

20th May 2020, 05:07

Retrograde field behavior has nothing to do with text or fades.

I never said it did.

"And an interlaced content" sample was requested. It's a separate topic. I suggested someone post both earlier. People might see "combing" on text or fades and identify it as "interlaced", but I'm not interested in text or fades - that has already been established. I'm asking if there is actual interlaced content other than text or fades - that affects how you might approach VFR

Katie Boundary

20th May 2020, 05:15

OOh, gotcha. Well I already posted samples of retrograde field behavior a long time ago in the two other threads dedicated to that exact topic. I'm not sure what would be gained from posting them a third time, since the consensus hasn't changed: You have to either throw half the fields away or blend. Or manually correct the field order one field at a time with dozens of trim commands, if a sane field order even exists in the first place.

I'm pretty sure you already know what interlacing looks like, so I'm not sure why you'd need a sample of that either.

EDIT: if you mean that you specifically want samples of these things from DS9, then I'm the wrong person to ask, since I don't have DS9 at home.

poisondeathray

20th May 2020, 05:21

OOh, gotcha. Well I already posted samples of retrograde field behavior a long time ago in the two other threads dedicated to that exact topic. I'm not sure what would be gained from posting them a third time, since the consensus hasn't changed: You have to either throw half the fields away or blend.

But does it exist here in DS9? Did you confirm it ?

People use various terms all the time, but sometimes some people describe things incorrectly.

Or some people used ffms2 in this thread . I've seen that cause field misorder - is that the cause ? Or something else in the script ? Or some other program ?

I'm pretty sure you already know what interlacing looks like, so I'm not sure why you'd need a sample of that either.

Same as above . Did you verify? Any content other than text or fades ?

3:2 telecine can look like "interlacing" too in layman's terms on some frames . But it's not interlaced content.

Katie Boundary

20th May 2020, 05:25

My Star Trek DVD collection is limited to the first ten movies and Voyager. It's up to Joel to provide DS9 clips. But if h_h wants to lose a bet with me about his/her incorrect and easily disprovable claims about the threads that I start, then I promise that DS9 will be the first thing I buy with his/her money.

hello_hello

20th May 2020, 05:33

Katie Boundary

20th May 2020, 09:42

This thread stopped being just about DS9 a looooong time ago.

Swede

20th May 2020, 10:26

This thread stopped being just about DS9 a looooong time ago.

Yes it has. AND the second anyone posts anything BUT you will be stricken, hard, and the thread will be closed. Be careful now! All of you!

Katie Boundary

20th May 2020, 15:15

I think I understand what PDR means when he says "actual interlaced content other than text or fades". You mean 60 hz content where the WHOLE image is changing every 1/60th of a second, not where parts of it change every 1/30th of a second and other parts change every 1/24th of a second, right? I agree, finding out how much of that content exists in DS9 has huge ramifications, for reasons explained in post #257

EDIT: I am such an idiot. There's no need to specify field parity for TFM at all. You can just let line C use whatever field AVIsynth says is first, and let line D use complementparity():

Y=nnedi3(field=-2)

A=yadifmod2(mode=1,edeint=Y).selecteven()
B=yadifmod2(mode=1,edeint=Y).selectodd()
C=Tfm(clip2=A)
D=complementparity().Tfm(clip2=B)
Interleave(C,D)

(unnecessary parameters removed for readability)

Then it'll work on both TFF and BFF content without needing manual adjustment.

JoelHruska

21st May 2020, 03:26

Ok, so, a few things:

1). I messaged KB about what she calls "retrograde field behavior" because when I first attempted to run various TIVTC scripts against DS9, one of the problems all of these scripts introduced into various *specific* video sequences was frames moving backwards. This problem is actually what pushed me away from TIVTC and towards QTGMC in the first place. The reason I messaged her privately, ironically, is precisely because I was trying to avoid drama here. I followed KB's instructions for looking for this behavior. I did not see it.

I am not here to argue with anyone. I am not here to take sides or cause fights.

2). I am not going to keep anything in the dark. The entire stated purpose of my project is to create a tutorial.

3). H_H, I saw your response to me and I'll respond back to you tomorrow. Had a major work project for the last couple days.

I appreciate people being helpful with this project. I know I have been slow to test some of these workflow ideas -- I've been hit with other projects and I'm preparing to move -- but I intend to spend some time with it this weekend.

JoelHruska

21st May 2020, 04:00

One update to the above. I have done what some of you have recommended: Specifically, I analyzed the file with DGIndex, created a D2V file, and then used StaxRip to rip the VOB to an MKV. The resulting file is in 29.97 fps, not 23.976 fps -- but this injects 3:2 pulldown where none previously existed.

Do I really want to convert the episode into 29.97 fps?

manono

21st May 2020, 04:52

...and then used StaxRip to rip the VOB to an MKV
Bad idea. After decrypting the DVD to the hard drive, you make the D2V file for the episode and then use MPEG2Source on it. That's thoroughly explained in the docs accompanying the DGMPGDec package. Forget repackaging to an MKV. I don't know whose toes I stepped on with that statement.

The resulting file is in 29.97 fps, not 23.976 fps -- but this injects 3:2 pulldown where none previously existed.

All NTSC DVDs output 29.97fps. If it's really progressive 23.976fps, you IVTC it back to 23.976fps. So, the latter part of your statement is incorrect. The only time you should get 23.976fps out of DGIndex is by setting Field Operation to Force Film and you do that only if it's 100% film. Some say 95% film is good enough, but not I. You can open the D2V and at the bottom it'll tell you the percentage of film.

Stereodude

21st May 2020, 05:06

Do I really want to convert the episode into 29.97 fps?
It already is i60 (when the soft pulldown flag is honored). You don't want an input file that changes framerates. That's how you have audio sync issues. Because you have a mix of i60 encoded content and p24 content with a soft pulldown you need to apply the soft pulldown and work from there. The flags=5 argument in TFM will undo the soft pulldown perfectly without relying on visually matching fields.

Katie Boundary

21st May 2020, 07:16

1). I messaged KB about what she calls "retrograde field behavior" because when I first attempted to run various TIVTC scripts against DS9, one of the problems all of these scripts introduced into various *specific* video sequences was frames moving backwards. This problem is actually what pushed me away from TIVTC and towards QTGMC in the first place. The reason I messaged her privately, ironically, is precisely because I was trying to avoid drama here. I followed KB's instructions for looking for this behavior. I did not see it.

See, this is why I kept hounding everyone here about figuring out what was going on with the field order. If using "honor pulldown flags" in DGindex and then bob() in AVIsynth, and doing absolutely nothing else, shows absolutely no stuttering/RFB, but some other workflow does, then the other workflow has a huge screw-up somewhere. So we need to know: are you blindly duplicating frames to get your frame rate up to 48 fps before applying any kind of deinterlacing? What exactly are you doing that produced this effect?

BTW, the term "retrograde field behavior" isn't mine. I got it from John Meyer.

Do I really want to convert the episode into 29.97 fps?

No, you want to convert it to 59.94 :)

manono

21st May 2020, 07:30

Because you have a mix of i60 encoded content and p24 content with a soft pulldown you need to apply the soft pulldown and work from there.
Oh, there's soft pulldown in these? If there have been samples posted, I haven't looked at them and don't intend to go back through this thread to find them. Then, yes, avoid field-matching if possible by using the D2V argument to read the soft pulldown from the D2V file. It'll go something like this:

TFM(D2V="Video.d2v",Flags=5)

I've always used the default 4, but if you prefer 5 for these...

hello_hello

21st May 2020, 10:30

The section poisondeathray linked to looks interlaced. I originally thought it was, but it's progressive with the fields out of alignment (if that's the correct description), however the overlaid shooting is interlaced, either because it was overlaid with the fields out of phase with the CGI, or it is interlaced, or telecined, or something.
https://forum.videohelp.com/threads/396652-Deinterlacing-Deep-Space-Nine-NTSC#post2578796

Katie's method without YadifMod, given TFM's taking pixels from YadifMod, which is in turn taking them from nnedi3 anyway, as best as I can tell. I also disabled micmatching to fix the jitter towards the end.
Y=nnedi3(field=-2)
A=Y.selecteven()
B=Y.selectodd()
C=Tfm(clip2=A,micmatching=0)
D=complementparity().Tfm(clip2=B,micmatching=0)
Interleave(C,D)
space battle Katie.mkv (https://ufile.io/a930oo8r)

Katie's method unmodified
Y=nnedi3(field=-2)
A=yadifmod2(mode=1,edeint=Y).selecteven()
B=yadifmod2(mode=1,edeint=Y).selectodd()
C=Tfm(clip2=A)
D=complementparity().Tfm(clip2=B)
Interleave(C,D)
space battle katie with yadif.mkv (https://ufile.io/fmukhe2i)

VFR
# TFM(Output="D:\TFM.txt", PP=5, micmatching=0)
# TDecimate(Mode=4, Hybrid=2, Output="D:\TDecimate.txt")
TFM(Input="D:\TFM.txt", PP=5, micmatching=0)
TDecimate(Mode=5, Hybrid=2, Input="D:\TDecimate.txt", tfmIn="D:\TFM.txt", mkvout="D:\Timecodes.txt")
space battle VFR.mkv (https://ufile.io/uyrm10ih)

VFR with a QTGMC de-interlaced clip
DeintClip=QTGMC().SelectEven()
# TFM(Output="D:\TFM.txt", PP=5, Clip2=DeintClip, micmatching=0)
# TDecimate(Mode=4, Hybrid=2, Output="D:\TDecimate.txt")
TFM(Input="D:\TFM.txt", PP=5, Clip2=DeintClip, micmatching=0)
TDecimate(Mode=5, Hybrid=2, Input="D:\TDecimate.txt", tfmIn="D:\TFM.txt", mkvout="D:\Timecodes.txt")
space battle VFR QTGMC.mkv (https://ufile.io/z0x8spkh)

VFR followed by QTGMC in progressive mode
# TFM(Output="D:\TFM.txt", PP=5, micmatching=0)
# TDecimate(Mode=4, Hybrid=2, Output="D:\TDecimate.txt")
TFM(Input="D:\TFM.txt", PP=5, micmatching=0)
TDecimate(Mode=5, Hybrid=2, Input="D:\TDecimate.txt", tfmIn="D:\TFM.txt", mkvout="D:\Timecodes.txt")
QTGMC(InputType=1, Preset="Very Slow")
space battle VFR QTGMC P.mkv (https://ufile.io/l2pkeu4j)

I followed each encode with some cropping and resizing, because I felt like looking at the correct aspect ratio, plus it'll be resized at some stage anyway.
CropResize(0,0, 6,2,-4,-2, InDAR=15.0/11.0, ResizeWO=true, Resizer="Resize8")

I suspect, but haven't seen a sample of it, the "retrograde field" problem could be fixed by using TFM in a somewhat standard manner with micmatching disabled. It might just be picking the wrong match on occasion.
If there are purely interlaced sections, and the sample above may suggest otherwise, I guess Katie's method should bob them to 59.94fps, but I'm not sure that's always a good idea because it might make them stand out even more from the film sections, and more of a pain than it's worth if they're few and far between, but either way, have we learned how Katie goes about decimating all those duplicate frames after outputting everything at 59.94fps? I'm sure she's mentioned doing it, but not how she does it. It'd be tedious to do manually, especially when it comes to creating matching timecodes for a VFR.

For the VFR encodes, It's hard to avoid seeing "quality pumping" for the frames where the de-interlacing kicks in to fix the combing in the "shooting". In that respect, I think the VFR encode with QTGMC de-interlacing looks the best, but it does cause some aliasing (not as much as nnedi3 de-interlacing on it's own though). The other problem is you have to use the de-interlaced clip for the analysis pass, and if it's a QTGMC clip that'll make the analysis pass much slower.

nnedi3 deinterlaced
https://i.postimg.cc/PNHbf5Px/nnedi3-deinterlaced.png (https://postimg.cc/PNHbf5Px)

qtgmc deinterlaced
https://i.postimg.cc/5678m5pY/qtgmc-deinterlaced.png (https://postimg.cc/5678m5pY)

tfm deinterlaced, pp=5
https://i.postimg.cc/cgyRnshF/tfm-deinterlaced-pp-5.png (https://postimg.cc/cgyRnshF)

Stereodude

21st May 2020, 15:30

Oh, there's soft pulldown in these? If there have been samples posted, I haven't looked at them and don't intend to go back through this thread to find them. Then, yes, avoid field-matching if possible by using the D2V argument to read the soft pulldown from the D2V file. It'll go something like this:

TFM(D2V="Video.d2v",Flags=5)

I've always used the default 4, but if you prefer 5 for these...
A value of 4 for flags doesn't address the primary issue I saw in the few episodes I tested. The problem is that TFM sees combing where there is none and deinterlaces large areas of the frames despite it being progressive with no real combing. 5 puts an end to that. I didn't seen any examples of combing (except in the occasional scene change) which 5 still checks and addresses.

Katie Boundary

21st May 2020, 17:37

*facepalm*

h_h, if you're going to use something that looks superficially similar to my approach, at least use something that was actually designed to be used, not something that was posted only for demonstrative purposes. It is insulting and borderline slanderous to refer to a script that I would never use or approve of as "Katie's method".

And in case you're wondering, Yadifmod2 provides temporal smoothing, which NNEDI3 does not. The whole idea behind yadifmod2 is to add temporal interpolation to bobbers that don't have it. The effect of this is to eliminate flicker along object edges during interlaced fades.

hello_hello

21st May 2020, 19:26

h_h, if you're going to use something that looks superficially similar to my approach, at least use something that was actually designed to be used, not something that was posted only for demonstrative purposes. It is insulting and borderline slanderous to refer to a script that I would never use or approve of as "Katie's method".

I didn't use something superficially similar. There's a sample encode using the exact method you posted. I thought you'd be happy I showed it works. I was going to add micmatching=0 to it, to fix a glitch, but then of course I'd be accused of changing it and still calling it your method.

Why not post an encode from a script that was actually deigned to be used instead of pushing some generic idea? We've heard the talk. What would you change that'd make it look all that different anyway? Was TFM taking too much from the nnedi3 clip? To little? It was hardly an unfair comparison, as I didn't use different TFM settings for the VFR samples, aside from micmatching=0. Tell me what to change for your method and I'll try it for a VFR encode too.

I've tried setting cthresh and mthresh to 2 as you've suggested previously, and linked to samples of it in previous posts.

I asked the question again, but you've chosen to ignore it again. You've referred to decimating a 59.94fps output for a VFR. How do you go about it?

And in case you're wondering, Yadifmod2 provides temporal smoothing, which NNEDI3 does not. The whole idea behind yadifmod2 is to add temporal interpolation to bobbers that don't have it. The effect of this is to eliminate flicker along object edges during interlaced fades.

Yep, I've wondered about that numerous times without a response. Now you've finally supplied one, so I've looked, and I can see a picture de-interlaced via YadifMod slightly smooths the aliasing caused by taking pixels from a nnedi3 clip. I hadn't noticed the difference before because there's still aliasing and I wasn't aware of a reason to look for it.
If I supply TFM with a de-interlaced clip that needs temporal smoothing, I'll probably do it that way from now on. I might even try it with a QTGMC de-interlaced clip to see how much difference it makes.

hello_hello

21st May 2020, 20:11

NNEDI3 retains more detail than QTGMC when TFM is de-interlacing where it shouldn't, but you pay for it with aliasing.

Interestingly, with QTGMC de-interlacing, Yadif seems like it might help retain detail when TFM is de-interlacing where it shouldn't, but you pay for it with aliasing.

I used YadifMod2 out of habit, but the result should be the same.

DClip1 = NNEDI3(field=-2).SelectEven() # or QTGMC()
DClip2 = NNEDI3(field=-2) # or QTGMC()
DClip3 = yadifmod2(mode=1,edeint=DClip2).SelectEven()
TFM(clip2=DClip1) # or TFM(clip2=DClip3)

http://www.framecompare.com/screenshotcomparison/77D67NNX

http://www.framecompare.com/screenshotcomparison/99JMFNNU

lansing

21st May 2020, 22:22

Use http://www.framecompare.com when doing comparison, it's a lot easier to show the differences

hello_hello

21st May 2020, 22:51

I wasn't aware of that site. I'll try it next time.

Edit: I tried it last time.

JoelHruska

22nd May 2020, 02:34

Bad idea. After decrypting the DVD to the hard drive, you make the D2V file for the episode and then use MPEG2Source on it. That's thoroughly explained in the docs accompanying the DGMPGDec package. Forget repackaging to an MKV. I don't know whose toes I stepped on with that statement.

I don't think I've been clear.

If you want to use MPEG2Source in StaxRip, you load a D2V file that you've created in DGIndex... which is what I did. I use StaxRip as a front-end for running AviSynth. I used MKV for output because StaxRip automatically selects it. I may not have a particular sense of which file type to output in, but that also means I couldn't care less (except, perhaps, for retaining workflow compatibility) whether output is in MKV or not. So what ought it be in?

All NTSC DVDs output 29.97fps. If it's really progressive 23.976fps, you IVTC it back to 23.976fps. So, the latter part of your statement is incorrect. The only time you should get 23.976fps out of DGIndex is by setting Field Operation to Force Film and you do that only if it's 100% film. Some say 95% film is good enough, but not I. You can open the D2V and at the bottom it'll tell you the percentage of film.

Alright. Sacrifice of Angels is 86% film. That would increase if you excised the credits. It would not excise enough to bring the number above 95%, I don't think.

Alright. So, at this point in the process, I have a D2V file for Sacrifice of Angels. What I do not have is an actual video file I can get a frame count on. Attempting to load the VOB files into Vdub2 produces very strange results with only a few seconds or minutes of footage available from a 1GB file. There's no way to load a D2V into Vdub2. What app do you use for the frame-by-frame checking to see where TIVTC needs to be applied?

manono

22nd May 2020, 03:01

What app do you use for the frame-by-frame checking to see where TIVTC needs to be applied?
MPEG2Source in an AviSynth script file (an .avs). I then open it in Virtual Dub (or you can use VDub2). Again, read the docs included with DGMPGDec. Although it has a good reputation, you don't need StaxRip to do what you're asking. Then you use TIVTC as explained by Stereodude and me earlier.

I don't think I've been clear.
You provided an MKV. Inside that MKV was an M2V. But you can't create a good D2V file when it's inside an MKV. Anyone intending to help and intending to use MPEG2Source has to first extract that M2V from the MKV. It wastes time.

If 86% film, then that 86% is treated as if you used Force Film in DGIndex (giving you a progressive 23.976 video for those portions) and the rest undergoes a full IVTC (field-matching and decimation) with the whole thing winding up as progressive 23.976fps.

...frame-by-frame checking to see where TIVTC needs to be applied? You don't need to do any checking. With TIVTC set up properly, it can do the job all by itself. If there's any real interlaced video or any progressive 29.97fps portions, that'll throw a monkey wrench into the whole procedure.

JoelHruska

22nd May 2020, 03:06

Also, I've just noticed something really important.

My QTGMC repair script creates much worse-looking output if run against a VOB than if run against MakeMKV. Proof:

Look at the left-hand side of the Jem'Hadar fighter on the far left. The left-hand side of this image is 29.97 VOB with my QTGMC repair script applied. The right-hand side image is 23.976 fps output created with MakeMKV and processed with the same QTGMC script.

I did all of my tuning on that repair script using MakeMKV, and you can tell.

First frame comparison:

https://i.imgur.com/ndXgnKq.png

Second frame comparison:

https://i.imgur.com/E7v5bfT.png

The left-hand image is duller and markedly less sharp.

Here, the difference is downright egregious:

https://i.imgur.com/y1KW4so.png

Note: In a stunning display of competence on my behalf, the left-hand file names above are incorrect. If you've used Staxrip, you know you can change your script settings and just hit "Encode" again without specifically changing the output file name. I have sometimes forgotten to actually change out the file names. In this case, I know what the output actually is, because I ran the encode on the left today and have documented the settings on the right. The left-hand images are not unprocessed, despite the fact that they say they are.

This is after running the exact same QTGMC script against both videos.

I now understand why H_H and some others have cautioned against the use of QTGMC. When my repair script is used against a VOB-ripped file at 29.97 fps, the end result is rather awful looking. The MakeMKV output processed at 23.976 fps is *substantially* better than the VOB.

All of the CGI scenes in the episode look substantially worse when processed with QTGMC in 29.97 fps than when processed by QTGMC in 23.976 fps.

manono

22nd May 2020, 03:13

My QTGMC repair script creates much worse-looking output if run against a VOB than if run against MakeMKV.
Nonsense.

Apples to oranges, don't you think? Maybe you should IVTC the episode down to 23.976fps first? And using the procedure I described? I didn't bother looking at the pics.

JoelHruska

22nd May 2020, 03:22

MPEG2Source in an AviSynth script file (an .avs). I then open it in Virtual Dub (or you can use VDub2). Again, read the docs included with DGMPGDec. Although it has a good reputation, you don't need StaxRip to do what you're asking. Then you use TIVTC as explained by Stereodude and me earlier.

Alright. I'll specifically go digging around there.

You provided an MKV. Inside that MKV was an M2V. But you can't create a good D2V file when it's inside an MKV. Anyone intending to help and intending to use MPEG2Source has to first extract that M2V from the MKV. It wastes time.

Nope. Definitely have not been clear.

Let me break down exactly what I'm doing:

1). Rip the DVD in DVD Decrypter. This creates the following directory and populates it with VOB files: C:\DS9S6D2\VIDEO_TS
2). Load the VOB file into DGIndex. Confirm that DGIndex is set to honor pulldown flags.
3). Run the project through DGIndex. This creates a D2V file as output.
4). Load the D2V file into StaxRip. This opens the project file using MPEG2Source. Screenshot:

https://i.imgur.com/xD7Enho.png

The output of this process is the MKV file.

JoelHruska

22nd May 2020, 03:25

Nonsense.

Apples to oranges, don't you think? Maybe you should IVTC the episode down to 23.976fps first? And using the procedure I described? I didn't bother looking at the pics.

If you didn't bother looking at the images, on what basis are you declaring this nonsense? I've been fine-tuning that QTGMC output on 23.976 fps source for three months. I promise you, I'm qualified to determine when it looks different. The reason I'm bothering to post about it is because H_H was comparing my output earlier when I first started posting, and he provided feedback based on scripts he ran. But if he ran my QTGMC script against 29.97 content then he didn't see the output I saw when I opted to use the script in the first place.

Given that the entire point of being here is to get advice on the best practices, methods, and comparative results of different encode methods, it would seem important that we all be on the same page when attempting to evaluate different material.

Since you declare my post "Nonsense" followed by an acknowledgement that 23.976 and 29.97 are two different frame rates and thereby likely to produce two different results, I have no idea how to read your comment. It may not be relevant to you that the output you see off my script is substantially different than what I see, but having spent months experimenting with QTGMC to arrive at the results I had arrived at, I assure you, it's relevant as heck to me.

poisondeathray

22nd May 2020, 04:06

I now understand why H_H and some others have cautioned against the use of QTGMC. When my repair script is used against a VOB-ripped file at 29.97 fps, the end result is rather awful looking. The MakeMKV output processed at 23.976 fps is *substantially* better than the VOB.

This shows you don't understand or didn't read. By your own admission you said you are new to this.... so pay attention

When you IVTC the VOB, and then apply your repair script - you will get same results if that was a 23.976p section as makemkv at 23.976p and then applying the repair script. Try it and prove it to yourself

Again, you might want to use that repair script along with QTGMC in progressive mode on those 5% problem sections. But on the 95% + sections, no way, you kill too much detail

JoelHruska

22nd May 2020, 04:13

Manono,

I have created an AVSI file per your instructions, using the DGDecode manual for syntax advice. The AVSI is very simple:

LoadPlugin("C:\Path\DGDecode.dll")
MPEG2Source("C:\Path\VTS_2_1.d2v")

I load the AVSI into Vdub2. Vdub2 displays the following screen:

https://i.imgur.com/TzTQGPQ.png

When I loaded the script the very first time, the default frame rate was 15 fps. I adjusted it to 29.97 fps and it has stayed that way thereafter. However, upon clicking "Ok," nothing further happens. No video loads. Vdub2 displays the following:

https://i.imgur.com/5OlHn7H.png

I have tested VirtualDub2_44282 and _44237. I have experimented with changing the various options in the Input Format, as well as frame order. Nothing has mattered. I experimented with adding "info=1" to see if this would display video information when the load fails in Vdub2. It did not. I created multiple D2V files using DGIndex in case one of them had been corrupted. Neither file worked.

Any idea what the problem is?

JoelHruska

22nd May 2020, 04:17

This shows you don't understand or didn't read.

The former. I have no interest in wasting time, either mine or yours. I have spent hours reading the posts in this specific thread and continue to return to the earlier posts to see the entire flow of conversations. I knew that you were advocating working in VOB -- I saw and understood that. What I did not understand was that my QTGMC script would produce different output on a 29.97 VOB compared to a 23.976 MKV. In my own defense, I *thought* I had checked this possibility. I have seen my script run against 24.66 fps and 23.976 fps, as well as 119.88 fps. I've seen it run against a CFR file from Handbrake as well as a VFR file. I would not have been surprised to see it perform slightly differently, but the output on 29.97 is far worse than anything I saw at 23.976.

Clearly I did not understand and I apologize for that.

poisondeathray

22nd May 2020, 04:19

The file extension should be ".avs" only , not ".avsi" - the latter are auto importing avisynth scripts

In the vdub2 open file dialog box, it should say AVIFile/AVISynth input driver, not raw video input driver

manono

22nd May 2020, 04:20

Nope. Definitely have not been clear.

Damn, I've been confusing 2 separate threads. A different fellow uploaded a sample from a VOB packaged as an MKV. I haven't even seen any samples from you. I apologize.

Now, pay attention to what pdr wrote. Some (hello_hello, among others) sometimes use QTGMC specifically for its cleaning properties. That's fine. It seems to me, though, that if you want to do some 'cleaning' (spatial and/or temporal denoising) you use filters designed specifically for that. As pdr mentions, that cleaning QTGMC does by default also removes a little bit of detail. And, since you're upscaling to Hi-Def, you want all the initial detail you can get.

Edit: Now I see you've seen the light. :)

JoelHruska

22nd May 2020, 04:43

PDR,

Thank you. This worked -- once I found a DLL that can supposedly function to decode MPEG2Source with a 64-bit DLL:

http://avisynth.nl/index.php/MPEG2DecPlus

Can I rely on MPEG2DecPlus64.dll to provide equivalent MPEG2 decoding to DGDecode? I would prefer to use an x64 DLL if at all possible. If I need to specifically drop back to 32-bit for everything to make DGDecode work and there's no functional x64 equivalent, I can bring up a new system tomorrow.

poisondeathray

22nd May 2020, 04:53

Can I rely on MPEG2DecPlus64.dll to provide equivalent MPEG2 decoding to DGDecode?

Yes, that is the correct version and works fine

It's the other x64 version by joshyd (DGDecode_3-19-2010) that has problems

JoelHruska

22nd May 2020, 05:00

I avoided that one on the basis of age. The current 32-bit DGDecode.dll is from 2011. Seemed better to look for a later source than the earlier DLL based on earlier code.

Katie Boundary

22nd May 2020, 06:01

I will be running DS9 tests soon, but for now, I'm still using Andromeda as my guinea pig for various deinterlacing methods. Just to be ABSOLUTELY sure about the relative merits of NNEDI3+Yadifmod2 versus QTGMC, I put the following script together:

mpeg2source("212.d2v")

A=qtgmc(sourcematch=3,lossless=1).selecteven()
B=qtgmc(sourcematch=3,lossless=1).selectodd()
C=Tfm(field=1,mode=0,slow=2,mchroma=false,cthresh=1,MI=40,blockx=8,mthresh=2,clip2=A,micmatching=0)
D=Tfm(field=0,mode=0,slow=2,mchroma=false,cthresh=1,MI=40,blockx=8,mthresh=2,clip2=B,micmatching=0)

E=Interleave(C,D)

Y=nnedi3(field=-2,nsize=3,nns=4,qual=2,etype=1)

L=yadifmod2(mode=3,edeint=Y).selecteven()
M=yadifmod2(mode=3,edeint=Y).selectodd()
N=Tfm(field=1,mode=0,slow=2,mchroma=false,cthresh=1,MI=40,blockx=8,mthresh=2,clip2=L,micmatching=0)
O=Tfm(field=0,mode=0,slow=2,mchroma=false,cthresh=1,MI=40,blockx=8,mthresh=2,clip2=M,micmatching=0)
P=Interleave(N,O)

stackvertical(E,P)

(I kept the manually specified field arguments instead of using the complementparity approach because I know the field parity already)

This was the result:

https://i.imgur.com/OA212K4.png

Remember, that's "lossless" QTGMC at the top, and Yadifmod2+NNEDI3 at the bottom. HuRr DuRr MuH aLiAsInG.

I didn't use something superficially similar. There's a sample encode using the exact method you posted.

No, you posted encodes using a lobotomized script that I provided for the purpose of demonstrating where Complementparity() would go. HUGE difference.

I thought you'd be happy I showed it works. I was going to add micmatching=0 to it, to fix a glitch, but then of course I'd be accused of changing it and still calling it your method.

All of my ACTUAL methods use micmatching=0 and about a dozen other settings that I cut out of that particular bit of code because, as I stated, the complementparity demonstration code was NOT meant for actual use.

So what ought it be in?

For compatibility: AVI
For variable frame rate: MP4

MP4 is an official industry standard recognized by the International Organization for Standardization as ISO/IEC 14496-14:2003, whereas MKV is developed and supported by a couple of hackers in France in their spare time, so MP4 should offer slightly better compatibility than MKV. However, this is purely speculation, and I don't know of any software that works with MP4 but not MKV.

What app do you use for the frame-by-frame checking to see where TIVTC needs to be applied?

Just open an AVIsynth script in Virtualdub?

Not Virtualdubmod or Virtualdub2 or TheVirtualdubStrikesBack or MyOwnVirtualDubWithBlackjackAndHookers...

If 86% film, then that 86% is treated as if you used Force Film in DGIndex (giving you a progressive 23.976 video for those portions) and the rest undergoes a full IVTC (field-matching and decimation) with the whole thing winding up as progressive 23.976fps... If there's any real interlaced video or any progressive 29.97fps portions, that'll throw a monkey wrench into the whole procedure.

Ummm... are you assuming that the episode was encoded at 86% soft pulldown and 14% hard pulldown, just for shits and giggles? Because the DVD authors wanted to screw with us? I mean, that's not a totally insane assumption, and I've seen it before in Farscape and a few episodes of Captain Planet. It just happens to not be true in this particular case. The "real interlaced video or any progressive 29.97fps portions" make up almost the entirety of the other 14%.

Attempting to convert the whole episode to 23.976 is a very bad idea.

Let me break down exactly what I'm doing:

1). Rip the DVD in DVD Decrypter. This creates the following directory and populates it with VOB files: C:\DS9S6D2\VIDEO_TS
2). Load the VOB file into DGIndex. Confirm that DGIndex is set to honor pulldown flags.
3). Run the project through DGIndex. This creates a D2V file as output.
4). Load the D2V file into StaxRip. This opens the project file using MPEG2Source.

All right, cool. This is a pretty sane and simple workflow that we can replicate for the purpose of diagnosing any errors you encounter.

LoadPlugin("C:\Path\DGDecode.dll")
MPEG2Source("C:\Path\VTS_2_1.d2v")

Why are you manually loading an autoload plugin?

Can I rely on MPEG2DecPlus64.dll to provide equivalent MPEG2 decoding to DGDecode? I would prefer to use an x64 DLL if at all possible. If I need to specifically drop back to 32-bit for everything to make DGDecode work and there's no functional x64 equivalent, I can bring up a new system tomorrow.

Why are we now screwing around with 64-bit anything?

manono

22nd May 2020, 06:38

Ummm... are you assuming that the episode was encoded at 86% soft pulldown and 14% hard pulldown, just for shits and giggles?
As I've not seen so much as a sample, maybe it's just wishful thinking. However, I have seen combinations of hard and soft pulldown like that much more often than I've seen soft pulldown mixed with video at similar percentages.

Attempting to convert the whole episode to 23.976 is a very bad idea.
Then you make the case for bobbing it and calling it a day. That would be a helluva lot easier, that's for sure. Either that or doing a VFR encode, something for which I, personally, have no use.

Why are you manually loading an autoload plugin?
I do that myself. For one thing, it keeps a screwed up plugins folder from also screwing up your scripts, things such as multiple and conflicting versions of the same filter. Like you, I also thought it screwy that he was trying to do everything in 64-bit.

Katie Boundary

22nd May 2020, 07:36

UPDATE: I have independently confirmed the existence of at least one instance of retrograde field behavior: when the "Star Trek - Deep Space Nine" logo first fades in near the beginning of the opening credits. In addition, I'm having a hard time finding a combination of blockx and MI settings that will stop TFM from incorrectly flagging large amounts of progressive content as interlaced while still catching real interlaced content. The penalty for this is high - it causes the stars to blink in and out during space shots. I may switch to recommending blending instead of interpolation. I'll keep you all updated.

EDIT: for the sake of fairness, I tested QTGMC's performance as well. One of these images is my traditional TFM+Yadifmod2+NNEDI3 method. One of them is a variation of my method, using TFM+"Lossless" QTGMC. One of them is a variation of my method with TFM set to blend instead of bobbing/interpolation. Can you guess which is which?

https://i.imgur.com/oII2fLy.png

Finally, I recommend adding d2v="601.d2v",flags=1 to the TFM settings in both the C and D lines. This will prevent TFM from checking for interlacing during any soft pulldown, which cuts the false positive rate dramatically. The scene that benefits from this the most, by far, is the close-up of the comet's tail.

EDIT #2: Get ready because this next script is going to REALLY blow your damn minds...

Mpeg2source("601.d2v")

A=Tfm(field=1,mode=0,slow=2,pp=2,mchroma=false,cthresh=-1,micmatching=0).converttorgb().generalconvolution(matrix = "0 -1 0 0 4 0 0 -1 0",divisor=2,auto=false).converttoyv12()
B=Tfm(field=0,mode=0,slow=2,pp=2,mchroma=false,cthresh=-1,micmatching=0).converttorgb().generalconvolution(matrix = "0 -1 0 0 4 0 0 -1 0",divisor=2,auto=false).converttoyv12()
C=Tfm(field=1,mode=0,slow=2,mchroma=false,cthresh=-1,clip2=A,d2v="601.d2v",flags=1,micmatching=0)
D=Tfm(field=0,mode=0,slow=2,mchroma=false,cthresh=-1,clip2=B,d2v="601.d2v",flags=1,micmatching=0)

interleave(C,D)

Film content is, as usual, properly field-matched and then left the goddamn hell alone. Orphaned fields get blended with their nearest matches, then vertically sharpened to undo the resulting softness. This prevents the image from suddenly getting blurrier and sharper when transitioning between film content and interlaced content.

It will, of course, still spoil any true 60 hz content by creating ghosting and cutting the effective frame rate, but I think that's a fair price to pay here.

hello_hello

22nd May 2020, 09:15

I will be running DS9 tests soon, but for now, I'm still using Andromeda as my guinea pig for various deinterlacing methods. Just to be ABSOLUTELY sure about the relative merits of NNEDI3+Yadifmod2 versus QTGMC, I put the following script together:
(I kept the manually specified field arguments instead of using the complementparity approach because I know the field parity already)

This was the result:

It seems you found a frame where yadif/nnedi3 looks better while using ridiculously low threshold settings and sourcermatch etc for QTGMC. Well done. Here it is again, same TFM settings, no sourcematch or lossless for QTGMC.
Similar aliasing, and a tad sharper than Yadif/nnedi3.
Most of the time when QTGMC looks less sharp it'll be because it's removed noise, as it did for the example I posted earlier.

https://i.postimg.cc/hzjrcp2n/QTGMC.png (https://postimg.cc/hzjrcp2n)

No, you posted encodes using a lobotomized script that I provided for the purpose of demonstrating where Complementparity() would go. HUGE difference.

So I'll ask again. What difference would it have made using a non lobotomised version? Would TFM have taken more pixels from nnedi3 or less? Where's the sample of how you'd do it to show it's improved?

All of my ACTUAL methods use micmatching=0 and about a dozen other settings that I cut out of that particular bit of code because, as I stated, the complementparity demonstration code was NOT meant for actual use.

And I already said micmatching=0 would have fixed a glitch towards the end. That's it.
Lets be fair though and check out the huge difference you can expect to see with Katie's TFM settings vs the defaults, with the exception of micmatching=0 both times. I'll confess I've only checked the same frame again. I'm sure you'll be able to post screenshots where your settings are a vast improvement, won't you?

Katie TFM, Yadif/nnedi3, vs Default TFM, Yadif/nnedi3.
http://www.framecompare.com/image-compare/screenshotcomparison/77DPLNNX
I assume there's a difference, as the png files are slightly different sizes.

Are you not answering my question as to how you decimate for a VFR due to a lack of reading skills or because you're ignoring it?

All of my ACTUAL methods use micmatching=0 and about a dozen other settings that I cut out of that particular bit of code because, as I stated, the complementparity demonstration code was NOT meant for actual use.

So I'll ask again. What difference would it have made. Shouldn't you check a non lobotomised version is going to look better before complaining? Where's your sample encode?

manono

22nd May 2020, 09:18

Finally, I recommend adding d2v="601.d2v",flags=1 to the TFM settings in both the C and D lines. This will prevent TFM from checking for interlacing during any soft pulldown, which cuts the false positive rate dramatically. The scene that benefits from this the most, by far, is the close-up of the comet's tail.
Stereodude said something similar a couple of pages ago, except he likes Flags=5:

The problem is that TFM sees combing where there is none and deinterlaces large areas of the frames despite it being progressive with no real combing. 5 puts an end to that. I didn't seen any examples of combing (except in the occasional scene change) which 5 still checks and addresses.

hello_hello

22nd May 2020, 10:30

UPDATE: I have independently confirmed the existence of at least one instance of retrograde field behavior: when the "Star Trek - Deep Space Nine" logo first fades in near the beginning of the opening credits.

I can't see it. It looks fine to me if I change the field order.

N=Tfm(field=0,mode=0,micmatching=0)
O=Tfm(field=1,mode=0,micmatching=0)
Interleave(N,O)

I've no idea how you make your field order work at all for the DS9 sample.

Film content is, as usual, properly field-matched and then left the goddamn hell alone. Orphaned fields get blended with their nearest matches, then vertically sharpened to undo the resulting softness. This prevents the image from suddenly getting blurrier and sharper when transitioning between film content and interlaced content.

The comet tail is noticeably mushy. The CGI shot just before it switches to film has horrible quality pumping. The quality pumping seems worse in general. The sharpening makes the aliasing more noticeable.
I'm looking at an encode using your exact settings, including the field order. I don't think switching the field order to stop the frames bouncing back and forth would change that. Edit: I checked and it didn't.

I assume the transitions between shots have been de-interlaced by TFM and treated as video, so the transitions need to be kept at 29.97fps to stay smooth for a VFR. How do you do about picking them out if you decimate?

Katie Boundary

22nd May 2020, 10:31

All right, time to feed the troll one last time.

Are you not answering my question as to how you decimate for a VFR due to a lack of reading skills or because you're ignoring it?

I'm ignoring it because I don't give a tenth of a rat's ass about VFR, and have been advocating against VFR from the very beginning.

So I'll ask again. What difference would it have made.

The difference is that one is my method and the other isn't.

And now, back to serious discussion:

Stereodude said something similar a couple of pages ago, except he likes Flags=5

Flags=1 is even stricter than 5, because 5 will look for scene changes and then perform deinterlacing around those, whereas 1 will not look for interlacing in film content ever. If there are any orphaned fields around scene changes, then there will also be an interruption in the pulldown pattern, and TFM will start checking for interlacing again anyway. I'm therefore not sure what 5 would accomplish other than getting false positives around scene changes.