TDeint and TIVTC

[tritical]

Is there anything in the way of logs that would help in tracking down what's happening at 1307?

If you could post the d2v file and a list of which frames show up combed when matched from the d2v flags with flags=1 that would help the most.

It appears that micmatching is not used when TFM is taking the field flag information for that frame from the d2v. In order to be helpful with needed u/b matches around scenechanges, I would think you'd need it to be always on, not just when out of pattern.

It then somewhat defeats the purpose of matching off the flags... which is to avoid needing to compare matches and check if frames are combed, but I might consider changing it in the future. There hasn't really been enough testing with flags=1 on sources that have film percentages in the 10 to 90 range (I don't have any atm) to determine exactly how everything should work.

[TheBashar]

Originally Posted by tritical

If you could post the d2v file and a list of which frames show up combed when matched from the d2v flags with flags=1 that would help the most.

Sure thing! The following 1.3MB linked 7z archive contains:

• ep.d2v - Self Explanatory
• ept.avs - Contains FirstPass() function used for output
• test.log - 33MB log of avs processing
• scenex.txt - List of scene change interlaces

http://nerdswap.com/files/9ae51aaa8b...19thebashar.7z

There are many frames with interlacing, but the scenex.txt file lists only the frames with blatant scene change whole frame interlacing. I verified these by hand with the ept.avs script after commenting out the tdeint call. I included the debug log in case it would be informative. I used it (comparing tdeint to tfm) to locate the interlacing frames to be verified.

Originally Posted by tritical

It then somewhat defeats the purpose of matching off the flags... which is to avoid needing to compare matches and check if frames are combed, but I might consider changing it in the future.

That makes sense. Unfortunately there are some frames that have been encoded with with some interlacing/combing in them. I would really like some way to detect those frames so that they can be subject to TFM's decombing options.

I humbly suggest changing the micmatching option to:

[EDIT: Removed suggestion due to better idea in next post.]

[TheBashar]

Tritical,

I was thinking some more about my suggestion that I posted last night, and I think there's a better suggestion to be had. My previous suggestion had the problem that it mixed two concepts (trusting d2v flags & micmatching method) in one parameter.

I would like to humbly suggest adding an additional parameter to d2v flag controls and modifying the micmatching. What would be most beneficial for me would be something like:

d2vsuggest (bool):

• false - [default] When d2vsuggest is false, flags taken from the d2v according to the "flags" parameter are implictly trusted as long as they are in-pattern.
• true - When d2vsuggest is true, flags taken from the d2v according to the "flags" parameter are treated as suggestions even if they are in-pattern.

Explanation of "treated as suggestions": There are several matching modes (6 right now) in TFM. When a d2v file would be provided, and the flags used, and they are in-pattern, and d2vsuggest=true, the field match suggested by the d2v flags would be moved from its place in the selected mode's matching sequence to front of the line. For example, in mode=3 with a d2v suggested field match of "n", the match sequence would become (n + p/c + u/b). It should be noted that with d2vsuggest=true, the current match would always checked for combing.

With that concept broken off, my suggestion of micmatching simplifies to just whether or not the MIC has to rise above some small threshold X or the MI. The small threshold that I am referring to is what you refer to as "significantly lower" in the current micmatching description (45 I think). So it would become something like:

micmatching (int):

• 0 - [default] micmatching is not used.
• 1 - Frames that are checked for combing and have MIC > X are subjected to micmatching. MIC are calculated for all other matches and match is overriden if new match MIC < old match MIC - X.
• 2 - Frames that are checked for combing and have MIC > MI are subjected to micmatching. MIC are calculated for all other matches and match is overriden if new match MIC < old match MIC - X.

Personally, for the sources I'm presently working with d2vsuggest=true and micmatching=2 would offer significant benefits. While they do add a speed penalty, it comes in the form of only checking each frame for combing in the majority of cases. Only a minorty of matches would be subjected to the full micmatching every match possible combing check.

Does any of this sound valuable to you? If so, I'd be willing to take a crack at the source code. I'm not beneath trying to help code solutions that would help me.

[tritical]

When I click that link it brings up a page that says "you have clicked on a stale link".

Adding an option like d2vsuggest would be possible.

With that concept broken off, my suggestion of micmatching simplifies to just whether or not the MIC has to rise above some small threshold X or the MI. The small threshold that I am referring to is what you refer to as "significantly lower" in the current micmatching description (45 I think). So it would become something like:

micmatching (int):
0 - [default] micmatching is not used.
1 - Frames that are checked for combing and have MIC > X are subjected to micmatching. MIC are calculated for all other matches and match is overriden if new match MIC < old match MIC - X.
2 - Frames that are checked for combing and have MIC > MI are subjected to micmatching. MIC are calculated for all other matches and match is overriden if new match MIC < old match MIC

The significantly lower threshold is a relative percentage threshold and not exclusively a hard threshold. Currently, it is 1/3 of the next lowest match, but to prevent weird things with low values (like 1 vs 5) there is a hard threshold minimum of 15. When I first added it I hijacked micout for micmatching so it calculated all of them regardless. I went ahead and separated the two so that when micmatching=true it only calculates the other mics when the final match has a value > 15, most frames don't produce a value that large. Adding an option to only do it when the final match value > MI is possible as well though.

[TheBashar]

Originally Posted by tritical

When I click that link it brings up a page that says "you have clicked on a stale link".

Damn, sorry about that. Try this:
http://nerdswap.com/files/9ae51aaa8b...19thebashar.7z

Thanks!

[TheBashar]

Originally Posted by tritical

I went ahead and separated the two so that when micmatching=true it only calculates the other mics when the final match has a value > 15, most frames don't produce a value that large. Adding an option to only do it when the final match value > MI is possible as well though.

I guess it's pretty arbitray, but I think the last released version had a threshold of 45 not 15. The absolute difference has to be 15 and it has to be 3 times less... so combined I think that means it should be value > 45.

[tritical]

The 15 was the minimum value between the lowest match and the next lowest match, the 1/3 was also between the lowest match and the next lowest match. That is if you had 5 21 30 25 25 it would override because abs(5-21)>15 and 5*3 < 21, but if you had 0 5 5 5 5 or 10 28 28 28 28 it wouldn't.

[TheBashar]

Oops, I multiplied by 3 on the wrong side. Thanks for clearing that up.

Sorry to be such a PITA with all the d2v issues. I'm just spoiled. I really like the speed up from the d2v info but I really like the quality I can get from using tfm's motion adaptive masking in conjunction with tdeint (PP=6 with clip2).

[tritical]

Its not a pain, any suggestions are welcome I just don't always have the time to implement/test things right away.

Anyways, I looked at the d2v and scenex.txt files and, unfortunately, all the frames that are listed are right in the middle of long 3:2 pattern runs and also have the progressive frame flag set on top of that. Is that stream off a dvd or is it some type of capture? I've seen a similar situation before on an hdtv capture, just wanted to know where yours came from.

Any way that I would be able to get part of it? Say the first 5000 to 10000 frames? If so, you could just upload it to the ftp address I posted before:

ftp://upload:upload@68.119.245.113:17252/

[TheBashar]

Originally Posted by tritical

Anyways, I looked at the d2v and scenex.txt files and, unfortunately, all the frames that are listed are right in the middle of long 3:2 pattern runs and also have the progressive frame flag set on top of that. Is that stream off a dvd or is it some type of capture?

That's what I was afraid of. I thought perhaps the problem was some of the interlaced scenechanges were encoded into a progressive frame. I think if the d2v says its interlaced (from telecine) it's probably trustable, but it would have no idea if a regular frame would be interlaced from the source.

It's a DVD. I'm trying to get my partial SG-1 collection cleaned up (some have tons of grain) and made available to the HTPC box. From what I gather, these TV show DVD collections are usually pretty challenging. There's the telecine, then (as seems to be the case here) there's some natively interlaced stuff, there's pattern changes in the CGI sections, different patterns in slow motion sections (surprising how often), and often less than ideal cuts.

Originally Posted by tritical

Any way that I would be able to get part of it? Say the first 5000 to 10000 frames?

I suppose a small portion of demuxed m2v would be a reasonable fair use for analysis purposes.

Thanks for all your help!

[levi]

I really like tivtc. Use it on occassions where decomb fails. I haven't used it in a while, tho. Right now I'm trying to figure out which dgmpgdec version I should use with TIVTCv09101.zip. Keep getting d2v 'unsupported format' errors. Thanks for the excellent tool & keeping it up to date!

[tritical]

v0.9.10.1 supports every dgindex project file type ever used up to v11 (plus dvd2avi 1.76, dvd2avi 1.77, and dvd2avidg) which is the latest unless it was changed again in 1.4.4. What version are you getting the unsupported format error with?

[levi]

hmm

ok, i tried several versions. I'm running on 64bit (intel) Windows. Could that cause a problem?

[tritical]

It could be, but I've never used a 64bit version of windows and don't know much of anything about the issues related to running 32bit programs with it.

I tested tfm with the d2v you posted (it is v11) and it worked fine here. The debug output from tfm shows:

TFM: v0.9.9.4 by tritical
TFM: successfully opened specified d2v file.
TFM: newest style (dgindex 1.2+) d2v detected.
TFM: auto detected field order from d2v is TFF.
TFM: no errors found in d2v.
TFM: d2vflags = 131242 out_of_pattern = 38595 (70.6% FILM)

[Guest]

Yeah, but did you notice his AVS script refers to a different D2V file???

[foxyshadis]

Is tdec mode=5 reading of tfm output file broken? Or am I just giving something the wrong inputs?

1st:
x=x.tfm(mode=0,ovr="sisters.tfmovr.txt",output="sisters.tfm.txt")
x=x.tdecimate(mode=4,output="sisters.tdec.txt")
return x.crop(0,0,4,4)

2nd:
x=x.tfm(ovr="sisters.tfmovr.txt",input="sisters.tfm.txt")
x=x.tdecimate(mode=5,hybrid=2,vfrDec=0,input="sisters.tdec.txt",tfmin="sisters.tfm.txt",mkvout="sisters.mkvtmp.txt")

2nd pass load always generates an exception "TDecimate: tfmIn file error (invalid specifier)!" I looked in the source and it's coming from TDecimate.c line 2576, apparently expecting a + or - after every framenum and match. I inserted a - on each line and it loaded like a charm. Otherwise they're unedited.

sisters.tfm.ovr is just "0,11853 c" since it's not telecined or combed, a few parts of the video just have dupe frames inserted every 5th and it's crazy jerky.

[tritical]

Yep, it is broken for the case when there aren't +/- markings (combed/not-combed) in the tfm output file because I forgot about the new line feed character. However, the only way to generate a tfm output file that doesn't have +/- markings is by setting PP=0, and I don't see that in your script. Anyways, it will be fixed in the next release.

[foxyshadis]

Sorry, I guess I shouldn't have edited the calls at all, I assumed they were equivalent. My bad.

[tritical]

[link removed]. This version fixes the tdecimate error when reading in tfmIn files w/o +/- markings, micmatching=true only calculates the extra mic values when it is possible the match could be overridden, and adds a flags=4 option. The flags=4 option is equivalent to flags=1, except that d2v matches are checked for being combed, and if a d2v match is detected as combed tfm uses its own matching instead.

[TheBashar]

Marvelous! Thank you. I can't wait to give flags=4 a try!