How to filter this 1080i program?

[laserfan]

Sorry guys I have spent weeks, maybe months trying to figure-out how best to convert NBC's ER 1080i HD transport streams (captured OTA with my MyHD MDP-130) and am at a complete loss. It seems in the end that a simple SeparateFields() works well enough (making an XviD file) but I wonder: is there any way to make this play-back smoother? It is (to me anyway) a bizarre and ugly MPEG2, BFF and lots of video intermixed w/Film. Here are DGIndex parse and d2v files of a small snippet of the program. If anyone can suggest specific filters/settings/methods to try on this I would appreciate it, this seems typical of an ER episode though:

D2V Parse Output

Encoded Frame: Display Frames....Flags Byte (* means in 3:2 pattern)
--------------------------------------------------------------------

[Field Operation None, using flags]
[GOP]
[Clip is BFF]
0 [I]: 0,0......................0
1 [B]: 1,1,2....................1 *
2 [B]: 2,3......................2 *
3 [P]: 3,4,4....................3 *
4 [B]: 5,5......................0 *
5 [B]: 6,6,7....................1 *
6 [P]: 7,8......................2 *
7 [B]: 8,9,9....................3 *
8 [B]: 10,10....................0 *
9 [P]: 11,11,12.................1 *
10 [B]: 12,13...................2 *
11 [B]: 13,14,14................3 *
12 [P]: 15,15...................0 *
[GOP]
13 [B]: 16,16,17................1 *
14 [B]: 17,18...................2 *
15 [I]: 18,19...................2
16 [B]: 19,20...................2
17 [B]: 20,21...................2
18 [P]: 21,22...................2
19 [B]: 22,23...................2
20 [B]: 23,24...................2
21 [P]: 24,25...................2
22 [B]: 25,26...................2
23 [B]: 26,27...................2
24 [P]: 27,28...................2
25 [B]: 28,29...................2
26 [B]: 29,30...................2
27 [P]: 30,31...................2
[GOP]
28 [B]: 31,32...................2
29 [B]: 32,33...................2
30 [I]: 33,34...................2
31 [B]: 34,35...................2
32 [B]: 35,36...................2
33 [P]: 36,37...................2
34 [B]: 37,38,38................3 *
35 [B]: 39,39...................0 *
36 [P]: 40,40,41................1 *
37 [B]: 41,42...................2 *
38 [B]: 42,43,43................3 *
39 [P]: 44,44...................0 *
40 [B]: 45,45,46................1 *
41 [B]: 46,47...................2 *
42 [P]: 47,48,48................3 *
[GOP]
43 [B]: 49,49...................0 *
44 [B]: 50,50,51................1 *
45 [I]: 51,52...................2 *
46 [B]: 52,53...................2
47 [B]: 53,54...................2
48 [P]: 54,55...................2
49 [B]: 55,56...................2
50 [B]: 56,57...................2
51 [P]: 57,58...................2
52 [B]: 58,59...................2
53 [B]: 59,60...................2
54 [P]: 60,61...................2
55 [B]: 61,62...................2
56 [B]: 62,63...................2
57 [P]: 63,64...................2
[GOP]
58 [B]: 64,65...................2
59 [B]: 65,66...................2
60 [I]: 66,67...................2
61 [B]: 67,68...................2
62 [B]: 68,69...................2
63 [P]: 69,70...................2
64 [B]: 70,71...................2
65 [B]: 71,72...................2
66 [P]: 72,73...................2
67 [B]: 73,74...................2
68 [B]: 74,75...................2
69 [P]: 75,76...................2
70 [B]: 76,77,77................3 *
71 [B]: 78,78...................0 *
72 [P]: 79,79,80................1 *
[GOP]
73 [B]: 80,81...................2 *
74 [B]: 81,82,82................3 *
75 [I]: 83,83...................0 *
76 [B]: 84,84,85................1 *
77 [B]: 85,86...................2 *
78 [P]: 86,87,87................3 *
79 [B]: 88,88...................0 *
80 [B]: 89,89,90................1 *
81 [P]: 90,91...................2 *
82 [B]: 91,92,92................3 *
83 [B]: 93,93...................0 *
84 [P]: 94,94,95................1 *
85 [B]: 95,96...................2 *
86 [B]: 96,97,97................3 *
87 [P]: 98,98...................0 *
[GOP]
88 [B]: 99,99,100...............1 *
89 [B]: 100,101.................2 *
90 [I]: 101,102,102.............3 *
91 [B]: 103,103.................0 *
92 [B]: 104,104,105.............1 *
93 [P]: 105,106.................2 *
94 [B]: 106,107,107.............3 *
95 [B]: 108,108.................0 *
96 [P]: 109,109,110.............1 *
97 [B]: 110,111.................2 *
98 [B]: 111,112,112.............3 *
99 [P]: 113,113.................0 *
[GOP]
100 [B]: 114,114,115............1 *
101 [B]: 115,116................2 *
102 [I]: 116,117,117............3 *
103 [B]: 118,118................0 *
104 [B]: 119,119,120............1 *
105 [P]: 120,121................2 *
106 [B]: 121,122,122............3 *
107 [B]: 123,123................0 *
108 [P]: 124,124,125............1 *
109 [B]: 125,126................2 *
110 [B]: 126,127,127............3 *
111 [P]: 128,128................0 *
112 [B]: 129,129,130............1 *
113 [B]: 130,131................2 *
114 [P]: 131,132,132............3 *
[GOP]
115 [B]: 133,133................0 *
116 [B]: 134,134,135............1 *
117 [I]: 135,136................2 *
118 [B]: 136,137,137............3 *
119 [B]: 138,138................0 *
120 [P]: 139,139,140............1 *
121 [B]: 140,141................2 *
122 [B]: 141,142,142............3 *
123 [P]: 143,143................0 *
124 [B]: 144,144,145............1 *
125 [B]: 145,146................2 *
126 [P]: 146,147,147............3 *
127 [B]: 148,148................0 *
128 [B]: 149,149,150............1 *
129 [P]: 150,151................2 *
[GOP]
130 [B]: 151,152,152............3 *
131 [B]: 153,153................0 *
132 [I]: 154,154,155............1 *
133 [B]: 155,156................2 *
134 [B]: 156,157,157............3 *
135 [P]: 158,158................0 *
136 [B]: 159,159,160............1 *
137 [B]: 160,161................2 *
138 [P]: 161,162................2
139 [B]: 162,163................2
140 [B]: 163,164................2
141 [P]: 164,165................2
142 [B]: 165,166................2
143 [B]: 166,167................2
144 [P]: 167,168................2
[GOP]
145 [B]: 168,169................2
146 [B]: 169,170................2
147 [I]: 170,171................2
148 [B]: 171,172................2
149 [B]: 172,173................2
150 [P]: 173,174................2
151 [B]: 174,175................2
152 [B]: 175,176................2
153 [P]: 176,177................2
154 [B]: 177,178................2
155 [B]: 178,179................2
156 [P]: 179,180................2
157 [B]: 180,181................2
158 [B]: 181,182................2
159 [P]: 182,183................2
[GOP]
160 [B]: 183,184................2
161 [B]: 184,185................2
162 [I]: 185,186................2
163 [B]: 186,187................2
164 [B]: 187,188................2
165 [P]: 188,189................2
166 [B]: 189,190................2
167 [B]: 190,191................2
168 [P]: 191,192................2
169 [B]: 192,193................2
170 [B]: 193,194................2
171 [P]: 194,195,195............3 *
172 [B]: 196,196................0 *
173 [B]: 197,197,198............1 *
174 [P]: 198,199................2 *
[GOP]
175 [B]: 199,200,200............3 *
176 [B]: 201,201................0 *
177 [I]: 202,202,203............1 *
178 [B]: 203,204................2 *
179 [B]: 204,205,205............3 *
180 [P]: 206,206................0 *
181 [B]: 207,207,208............1 *
182 [B]: 208,209................2 *
183 [P]: 209,210,210............3 *
184 [B]: 211,211................0 *
185 [B]: 212,212,213............1 *
186 [P]: 213,214................2 *
187 [B]: 214,215,215............3 *
188 [B]: 216,216................0 *
189 [P]: 217,217,218............1 *
[GOP]
190 [B]: 218,219................2 *
191 [B]: 219,220,220............3 *
192 [I]: 221,221................0 *
193 [B]: 222,222,223............1 *
194 [B]: 223,224................2 *
195 [P]: 224,225,225............3 *
196 [B]: 226,226................0 *
197 [B]: 227,227,228............1 *
198 [P]: 228,229................2 *
199 [B]: 229,230,230............3 *
200 [B]: 231,231................0 *
201 [P]: 232,232,233............1 *
202 [B]: 233,234................2 *
203 [B]: 234,235,235............3 *
204 [P]: 236,236................0 *
[GOP]
205 [B]: 237,237,238............1 *
206 [B]: 238,239................2 *
207 [I]: 239,240,240............3 *
208 [B]: 241,241................0 *
209 [B]: 242,242,243............1 *
210 [P]: 243,244................2 *
211 [B]: 244,245,245............3 *
212 [B]: 246,246................0 *
213 [P]: 247,247,248............1 *
214 [B]: 248,249................2 *
215 [B]: 249,250,250............3 *
216 [P]: 251,251................0 *
217 [B]: 252,252,253............1 *
218 [B]: 253,254................2 *
219 [P]: 254,255,255............3 *
[GOP]
220 [B]: 256,256................0 *
221 [B]: 257,257,258............1 *
222 [I]: 258,259................2 *
223 [B]: 259,260,260............3 *
224 [B]: 261,261................0 *
225 [P]: 262,262,263............1 *
226 [B]: 263,264................2 *
227 [B]: 264,265,265............3 *
228 [P]: 266,266................0 *
229 [B]: 267,267,268............1 *
230 [B]: 268,269................2 *
231 [P]: 269,270,270............3 *
232 [B]: 271,271................0 *
233 [B]: 272,272,273............1 *
234 [P]: 273,274................2 *
[GOP]
235 [B]: 274,275,275............3 *
236 [B]: 276,276................0 *
237 [I]: 277,277,278............1 *
238 [B]: 278,279................2 *
239 [B]: 279,280,280............3 *
240 [P]: 281,281................0 *
241 [B]: 282,282,283............1 *
242 [B]: 283,284................2 *
243 [P]: 284,285................2
244 [B]: 285,286................2
245 [B]: 286,287................2
246 [P]: 287,288................2
247 [B]: 288,289................2
248 [B]: 289,290................2
249 [P]: 290,291................2
[GOP]
250 [B]: 291,292................2
251 [B]: 292,293................2
252 [I]: 293,294................2
253 [B]: 294,295................2
254 [B]: 295,296................2
255 [P]: 296,297................2
256 [B]: 297,298................2
257 [B]: 298,299,299............3 *
258 [P]: 300,300................0 *
259 [B]: 301,301,302............1 *
260 [B]: 302,303................2 *
261 [P]: 303,304,304............3 *
262 [B]: 305,305................0 *
263 [B]: 306,306,307............1 *
264 [P]: 307,308................2 *
[GOP]
265 [B]: 308,309,309............3 *
266 [B]: 310,310................0 *
267 [I]: 311,311,312............1 *
268 [B]: 312,313................2 *
269 [B]: 313,314,314............3 *
270 [P]: 315,315................0 *
271 [B]: 316,316,317............1 *
272 [B]: 317,318................2 *
273 [P]: 318,319,319............3 *
274 [B]: 320,320................0 *
275 [B]: 321,321,322............1 *
276 [P]: 322,323................2 *
277 [B]: 323,324,324............3 *
278 [B]: 325,325................0 *
279 [P]: 326,326,327............1 *
[GOP]
280 [B]: 327,328................2 *
281 [B]: 328,329,329............3 *
282 [I]: 330,330................0 *
283 [B]: 331,331,332............1 *
284 [B]: 332,333................2 *
285 [P]: 333,334,334............3 *
286 [B]: 335,335................0 *
287 [B]: 336,336,337............1 *
288 [P]: 337,338................2 *
289 [B]: 338,339,339............3 *
290 [B]: 340,340................0 *
291 [P]: 341,341,342............1 *
292 [B]: 342,343................2 *
293 [B]: 343,344,344............3 *
294 [P]: 345,345................0 *
[GOP]
295 [B]: 346,346,347............1 *
296 [B]: 347,348................2 *
297 [I]: 348,349,349............3 *
298 [B]: 350,350................0 *
299 [B]: 351,351,352............1 *
300 [P]: 352,353................2 *
301 [B]: 353,354,354............3 *
302 [B]: 355,355................0 *
303 [P]: 356,356,357............1 *
304 [B]: 357,358................2 *
305 [B]: 358,359,359............3 *
306 [P]: 360,360................0 *
307 [B]: 361,361,362............1 *
308 [B]: 362,363................2 *
309 [P]: 363,364,364............3 *
[GOP]
310 [B]: 365,365................0 *
311 [B]: 366,366,367............1 *
312 [I]: 367,368................2 *
313 [B]: 368,369................2
314 [B]: 369,370................2
315 [P]: 370,371................2
316 [B]: 371,372................2
317 [B]: 372,373................2
318 [P]: 373,374................2
319 [B]: 374,375................2
320 [B]: 375,376................2
321 [P]: 376,377................2
322 [B]: 377,378................2
323 [B]: 378,379................2
324 [P]: 379,380................2
[GOP]
325 [B]: 380,381................2
326 [B]: 381,382................2
327 [I]: 382,383................2
328 [B]: 383,384................2
329 [B]: 384,385................2
330 [P]: 385,386................2
[GOP]
331 [I]: 386,387................2
332 [P]: 387,388................2
[EOF]

DGIndexProjectFile13
1
D:\ERtod.ts

Stream_Type=2
MPEG2_Transport_PID=1000,1001
MPEG_Type=2
iDCT_Algorithm=6
YUVRGB_Scale=1
Luminance_Filter=0,0
Clipping=0,0,0,0
Aspect_Ratio=16:9
Picture_Size=1920x1088
Field_Operation=0
Frame_Rate=29970 (30000/1001)
Location=0,0,0,2E94

900 1 0 376 0 0 90 f1 b2 e3 b0 f1 a2 f3 b0 e1 b2 f3 a0
900 1 0 1042648 0 0 71 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 1 0 2107668 0 0 32 32 92 b2 b2 a2 f3 b0 e1 b2 f3 a0 f1 b2 e3
900 1 0 3113468 0 0 30 71 92 b2 b2 a2 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 1 0 4114944 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2 f3 b0 e1
900 1 0 5109088 0 0 32 73 90 f1 b2 e3 b0 f1 a2 f3 b0 e1 b2 f3 a0
900 1 0 6207572 0 0 71 32 d3 b0 f1 a2 f3 b0 e1 b2 f3 a0
900 1 0 7144564 0 0 71 32 d3 b0 f1 a2 f3 b0 e1 b2 f3 a0 f1 b2 e3
900 1 0 8314488 0 0 30 71 92 f3 b0 e1 b2 f3 a0 f1 b2 e3 b0 f1 a2
900 1 0 9481216 0 0 73 30 d1 b2 f3 a0 f1 b2 a2 b2 b2 a2 b2 b2 a2
900 1 0 10609404 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 1 0 11633816 0 0 32 32 92 b2 b2 a2 b2 b2 a2 b2 b2 e3 b0 f1 a2
900 1 0 12561784 0 0 73 30 d1 b2 f3 a0 f1 b2 e3 b0 f1 a2 f3 b0 e1
900 1 0 13635076 0 0 32 73 90 f1 b2 e3 b0 f1 a2 f3 b0 e1 b2 f3 a0
900 1 0 14815528 0 0 71 32 d3 b0 f1 a2 f3 b0 e1 b2 f3 a0 f1 b2 e3
900 1 0 15972668 0 0 30 71 92 f3 b0 e1 b2 f3 a0 f1 b2 e3 b0 f1 a2
900 1 0 17151804 0 0 73 30 d1 b2 f3 a0 f1 b2 a2 b2 b2 a2 b2 b2 a2
900 1 0 18309884 0 0 32 32 92 b2 b2 a2 b2 f3 a0 f1 b2 e3 b0 f1 a2
900 1 0 19362684 0 0 73 30 d1 b2 f3 a0 f1 b2 e3 b0 f1 a2 f3 b0 e1
900 1 0 20421876 0 0 32 73 90 f1 b2 e3 b0 f1 a2 f3 b0 e1 b2 f3 a0
900 1 0 21609848 0 0 71 32 d3 b0 f1 a2 f3 b0 e1 b2 f3 a0 f1 b2 e3
900 1 0 22764920 0 0 30 71 92 b2 b2 a2 b2 b2 a2 b2 b2 a2 b2 b2 a2
900 1 0 23864156 0 0 32 32 92 b2 b2 a2
900 1 0 24285088 0 0 92 a2 ff

FINISHED 67.07% FILM

[Guest]

Post a link to ERtod.ts and tell us your script.

[Revgen]

Does it play in VLC player?

[laserfan]

neuron2 I have sent you a PM, thanks.

There is no problem with playback of the ts using any player--don't recall specifically if I've used VLC on it, but will if this is supposed to tell me something? The original clip plays smoothly.

My favorite HD TS player for the PC is Media Player Classic with the Elecard MPEG-2 decoder.

[Guest]

FTP password sent. Please advise by PM when the file is uploaded. Thank you.

[laserfan]

...tell us your script.BTW here are just a few of the filters/script entries I've tried, the first from AutoGK and the rest from StaxRip or suggested by MeGUI or my own attempts. Just trying anything to see what might work. Maybe the way these shows are encoded (i.e. maybe they are just junk), SeparateFields is the best I can do, but I'd sure like someone who has more experience/understanding than I tell me that!

1. KernelDeInt(movie,order=0,sharp=true)
2. FieldDeinterlace()
3. SeparateFields().SelectEven()
.
4. AssumeBFF()
Telecide(guide=1)
Decimate(mode=3,threshold=2.0)
.
5. tfm().tdecimate(hybrid=1)
6. Telecide(guide=1).Decimate()
.
7. AssumeFieldBased()
SmoothDeinterlace(tff=false, doublerate=true)

[Guest]

There are at least two bizarre things in the clip.

First, it has blended fields. I consider that strange for a broadcast HD series.

Second, there are strange fields. For example, a head turns. In one field just the eye moves and in the following field the rest of the face catches up. Stuff like that reminds me of motion compensated frame rate conversion.

Discarding a field and resizing is fine if you are going to a smaller format, like 720x480. But you'll still have blends and funny motions. If you want to keep it at full size, then you can try IVTC:

assumebff()
telecide(post=2,vthresh=25,blend=false)
decimate()

Again, you can't avoid the blends and funny motions.

Often, one can reduce blends by selectively discarding top or bottom fields, but this cherry picking usually adversely affects smoothness.

These artifacts are acceptable when played interlaced. But when made progressive, the presentation time of the artifacts is doubled, making them objectionable. I almost wonder if it is an intentional mechanism.

[laserfan]

neuron2 I really appreciate that you looked at this clip--it's been driving me nuts and I'm frankly glad that you found it bizarre. I don't think all the ER episodes are nearly this bad but in this episode's case the clip I gave you was right at the beginning of the show, and so it inspires one to look for oddities the rest of the way along.

In any case I will try your ideas; thanks very much for taking the time to look at this!

:thanks:

[laserfan]

...Discarding a field and resizing is fine if you are going to a smaller format, like 720x480. But you'll still have blends and funny motions. If you want to keep it at full size, then you can try IVTC:

assumebff()
telecide(post=2,vthresh=25,blend=false)
decimate()

Again, you can't avoid the blends and funny motions...I took my most recent ER recording and converted the whole thing again just discarding a field (separate and select) and thought it looked OK. Then I tried your script on a 4 minute segment (though this clip is a tff, go figure), and it did seem somehow smoother to me. So I opened each encoding w/VDubMod and stepped thru them.

The "discard field" method results in 29.971fps but about 1 "freeze frame" in 5. No doubt a result of not IVTC'ing at all.

Your method (23.976) viewed frame-by-frame clearly does not have any dupes, so that's the reason it looked smoother (duh). I went back and looked at a number of my older conversions and now I know they are all pretty much hosed-up with duplicate frames. :(

I guess I'm developing an appreciation for the problems of converting hybrid programs--in a way I wish I couldn't "see" the jerkiness, because running your hybrid script takes a LOT longer than the less-capable methods. Guess I'll continue to experiment and see if I can find something faster.

Thanks again neuron2 for your help. I've been really slow to pick-up on this stuff but at least now I feel I've made some headway... :o

[laserfan]

I applied the above script to the entire original program and it took 5 1/2 hours! But it looks really good. I had tried a script from your Decomb docs neuron2:

Telecide(guide=1)
Decimate(mode=3,threshold=1.0)

and got a "blended field" in the output, then I added Quality=3 to Decimate on a whim and it didn't make the blend go away, but somehow your magic script above works perfectly. Man, I'd have never wandered-in to that method on my own, thanks again!

Now I have a half dozen or so episodes that are 29.971 Xvids with every 5th frame a duplicate. Can anyone suggest if there is an easy way to fix this? I tried a simple VDubMod Direct Stream Copy with Decimate by 5 in Frame Rate conversion but ended-up w/15fps somehow...seems like there should be a way??

[Guest]

Now I have a half dozen or so episodes that are 29.971 Xvids with every 5th frame a duplicate. Can anyone suggest if there is an easy way to fix this?

avisource("file.avi")
decimate(5)

Load that AVS into VirtualDub and reencode.

VirtualDub decimation is not smart; it's not guaranteed to remove the duplicates.

[laserfan]

Thanks for that--next time I get to your neck-of-the-woods, Malnati's is on me!