DV Footage -> SVCD -- Choppy / Jerky scenes on TV

[max2k]

Hi Folks:

I've been spending nearly my whole weekend trying to figure this out ...

I have some footage shot using my Sony Digital 8 NTSC camcorder.

Want to try and make a bunch of SVCDs out of this ( using AVI2SVCD )
I have CCE 2.5 for encoding.

When I see the results on the TV - the scenes are all kind of jerky - choppy movement - not smooth by any measure.

I searched and saw somewhere that you need to execute "pulldown" to fix the CCE field order bug - did that - no improvement...

Any ideas on what I should do
(1) Keep the video interlaced or use Decomb to Deinterlace ?
(2) Execute ( or not execute ) Pulldown ?

Again this is from a DV footage from a NTSC camcorder.

Any help to keep me from pulling my hair out ?

[vidiot]

I did have the same difficulties with PAL DV once:

Some DV Codecs seem to behave different from others regarding to the field order. But I´m not totally sure about that.

Try to feed your encoder with VirtualDub and use the smartdeinterlacer from D. Graft (neuron2) with settings:
"phase shift" and field "swap after phase shift"

If your encoder is CCE just try to change the fields with mpegchanger
(I´ll guess you´ll find some programms here @doom...)

Harald

[bb]

:confused:

Phase shift correction? This applies to progressive video, but not to DV sources.

I'd try FieldDeinterlace() from the Decomb package.

max2k: Could you please describe a little better how your video is jerky/choppy? Does it jump up and down?

bb

[max2k]

Thanks for the ideas ...

Sorry I was not very clear -- i was at my wits end.

I don't think it jumps up and down -
more like when there is movement - as in when the camera is sweeping from left-to-right ( panning ) in a scene - the end result is not smooth - its as if "something" is lost in between.

Sorry - i make a lousy explanation -- maybe I can post a couple of 10 second clips for you all to see ?
The original AVI and what the resulting MPEG was like ?

Also the picture is just fine when I hook up the camcorder straight to TV ( to play the original tape ).

I tried FieldDeinterlace() - the result did not change.

@vidiot --> can you please give me the code snippet to use for the functions you mentioned ?

Thanks folks - let me know if posting a sample clip will make it easy to figure out the problem ...

EDIT: Added information:
* I use Premiere 6.0 to capture using firewire
* The codec that comes up by default is "Microsoft DV(NTSC)"
* It also says "Lower Field First"
* I installed the Panasonic codec - could not use it - it just doesn't show up as an option for me.

[SomeJoe]

How are you selecting your options in CCE, and how are you running pulldown?

In CCE, you should have:

- DVD compliant NOT checked
- Progressive frames NOT checked
- ZigZag Scanning Order NOT checked
- Drop Frame checked

Run pulldown.exe on the resulting MPEG file as follows:


pulldown.exe <source> <target> -nopulldown -norff -tff even -drop_frame true

[vidiot]

Originally posted by bb
:confused:

Phase shift correction? This applies to progressive video, but not to DV sources.

I'd try FieldDeinterlace() from the Decomb package.
bb

bb, I know that sounds weired!
It´s an "old" tip from codepage that I tried over a year ago - at this time I encoded my DV footage only with TMPenc.
I wouldn´t have mentioned it if I haven´t had luck with this...
(believe me).

I don´t know if I´m allowed to post text from this web side, which doesn´t seem to be available anymore.

IF NOT - please mod: delete the following!


-----------------------------
-----------------------------
DV Field order problem

Own observations and problem reports by several users have revealed that there must be compatibility issues in the way interlaced video fields are ordered (or decoded, as DV stores fields into 1 frame) against other video formats available.

Obviously, the captured DV shows inverted field order compared to TV captures as well as industrial DVDs, and not only that, it has inverted even and odd TV lines AND inverted sequence of field playback, so the effects compensate for still images but become apparent if something moves. It is not yet clear to me what really causes the problem. It survives encoding to different compression formats but I also had the case that an SVCD directly encoded from DV (with TMPGenc) and burned with NERO worked correctly on a desktop DVD player.

The Field order switches available in most softwares can not compensate for this problem. They can only switch spatial or temporal. Smart Deinterlacer for VirtualDub so far is the only one that can do both (see below).

The effect always shows up, when DV is converted to another video format interlaced, or when another interlaced source is converted to DV and then sent to tape. I tested MJPEG, MPEG2, DVDs, MediaPlayer, Matrox TVout, and people reported the problem with self made DVDs also. Even MPEG2 directly made in StudioDV, from original DV capture, may be inverted, as are typically all Pinnacle DC10 MJPEG captures when imported to StudioDV. I also tested capturing with DVIO and MPEG2 encoding with TMPGenc and the problem was also there. So it can't be a problem of StudioDV but is a problem of DV in general. The inversion compensates for mere editing and recording back to tape, so I first suspected that the Camcorders use inverse field order, but the real problem may as well be the soft DV codecs, because they interface the DV stream not only to other formats but also to MediaPlayer for playback. So far I've tested Microsoft's and MainConcept's codec because these are the best and most frequently used ones. Even if the Camcorder codecs would be the real course of the inversion, they can't be changed any more with tens of millions already in use, so a change in the soft codecs could be the only cure for the problem.

So far I could determine that this weird behavior very often occurs, but there are cases where it does not, so the only way to determine if a certain conversion process is affected is just to try it out very carefully. For example, interlaced SVCD from TMPGenc 2.0 played well on desktop DVD players. TMPGenc's frame doubling deinterlacer filters (Even/Odd...) also work properly.

Several software DVD players can deinterlace, and also my TV-out (Matrox) has a help file that misleads users to switch to one field with interlaced material. No wonder only a few people stumbled over the problem and even less could identify it.

As long as there is no other solution, here is a workaround:

Donald Graft's Smart deinterlacer filter for VirtualDub, latest version, has more field ordering options.
Just check phase shift, field swap after phase shift, disable motion processing. That does resolve it.

VirtualDub can encode to MPEG4 directly or frameserve to TMPGenc (MPEG2) or MS WindowsMedia8 encoder. All work well with interlaced material. A definite advantage for storing DV footage on a CD, if you have a good TV-output.

--------------------
--------------------

[max2k]

@SomeJoe:

Thanks for ur ideas ...

I'm not at my home PC right now - I do remember some of the CCE options
- progressive & Zig zag - not checked

Other options - I will have to go back and check them in the evening.

There was one other CCE options ( Linear Quant scale ) - should that be checked or not ??

Am pretty sure I did not provide the extra options to pulldown - I will try again with all your options as is.

I will let you know what happens later today.

Keep the ideas coming folks - u are the best !!!

[max2k]

Originally posted by vidiot
<snip>

As long as there is no other solution, here is a workaround:

Donald Graft's Smart deinterlacer filter for VirtualDub, latest version, has more field ordering options.
Just check phase shift, field swap after phase shift, disable motion processing. That does resolve it.

--------------------
--------------------

vidiot: Do you happen to have the AVS code for using these mentioned functions ?

I would have to put them in the AVS script for AVI2SVCD to use.

Thanks :)

[Swan]

Originally posted by max2k
vidiot: Do you happen to have the AVS code for using these mentioned functions ?

I would have to put them in the AVS script for AVI2SVCD to use.

Thanks :)
Isn't max2k's source interlaced?
Don't deinterlace if you're making an SVCD or DVD!

I think your problem is CCE and the way it handles DV (Bottom Field First) material.
After extensive testing (I too work with DV compressed avi)I've concluded the following:
If I check the box with the setting "Upper Field First", CCE converts Bottom Field First video to Top Field First (and does it well too) and sets the flag in the Mpeg-2 header to Top Field First. This is of course correct, but there is no need to convert the Field Order for SVCD or DVD.
If I uncheck "Upper Field First", CCE doesn't change the field order, but still sets the flag in the Mpeg-2 header to Top Field First.
This is wrong and is what causes the field order problem you're seeing.

Solution: After encoding with CCE and having the "Upper Field First" setting unchecked during the encoding, run the Mpeg-2 file through either Restream (available on Doom9's download page) or Darim Mpeg Changer, a small software that can be found via a search engine.
These programs remove the erroneous Top Field flag and all is well again. :-)

/Maria

[max2k]

Thanks Maria --> yes I think my camcorder captures only interlaced - I think the progressive ones are expensive ( and mine was definitely not the top end stuff !! )

Maybe my problem is what you describe - if I understand your post correctly, i can check the "Upper Field first" box in CCE - that just makes it all ok - maybe waste some processing time.
(OR) leave it unchecked and do the post-encoding steps you mentioned.

I might use CCE to make it OK -- how much is the performance ( or other ) penalty by letting CCE do the field conversion ?
Will the picture look normal in the end ?
( You can see what happens to someone that was bugged out for 2 days on a single problem ;) ;) ).

Thanks so much !

[vidiot]

Originally posted by max2k
vidiot: Do you happen to have the AVS code for using these mentioned functions ?

I would have to put them in the AVS script for AVI2SVCD to use.

Thanks :)
@max2k:
Too bad, - no I don´t have an avs to that!
But what about posting that question in the avisynth forum?


But before that just try to feed your encoder with VirtualDub via
the frameserver... - if this is going to help in your case there is still time enough for further questions.

Harald

[Swan]

Originally posted by max2k
I might use CCE to make it OK -- how much is the performance ( or other ) penalty by letting CCE do the field conversion ?
None that I've noticed. It is a speed demon. :-)
But I personally prefer TMPGEenc, especially for SVCD. You'll run into other problems when making SVCDs with CCE too. So be prepared for more hair pulling. Seriously. One tip: Mux with BBMpeg. :-)

Will the picture look normal in the end ?
Yes. I just want to make sure you understand that there is nothing wrong with your original DV avi file. The reason you're getting the shaky picture is because of a bug in CCE. If you'd encoded the video in TMPGEnc it would have detected the video as Bottom Field First and you'd never have had to be "bugged out" on this problem. ;-)
Yes, the picture looks normal if you let CCE create a Mpeg-2 file with Top Field First from a Bottom Field First video. It will also look normal if you just remove that Top Field First flag after encoding in CCE with Upper Field First *unselected*.
I myself prefer the later method, as there is no reason to convert the field order.
( You can see what happens to someone that was bugged out for 2 days on a single problem ;) ;) ).
Ha, ha. Glad to have (hopefully) been of help. Let me know if you have better success now after knowing about the bug in CCE.

/Maria

[SomeJoe]

Originally posted by Swan
After extensive testing (I too work with DV compressed avi)I've concluded the following:
If I check the box with the setting "Upper Field First", CCE converts Bottom Field First video to Top Field First (and does it well too) and sets the flag in the Mpeg-2 header to Top Field First.

Do you know what method CCE is using to convert the field order in this situation? All the times I have tried this, the conversion is not correct.

The only way I've even been able to get good interlaced encodes out of CCE (from lower-field-first DV) is to uncheck "Upper Field First", and then run pulldown/restream on the .m2v file.

[max2k]

Swan / SomeJoe / vidiot / bb:

Thank you all for your help.
I followed Maria's advice to overcome the CCE bug.

I encoded with CCE - left the field order as is.
I had to use "Restream" to fix the headers.
Then I used "bbMPEG" to mux the files and burnt the disc.

Voila == all the jitter / shake was gone and the video was as good as the original ( well - very close !! ).

@Maria: I will try using TMPGEnc - if that takes care of this !

Thanks all -- i'm happy to have a solution finally -- now only if I can find a way to automate all of this in AVI2SVCD ( running Restream etc ) - guess i'll go back to searching.

Hope this thread helps someone who hits by searching !

Thanks again :)

[Swan]

@max2k @Maria: I will try using TMPGEnc - if that takes care of this !
TMPGEnc analyzes the avi file when it loads it, so it encodes the field order correctly. I have never seen it misjudge a DV stream.
It always selects "Bottom Field First" by itself, certanly 100% spot on when using the wizard.
I am so glad my info helped you. I spent two days figuring this out in January when I got my first DV captures. Before that I'd had a regular "capture card" and those capture Top Field First. :-D

@SomeJoe
Do you know what method CCE is using to convert the field order in this situation? All the times I have tried this, the conversion is not correct.
I don't know what method it uses, but it can't be a complicated operation. I assume this because I have a Matrox G460 eTV graphics card. It can not output anything but video with Top Field First (TFF) order on the TV-Out correctly. BFF video is output to the TV, but with the wrong field order.
So, in order to output Bottom Field First (BFF) video to the TV, I load a nifty Directshow filter called DivXG400 which converts BFF to TFF on the fly. If it can be done on the fly, then it can't be a difficult thing to do for CCE as it encodes.

I tested the method I described to Max2k exhaustively and it works every time for me. I am working with interlaced PAL BFF avi files with DV compression. I guess you are working with NTSC interlaced material? Are you using pulldown.exe for removal of the TFF flag?
Or for 2-3 pulldown only, as you mention using Resteam too?

From what I have read in the readme to pulldown.exe, it does the 2-3 pulldown needed for playback on a NTSC device when having encoded a 23.97 fps video.
But it seems pulldown.exe can also remove the erroneous TFF flag CCE sets with setting the switch "-tff [odd, even]"
From the readme:
"To change the field order to odd on a stream that you accidentally encoded as even:
pulldown.exe source.m2v target.m2v -nopulldown -tff odd"

I just noticed that Doom9 also has "DoPulldown - GUI'ed version of pulldown.exe" on his download page. Might be worth checking out.:)

/Maria

[SomeJoe]

Originally posted by Swan
I don't know what method it uses, but it can't be a complicated operation. I assume this because I have a Matrox G460 eTV graphics card. It can not output anything but video with Top Field First (TFF) order on the TV-Out correctly. BFF video is output to the TV, but with the wrong field order.
So, in order to output Bottom Field First (BFF) video to the TV, I load a nifty Directshow filter called DivXG400 which converts BFF to TFF on the fly. If it can be done on the fly, then it can't be a difficult thing to do for CCE as it encodes.


Well, it actually IS a complicated operation, and I can't possibly see how CCE could be doing it correctly.

If the source DV file is BFF, and you leave the "Upper Field First" box unchecked in CCE, the resulting .m2v has TFF flags set. So what did it do?

The INCORRECT way is to swap the spatial order of the fields in each frame and then set the TFF flag. This is WRONG! The .m2v TFF flag controls temporal order, not spatial order, so swapping the data in the frame and setting the TFF flag are NOT inverses of each other.

The only possible CORRECT way, is to take the lower field of frame N and re-pair it with the upper field of frame N+1, and set the TFF flags. This works, but there is now a single field at the front of the file that has to be dropped.

Any way you look at it ... if you want your whole file encoded (all fields) from BFF DV, then the resulting .m2v MUST have TFF flags clear.

By the way, as far as your graphics card goes, it doesn't make sense that it can't output BFF video. The notion of top or bottom field first only exists in the digital realm, where codecs group fields into frames. An analog signal has no notion of which field is first ... the fields come one after the other with their alternating spatial polarity, but no single field is "grouped" with either the previous or the next field. If your graphics card has trouble with a certain field order, that is a software/driver problem, but nothing wrong with the hardware.

[RB]

Originally posted by SomeJoe
Any way you look at it ... if you want your whole file encoded (all fields) from BFF DV, then the resulting .m2v MUST have TFF flags clear.

Not necessarily. CCE always sets the TFF flag, whether or not "Upper Field First" is checked. When you check it however, what it simply does is shifting each frame up by one line. Not the best way to convert bff->tff, but it also works.

[Swan]

@SomeJoe
By the way, as far as your graphics card goes, it doesn't make sense that it can't output BFF video. The notion of top or bottom field first only exists in the digital realm,
OK, my phrasing was poor. Instead of saying "my graphics card doesn't support TV-Out with BFF..." I should have said "my graphics card's drivers doesn't correctly support TV-Out with BFF".
But that's nitpicking. The essence of what I am saying is the same: with a Matrox graphics card in the "G"-series, it is impossible to correctly output BFF material to the TV, unless one loads the DirectShowfilter "DivXG400" which switches field order on the fly.
The setting to enable is "shift image by one row", BTW.
And since DivXG400 can convert BFF to TFF on the fly, I personally conclude that a BFF to TFF conversion does not have to be a complicated thing to achieve.
You may of course draw whatever conclusions you want about the complexity of the task.

If the source DV file is BFF, and you leave the "Upper Field First" box unchecked in CCE, the resulting .m2v has TFF flags set. So what did it do?
Well, my experiments showed that CCE left the field order intact.
If it didn't, a simple removal of the TFF flag in the Mpeg-2 header wouldn't work.

After encoding a BFF file in CCE, with "Upper Field First" deselected, run your file through Bitrate Viewer. Then you'll see that CCE has set the flag to TFF. Then run the file through Restream (or any of the other tools we've talked about) and check the file in Bitrate Viewer again. You'll see the offending TFF flag is gone with the wind. And the file has the correct field order. I've created several SVCDs and encoded quite a few DVD compliant Mpeg-2 files from interlaced material in CCE using the method I described with removal of the TFF flag and all I can say is that it works. I don't know why it doesn't work for you.

In the scenario you describe (leave the "Upper Field First" box unchecked in CCE), CCE outputs an Mpeg-2 file which is BFF (just like the original DV avi file that was use as the source), but CCE has incorrectly set the flag in the Mpeg-2 header to "Top Field First". This is what leads to the field order problems when played back to an interlaced display (TV). The Mpeg decoder reads "Top Field First" and outputs the video Top Field First. But in fact, the video is Bottom Field First and voila! Field order is wrong! By running the video through Restream/Pulldown/Darim Mpeg Changer and removing the TFF flag, the Mpeg-2 decoder reads the header and sees "Bottom Field First" and outputs it that way, which is correct for DV sources.
And the field order is right again.

The only possible CORRECT way, is to take the lower field of frame N and re-pair it with the upper field of frame N+1, and set the TFF flags. This works, but there is now a single field at the front of the file that has to be
I don't know the mechanics behind CCE's conversion from BFF to TFF. All I can tell you is what I've seen with my two eyes. It works, it looks every bit as good as if I encode the exact same source file as BFF (deselecting "Upper Field First" and removing the TFF flag with Restream).

[bobcross]

Hi.
I have the same problem as max2K using TMPGenc.
I mean the result of encoding on the Tv is: "when there is movement - as in when the camera is sweeping from left-to-right ( panning ) in a scene - the end result is not smooth - its as if something is lost in between".
I use AVI2SVCD and I'd like to what can I do to obtain better results, step by step and, notably how I have to modify settings?
Thanks

[Swan]

@bobcross
Your problem is probably that that field order is not correctly set in TMPGEnc.
What is your source? DV?
Then it's Bottom Field First (BFF).
When the field order is wrong, it shows on everything but stills, it looks so odd it can't be mistaken for anything else. Once you've seen it, you identify it immediately the next time you see it.
If you only have a problem on pans, it may be something else that's wrong with your video.

But to avoid field order problems in TMPGEnc, just open the "Advanced" tab of TMPGEnc, under "Video Source Setting", make sure that you set "Field Order" to "Bottom Field First (field b)".
I've made it a habit to double check this setting before encoding.
TMPGenc's wizard usually gets it right, but even so, it's a good practise to also double check when using the wizard.
Unlike CCE, when setting the field order in TMPGEnc, one is telling TMPGenc what the source has (BFF or TFF). TMPGEnc can not convert field order, like CCE can.

[vidiot]

...i don´t like to see myself repeating the same thing over and over,
...but,...

could everyone with the same problem described by max2k and bobcross try to do the same thing I did? And no matter what the experience is:
post it please!

When I ran into the situation (Children on a swing @ a nice frequenzy wiht a little panning from camera)
:) ... that was the only thing that helped!

And I did try every fieldorder setting available in BOTH encoders ,
burning them to SVCD and made a comparison on tv...
-> In the end some settings worked better -> some not.
But nothing worked perfekt until i tried what I´ve said here before.

And yes - your guess is right - I don´t know why it worked then...:devil: -> but who cares for that if the video looks good?

kind regards
Harald

Edit:
Just saw this thread:
http://forum.doom9.org/showthread.php?threadid=53369

have to try it...

[bobcross]

It works!
Effectively I solved the problem. Thanks Swan.
Now movement scenes are clear and the SVCD quality is acceptable.
Since sometimes there are some "digitalise effect" in the images (notably with face's images), I'd like to obtain better results in quality of the movies.
May be I have to try and re-try different settings, so if anyone has any suggestion ... thanks.

P.S. The sharpness of the SVCD seems to be excessive. Could anyone suggest anythin to this matter?

[Swan]

It works!
Great!! :D

Since sometimes there are some "digitalise effect" in the images (notably with face's images), I'd like to obtain better results in quality of the movies.
Blocking? Ugly squares?
If so, in TMPGEnc (that's what you used, right?), one setting that does wonders is changing the default setting under "Motion Search Precision", which is "Motion Estimate Search (Fast)". Set it to "Highest Quality (Very Slow)". It will increase encoding time, but significantly increase the picture quality too. The setting's located under the "Video" tab.
P.S. The sharpness of the SVCD seems to be excessive. Could anyone suggest anythin to this matter?
English isn't my natural language, but are you saying the picture is too sharp? How can it be too sharp? Isn't sharpness a nice thing? ;)

/Maria