Motion Problems Encoding AVI Capture with HC Encoder

[rcubed]

Hi,
I decided to post this under the topic rather than as an addition to the HC Encoder thread. I hope that is ok.

Recently I have been interested in learning to use AviSynth and HC to encode AVI files captured from brodcast TV from cable with a Canopus AVDC100. (NTSC)

Previously I have been using either TMPGEncExpress3 or Procoder Express to do the encoding.

I have noticed that when I do an encode with HC Encoder, I get what I would call jumpy and or ghosted images when played back on my DVD player. I have tried feeding the output from the same AVS script to both the TMPGEnc3 Express and Procoder Express and do not experience this problem. This problem seems to occur on frames that contain large amounts of switching or motion. When the encoded video is viewed with Virtual Dub Mod on a frame by frame version, the frames look similar between the three encoders. (One of the differences may be in whether or not the encoder handles it as a P, B or I frame speculation on my part).

See attachment for an example of the AVI source frame sequence that causes problems. I have resampled the original frame capture (to meet the forum size rules) and I hope the images are viewable. (Note I had originally posted this under the HC thread but deleted it, the website kept insisting that the attachements (previously pngs) had be uploaded on the HC thread even though I had deleted the post :confused: so I converted them to jpgs - hope the quality is ok).

How do I correct or handle this with HC Encoder. I have tried open and closed GOPs and with Scene Change selected. Both the MPEG and MPEGSTD matrix exhibit the problem. Does HC have the motion seach capabilities of TMPGEnc or Procoder Express?

Thanks in advance

Rcubed

[manono]

Hi-

That's interlacing, of course, and if you have to get rid of it, then you either IVTC it or you deinterlace it, depending on what the source is like. However, there's nothing that says you should try and get rid of it. It was broadcast like that, and if encoded like that, it'll play back fine on your TV set (usually). If it can be IVTC'd, I usually do it, if for no other reason than compression efficiency.

If TMPGEnc and Procoder got rid of it, then most likely you turned on some kind of a deinterlacer in the encoder, purposely or not. HCEnc won't do anything to it that's not in the script, so, in the sense of not doing anything behind your back, it's by far the best encoder of the 3.

If you want advice on how to handle it in the script, then maybe upload a 10-15 second piece of the untouched source showing motion. However, that pic I downloaded is 653x338, which is a very strange resolution for an AVI. If it's been resized without taking into account the interlacing, it's going to be hard to do much, if anything, with it.

[rcubed]

Manono,
Thanks for your reply. I had cropped the frame captures to stay under the upload limit size for the files. Hence the weird size. The original frames are 720x480.

When encoding with HC without any corrections there are artifacts produced which are very noticeable on the TV. They are not present when viewed on the computer - due to how the software players handle interlaced video if I understand correctly.

A 15 second portion of the capture is about 56M. How can I get that to you? I'm not sure how to handle that. My AVDC produces about 1.2G for 5 minutes of capture of std NTSC.

Under TMPGEnc I use the std option which is deinterlace only if necessary. Procoder Express does what it wants, there are no options to select with it. It's the stripped down version of the full version of Procoder.

Most of the frame editing to remove commercials etc. I do with Virtual Dub and then do a direct stream copy to generate the input avi for encoding. I have several large hard drives in my PC to handle it and speed up processing.

Here is the AVS script I was using:

AVISource("H:\Test.avi")
KillAudio()
SeparateFields()
crop(8,21,-8,-21)
LanczosResize(704,240)
Weave()
AddBorders(8,0,8,0)
ConvertToYV12

(Please be gentle I'm just starting out with AviSynth. :o). I have evaluating several Resizers. The LanczosResize is one of the test cases. The cropping is to get to something approaching a 16:9 format with allowances for overscan on the TV. With HC I was selecting the 16:9 format for encoding. I found the KillAudio was necessary to allow the script to be tested with a player such as Win Media Player without intermittent bursts of random pixels. Normally I adjust the audio level before authoring with GoldWave and then input that audio track at authoring time, so killing the audio is not a problem.

I agree the HC encoder does a really good job. I use DVD-Rebuilder Pro using the HC encoder and have noticed increased detail over Shrink even using deep analysis, and is one of the reasons I'm interested in using it.

Thanks for your reply and anyother help and advance. :thanks:

rcubed

[manono]

Hi-

Yikes, 56 MB. Well, make it 10 seconds, and strip out the audio. You can upload to one of the sites mentioned in the sticky in the General Discussion Forum:

http://forum.doom9.org/showthread.php?t=96362

and I like these guys :

http://www.mediafire.com/

After the site you choose gives you the link, then paste it here.

Aren't crops, even after the fields have been separated, supposed to be in multiples of 2? In what colorspace are your caps? I'm pretty sure that's true if they're YV12. This could conceivably be the cause of your problem, as HCEnc is the only one that requires YV12 input, and TMPGEnc uses RGB24 and Procoder YUY2, I think. You might do a short encode in HCEnc, making sure to crop in multiples of 2, and see if it comes out any better looking.

We can discuss the rest of the script later, unless someone else wants to tackle the subject now.

[rcubed]

Manono,
I uploaded a sample of the AVI to mediafile here is the link

http://www.mediafire.com/?9ddd0lglmmz

The file is zipped that reduced the file from about 35 to 31M.

I was never clear on the modulo rules for cropping, do they apply to the resultant dimensions after cropping, or to each of the crop values?

I believe the AVI file is RGB. For sure it is interlaced. 420/2 =240-21-21=198 which is a multiple of 2 (if the modulo number applies to the resultant file). Just for sure I tried it with a crop of 20,-20 and the same artifacts are there. In the sample Frames 20, 206, and 270 show the "overlapped" images. Also when encoded and viewed on the TV the portion where the guy is walking across the frame gives a jerky result. The cited frames give a very short ghost, or flicker. As you said probably do to interlace.

There is graininess in the capture, the cable has that on occasion. It's a local station (60 mi to south) over hilly country. I'm not sure how the cable company gets the feed (I suspect they pull it off the air and put it on the cable). Typically the movie channels are a lot cleaner. I'm not sure if the cable company does some in digital, and others in analog.

Let me know what you find out/decide. I really appreciate you taking the time to look into this for me and helping a newbie. :thanks:

Thanks again,

rcubed (cuber for short)

[manono]

Stick Info() at the bottom of your script (before any ConvertTo) to find the colorspace of the initial cap. It applies to each of the crop values:
In YV12: width mod-2 height mod-2 if video is progressive height mod-4 if video is interlaced
http://avisynth.org/mediawiki/Crop

If the source is RGB and you only convert to YV12 at the end, you're probably OK. But I'll bow to those that actually know something about this subject. Looking at the sample now...

[manono]

Hi again-

The source is DV, Bottom Field First, YV12, and it's telecined film. So, you'll want to IVTC it back to progressive 23.976fps. Then you can crop and resize to your heart's content with no problems at all (still making sure each crop value is a multiple of 2). And, for HC, you won't even have to add the ConvertToYV12() at the end.

To tell the source was film and it's been telecined, you open it in VDub(Mod), scroll to a place with movement, and start advancing a frame at a time. If, in every 5 frame sequence, you see 3 progressive frames and 2 interlaced frames, it's been telecined and can be IVTC'd. If every frame is interlaced, then it was shot on video and is truly interlaced material. Telecined material can be IVTC'd. AviSynth has several good IVTCs. If you use TIVTC, then adding a simple:

TFM().TDecimate()

will work wonders for you. Also, if cutting ads, you might try and cut on a cycle border, like after frames ending with 4 or 9. That way the cadence doesn't change or break.

[rcubed]

Manono,
Thank you very much for the help. I tried your suggestion and that did work miracles. Looks like I have a lot of reading to do. I'm assuming that TMPGEnc does a lot of this under the covers?

The avi file is RGB24. I haven't had the opportunity to read through all of the filters. The info() supplied that information. Thanks for the mention.

Looking I at the Avisynth External Filters document, the TIVTC filters require a YU12 or YUY2 colorspace. Is there a preference for ordering the various filters (conditional on what color space the various filters might require). Again please excuse a possible dumb question from a Newbie about the filter orders. When I moved the ConvertToYV12 to the top and had the crops at 21,-21 Avisynth will complain the crops aren't a multiple of 2.

I have to confess I'm from the really old school that likes documentation on paper. I find sticking fingers between physical pages or inserting book marks or Post-its easier than looking at stuff on a monitor (it's an "Old F-rts" thing). I've been taking the Avisynth documents on syntax, internal and external filters and producing PDF documents that I can print out (import to word, format, convert to PDF). However, the one thing that electronic documents have over paper ones is the ability to rapidly search the document for keywords I really like that. This way I have the best of both worlds. I also buy a lot of paper and use a lot of toner or ink cartridges. The manuals are a lot of sheets :) I'm still working printing out the full set. BTW kudos to the people who have been working on putting the info on the Avisynth website:thanks:

It's getting late here in upstate NY so I think I'll try to get to bed, and really dig into it tomorrow. I'm a retired computer engineer (did logic design and CPU micro programming) so I have a lot of free time these days. :) :) :) Of course the wife has other ideas on how I should be spending my time. :p

Again I really appreciate your time and effort in helping. :thanks:

I've found the forums on the web extremely helpful and saving time. People are usually most helpful and I appreciate that. The only problem is finding the right forum. Doom9 rates very high on my list and is toward the top in my bookmarks.

rcubed (cuber)

[manono]

I'm assuming that TMPGEnc does a lot of this under the covers?
I doubt very much that it IVTC'd anything for you. Maybe it deinterlaced it if you set it up for progressive output, I don't know. I don't use TMPGEnc and you couldn't pay me to use it.
The avi file is RGB24
Which? Not the one you uploaded. Maybe something you fed into TMPGEnc?
Is there a preference for ordering the various filters?
If you're going to IVTC (or deinterlace), ordinarily you do that first. Afterwards you're working with progressive video, much easier.

[rcubed]

Manono,
I slept too long, hence the long time in responding. Bad night.

After thinking about it I agree about TMPGEnc.

About the colorspace I believe the AVI file I uploaded is RGB24.

I downloaded the file from mediafire to check what it had in it. I did a cmd line "fc" (file compare) with the original sample.avi. They were the same.

Here is a capture (DownloadedSample.jpg) of the downloaded unzipped file being played back with Media Player Classic using the following AVS script :

AVISource("E:\Documents and Settings\rcubed\Desktop\Sample.avi")
KillAudio()
Info()

Here is a capture (Downloadedwithfilters.jpg) of the downloaded AVI sampled with the following script:

AVISource("E:\Documents and Settings\rcubed\Desktop\Sample.avi")
KillAudio()
ConvertToYV12
tfm().tdecimate()
Info()

If I understand what you are saying the following script would be more appropriate ( hopefully not :confused):

AVISource("H:\Test.avi")
KillAudio()
ConvertToYV12
tfm().tdecimate()
crop(8,42,-8,-42)
LanczosResize(704,480)
AddBorders(8,0,8,0)

then feed that into HC with 16:9 and 3:2 pulldown encode options?

Thanks again.

rcubed

[manono]

OK, I'll take your word for it. I'm using Cedocida to decode it. Maybe it's changing it to YV12, I don't know. Or maybe your decoder is changing it to RGB. Check your decoder settings. Others know more about the subject than do I.

Do all filtering in the original colorspace, if possible. If not possible, then while it's still interlaced, (I think) it should be:

ConvertToYV12(Interlaced=True)
then feed that into HC with 16:9 and 3:2 pulldown encode options?
Yes.

[45tripp]

The source is DV, Bottom Field First, YV12, and it's telecined film.
About the colorspace I believe the AVI file I uploaded is RGB24.


It's neither.
For NTSC DV, chroma subsampling is 4:1:1

Stick Info() at the bottom of your script (before any ConvertTo) to find the colorspace of the initial cap.
It won't.
It'll give you the colourspace of what the decoder is outputing to avisynth.
With avisource() you have the vfw decoder working.
Avisynth by default requests YV12.
Most of the vfw dv decoders can't output yv12.
Cedocida and ffdshow can.

I see no reason to use anything else than Cedocida.
Litrature indicates so, and so did the few tests i've done.
Supposedly ffdshow is equally good, but i've been unable to replicate.
Or use directshow.

As for Tmpg,
you'd have to be more specific about the settings you used, and the result you received.
It can do the task.
How well? Very, I'd dare say.
How slow? That's the pertinent question.

gl

[manono]

Others know more about the subject than do I.
And you're one of them. Thanks for setting me straight, Immersion.

[45tripp]

Not even the army sets people straight anymore :devil:

you're welcome

[rcubed]

Immersion thanks for your comments. Not having been in the army, but having to deal with the armed forces as a military industrial contractor engineering employee (before retirement) seems about right (not to mention management at the workplace in some cases).

I'm not sure how to control pulling individual part of the message for responses, so I'll just do them one at time.

As a question how did you determine the chroma subsampling?

I have been trying to investigate the codec issue using Gspot. The normal DV codec I have installed is Panasonic DV. If one feeds the script with just the Souce input and killaudio, GSpot shows (src)-->>--(A)-->[Color Space Converter].--(B)-->[Video Renderer] where
(A) is MEDIASUBTYPE_RGB24,
(B) is MEDIASUBTYPE_RGB32
and the converter and video renderer are quartz.dll

If you add a ConvertToYV12 the script the sequence is
(src)-->>--(A)-->[AVI Decompressor].--(B)-->[Colore Space Converter}.-->[Video Renderer]
(A) is MEDIASUBTYPE_YV12
(B) is MEDIASUBTYPE_RGB32
(C) is MEDIASUBTYPE_RGB32
and the AVI Decompressor, Color Space Converter and VIDEO Rendere are again quartz.dll

I you just play the avi file "as is" there are two cases that show up one used the Panasonic Codec and the other uses an microsoft (supplied with XP). I'm still not sure what happens when you feed avs scripts to one of the other video players. Until know I have only been interested that it works (yea I know shame on me).

I use a program called AVI_IO and it needed the Panasonic DV codec to be able to capture with the ADVC100. I tried using other capture programs and all of them suffered from dropped frames. I've never had AVI_IO drop any. Plus it segments the AVI into sizes you can control not just one big AVI file, it will use multiple partitions (as specified in the setup) if the file won't fit on one partition (could also be drives), and VituralDub will read in the entire sequence because of the way they are numbered.

The Cedocida codec looks interesting and I may give it a try after I review how one gets them back out, if it causes problems with the captures. I'm hoping it may work with AVI_IO. There are no properties that one can select/adjust with the Panasonic one.

The results are very good with TMPGEnc. Basically I would resize using Lanczos-3. The Deinterlace parameters were set to Deinterlace only when necessary, (this greys out Deinterlace by top field which I assume has disabled that), and Interpolation-adjusted, Output: NTSC, prioritize quality, 720x480, Image 16:9, VBR, 9 bit interlaced, motion search std or high.

Slow, yes. I haven't done anything with it for a while, but I think 42 min of video was taking something between 2 and 3 hours. (I'll have to run one to be sure. I am currently running an AMD Athalon64 X2 4800+) with 2G of memory. The previous machine was an AMD Athalon XP 2800+ with 1G.

Procoder Express seemed to be about the same quality as TMPGEnc as described, but about 2/3 of the time that TMPGEnc took, However when the TV captures started looking like letterboxed 16:9. I started messing with cropping and encoding as 16:9. I have a 16:9 TV. One of the main things that bugged me about Procoder Express is that apparently unless the input is 720xsomething, and you try to produce a 16:9 output, it will letterbox it down, with black borders on the top and sides. By the time you messed around with VirtualDub to crop and resize the input to get what you want for Procoder, you had to do full processing as opposed to direct stream copy which does two things, about a 3x increase in AVI file size using either Huffy, or Lagarith for compression and reduced the processing frame rate to about 40 fps or slightly more.

After getting inputs from manono and playing with AviSynth I may go back and mess with the Deinterlace options etc in TMPGEnc just for fun.

From what I have learned in this discussion and in looking into it, one can input to either TMPGEnc or Procoder Express with a AviSynth script. Which simplifies the process and doesn't seem to increase the processing time. Not to mention the 42 minute captures end up being 27G or so for the Procoder input case. You can imagine what a 1.5 to 2hr movie requires when one crops and then feeds into Procoder - very careful disk (or partition) management.

Now that I am "into it more" I may consider messing with the other filters in TMPGEnc, but wasn't impressed enough with the results especially since the filters added significantly increased processing time (last time I tried it was with my 2800+).

HC coupled with AviSynth allows one to get a better understanding of what to do, how it's doing it, and a lot of options in processing which Procoder has very little options, and the documentation with TMPGEnc is a little confusing.

Again thanks to both of you for useful insights and help

rcubed (cuber)

[45tripp]

As a question how did you determine the chroma subsampling?

nothing to determine. it's a standard.
it's ntsc dv, therefore it's 4:1:1
http://en.wikipedia.org/wiki/DV


I you just play the avi file "as is" there are two cases that show up one used the Panasonic Codec and the other uses an microsoft (supplied with XP).
one vfw route the other directshow.

I use a program called AVI_IO
don't know it.

you want cedocida as your decoder.
switching back is as easy as changing a single registry value.


After getting inputs from manono and playing with AviSynth I may go back and mess with the Deinterlace options etc in TMPGEnc just for fun.
Once you start using avisynth, there's no turning back


HC coupled with AviSynth allows one to get a better understanding of what to do, how it's doing it, and a lot of options in processing which Procoder has very little options, and the documentation with TMPGEnc is a little confusing.

You can use any of the above encoders with avisynth.
All filtering should be done in avisynth and you're left with the encoding engines.
All three are very good, as you already can tell. Tmpg can be very slow though.

You mention vdub. You can frameserve with vdub too. You can also directly import the vdr signpost into tmpg. With the other two more tricks are needed. Like frameserving with vdub, loading the vdr signpost into avisynth, and frameserving with avisynth.

I don't really understand what problems you were having with the other encoders, but you shouldn't be having any.

gl

[rcubed]

Immerson,
I didn't see your post until late/early today (8/17 1:00 am)


I'm not having any problems with the other encoders I started getting interested in AviSynth and HC encoder jumped out as a freebie, from my experience with using DVD Rebuilder Pro. After I got into it, I realized one can feed the script into the other two encoders I mentioned. You are right about no turning back!! I still have to do some investigations on what Procoder Express allows and what you can "fiddle" with. Normally I had avi interlaced input and didn't mess with any input parameters and all of it used to be 4:3, now stuff is showing up that looks like letterboxed 16:9. Still don't know if Procoder is doing any internally or not. I does look like it handles the telecined material, not sure how the encoder handles it, whatever is does it seems to do it well. The same for TMPGEnc, but it is slooooowwwww.

I am trying the Cedocida codec (new version 0.2.0 is out - thank goodness someone complied the source, I tried by installing VC++ 2005 Express, and NASM, but had too many problems trying to do it.:confused: Kudo's to whoever did the job: thanks:), I found a program called vcswap that allows you to swap the two codecs easily. XP does a good job of removing the codec using the Control Panel, Sounds and Audio Devices etc.. path, but one doesn't want to remove it, just disable it. The advantage of vcswap is that it doesn't remove the code, it just deactivates it. I do mess with the Registry on occasion, however, with my typing skills it's a risk. Vcswap just takes clicks and/or drags to do the job. I'm going to have to take a look at the Registry and see if it is doing what I would have done to swap them. The Cedocida codec works ok with my capture program. I'm playing with each. Looks like Manono has Cedocida set up to output YV12 is why he thought the source was YV12 and not RGB.

On the issue of IVTC do you have an opinion on what one to use, any favorites? I've been playing with Decomb, and TFM. From the acknowledgments at the end of the Decomb Reference Manual, Manono was heavily involved with that one.

Again thanks to you and Manono for all the help. My wife is going to have to be dragging me away from my computer and telling me to get off to bed now that there is no turning back.:p

rcubed

[45tripp]

i encoded your sample for fun

avisource("sample.avi",false)
tfm().tdecimate()
crop(10,60,-10,-60)
FFT3DGPU(sigma=3.0, bt=3, bh=32 ,bw=32)
lanczos4resize(704,480)
tweak(cont=0.85, hue=1.5)

http://www.wikiupload.com/download_page.php?id=200361

[rcubed]

Immerson,
Wow :eek: that sure seems to clean up a lot of the noise. I get the idea that somehow that filter is using the processors on the graphics card to do it's thing? :cool:

I unfortunately had stopped printing my AviSynth External Filters document (pdf) at page 171. Just by coincidence FFT3DGPU started on page 172. I compared the HTML file that came with the DL to the one in the external filters document and the file that came with the filter is more recent. I'll convert that one and put it in my book. Looks like I have to do a lot of reading to really understand what/how this is doing. I have an ATI X1800 graphics card in an Asus A8N-SLI Premium mobo the script seems to run pretty fast. (about 16-17 fps average) :cool:

One question I assume you have the Cedocida codec on your machine (what version) and you are using that for AVI files. Also I am assuming you fed the sample.avi file that I uploaded as the input to the script w/o any other conversions. What settings do you have set for the Cedocida codec. I am also assuming that it is doing the conversion from RGB24 to YV12. I have been playing with Cedocida 0.2.0 and although I can get it to input YV12 to the script, the colors are banded etc. when I try to encode. I had to put a ConvertToYV12 at the beginning of the script to get it to work and drop back to my Panasonic codec. In the apps notes there are some words about selecting MPEG 2 but selecting either of them blows out of the script (or using the defaults).

NOTE I just put version 0.1.7 in instead of 0.2.0 and the 0.1.7 version works ok with the defaults selected. Looks like there might be some kind of bug in 0.2.0 or I'm overlooking something. I thought only $MS messed you up with updates :p

Again I AM IMPRESSED. You've given me something to really "chew" on. The more I get into this stuff the more I realize I don't know a lot and there is a lot to learn. Again thanks, I have to toddle off to bed before I fall face down into my keyboard:)

Thanks,

rcubed

[rcubed]

Immersion and others,
Neruon2 has posted a new version of the Cedocida Codec 0.2.0 release that fixes the problem I was experiencing. See the DV forum for the link.

rcubed

[45tripp]

I get the idea that somehow that filter is using the processors on the graphics card to do it's thing?
yes
as a speed up to fft3dfilter.


Just by coincidence FFT3DGPU started on page 172
i'm all about coincidence


One question I assume you have the Cedocida codec on your machine (what version) and you are using that for AVI files. Also I am assuming you fed the sample.avi file that I uploaded as the input to the script w/o any other conversions. What settings do you have set for the Cedocida codec. I am also assuming that it is doing the conversion from RGB24 to YV12.
coversion from 4:1:1 to yv12
yes to all the other stuff.
setting yv12 sampling to interlaced.


I unfortunately had stopped printing my AviSynth External Filters document (pdf) at page 171.

you're really getting into this i see.
I'm starting to side with your wife :devil:

Immersion and others,
Neruon2 has posted a new version of the Cedocida Codec 0.2.0 release that fixes the problem I was experiencing. See the DV forum for the link.
i used 0.17.
tried 0.20 after your post and was having troubles of my own.
i'll try the new build later.

gl