Error -100 and blocky keyframe probem

[Enrico Ng]

I am processing video that I captured 640x480 @29.97 using Divx 5 at 12000 kbps

I am suffering with two problems.

Sometimes on my second pass, I sometimes get this error from Virtualdub.
"Video compression error: An unknown error occured (data may be courrupted). Error code -100"
Also, I tried encoding two videos (4 jobs) in batch mode. First time I got the error on the 4th job, second time I got the error on the 2nd job.

I notice that the video looks ok, but like every second, there is a frame that contains very large blocks. I have keyframes set to 30 so thats like every second right?
So every second there is like a flash of large blocks.


Also another problem. When I do the first pass, I usually do not include the audio to save time. Then when I include the audio on the second pass, it will not finish. Vdub will be almost done, but it will just stay there forever. I think it has something to do with the file size being different since if I leave out the audio on the second pass, it works (except when I get the error message I spoke of).

[Teegedeck]

A quick shot: sounds to me like you were using the wrong stats-file. Try it without batch processing to verify that. Edit: see also kastro68's post here (http://forum.doom9.org/showthread.php?s=&goto=lastpost&threadid=50940).

[Enrico Ng]

thanks, I did read that message already, but if I were using batch, isnt it impossible for it to use the wrong stats file (unless I made a pretty big mistake).

My bigger problem is the blocky keyframe problem. Do you have any ideas why might cause this?
Each keyframe contains huge blocks. The next few also do but they go away with each frame until it looks very good, then the next keyframe comes. So when you watch the video, its like there is a flash or pulse of blockyness every second or few seconds depending on how often the keyframes are.

[Teegedeck]

What you describe means that frames are scaled completely wrong in 2nd pass. One obvious solution would be that the stats-data XviD uses in 2nd pass belongs to a different movie with - of course - keyframes at completely different places, so I-frames only get the bitrate that would be sufficient for a P-frame instead of the considerably higher bitrate that they should get. Another answer could be a blatant mistake in your XviD-settings like strange quantizer-limitations etc. etc. If you post your detailed settings, maybe that could help finding the source of the problem? Also batch-processing would be the first source of errors that I would eliminate if I had any problems.

[Enrico Ng]

This seems to mainly happen on long encodings like an hour instead of short 1min encodings.

I am using H.263 then new modulated hq.
I've tried just sticking with h.263 for both passes.

VHQ is set to 1
Max I frame is set to 30
chroma motion is enabled
I've tried quaterpel but it makes it look like crap.
I've tried max B-frames set to -1 and 2
everything else is default

These have been done in batch jobs overnight since they are long.
I guess I'll give it a try without the batch job.

I've been trying to find the best settings, but I can't really tell the difference between using H.263 and new modulated, or just using h.263 or mpeg.

[Teegedeck]

Originally posted by Enrico Ng
I am using H.263 then new modulated hq.
I've tried just sticking with h.263 for both passes.

? This sounds a bit like you've changed something between first and second pass, what would of course be no-good.


Max I frame is set to 30

This would mean an I-frame about every second; a huge waste of bitrate but it's your choice.


I've been trying to find the best settings, but I can't really tell the difference between using H.263 and new modulated, or just using h.263 or mpeg.
Modulated HQ actually breaks MPEG4-compliance, sorry.

That's all from me for now, good luck!

[Enrico Ng]

I read that I should try H.263 for the first pass then new modulated hq in second pass.

I was using a keyframe every second for my short clips.
I tried it for this longer one to see if the blocky keyframe problem was really from the keyframes. I tried setting it to 300 also, but its the same, I just get that flash of blockyness every 10 seconds instead.

I just am trying to figure out how to fix this.

H.263 and mpeg look pretty much the same to me
Quarterpel just seeks to mess up everything, although it is not as bad on longer encodings

[Enrico Ng]

I examined this blocky keyframe problem again and I do not think it is related to batch encoding. (I think you guys refered to it as "pumping")
I made a few short 2 min clips using various settings to try to see which yields the best quality.
H.263 with trillis and B-frame
H.263 then new modulated
mpeg b-frame
all with chroma m
divx5
and one more with xvid, I forget the settings.

I notie that the blocky keyframe is also visible in these clips but the blocks are not as bug. it is still obvious if I look closely.

It is pretty much un-noticable when I use mpeg (or divx5)
So I'm guessing it might be something realted to H.263

Also, I've been spending a long time trying to compare divx5 with xvid. (I know this has been discussed alot). But it seems the best I have done is using mpeg b-frame and it basically looks the same as divx.

I read that it should be possible to make xvid encodings look much better than divx5. Is this something that can be tweaking some of the settings like VHQ, b-frames, etc, or is it something more advanced.

[manono]

Hi-

If you have trouble telling H.263 from MPEG quant, then you'll have trouble telling XviD from DivX5, since DivX5 uses H.263. However, the differences should be easily noticed. Make sure your source is high quality, and anamorphic. Perhaps you're using a Bilinear resizer. Try using Bicubic or Lanczos if that's the case. And perhaps you're using a low resolution. Try using at least 640*xxx if that's the case. Make sure you're giving it enough bitrate. Maybe do a short section at Constant Quant 2, maybe without B-Frames. And then find something that shows a lot of detail, like a woman's hair or a closeup of a man's face (live action-not anime). Then you should be able to tell the difference. I'm not saying that's the way to encode the whole thing, but to give them good enough quality to be able to tell the differences more easily.

[Enrico Ng]

oh, perhaps I have left out an important piece of information.
The source video is video that was captured using a TV capture card.
They are compressed using Divx5, interlaced 640x480.
I had everything turned off and the bitrate set to 15000.

In virutaldub, I use the "smart deinterlacer" from Graft when I recompress using XviD.

I would have captured using XviD, but I have not found a setting that does not results in many dropped frames.

[Matthaeus]

Some advice:

1.If you have an interlaced source you don't need to de-interlace it. Just check the option 'Enable interlacing' in XviD Advanced Options.
2.About the blocky key-frames:
- Set Min I-frame quantizer to 2, Max I-frame quantizer to 5
- On the Two Pass tab Set 'I-frame boost %' to 20 or 25 to get higher quality key-frames!
- Do not use key-frames at every second!

[Enrico Ng]

Originally posted by Matthaeus
Some advice:

1.If you have an interlaced source you don't need to de-interlace it. Just check the option 'Enable interlacing' in XviD Advanced Options.
2.About the blocky key-frames:
- Set Min I-frame quantizer to 2, Max I-frame quantizer to 5
- On the Two Pass tab Set 'I-frame boost %' to 20 or 25 to get higher quality key-frames!
- Do not use key-frames at every second!

I tried the option, but it does not seem to look good. does it actually deinterlace it. I usually use the smart interlace filter since it seems to do a better job and is about the same amount of slowdown.

I tried you settings and it seems to have helped. I also changed the keyframes to every 5 seconds. Its harder to see it now since its every 5 seconds, but the blocky keyframe is still there, although it is much less visible. I set keyframes to every second before to help me see the effect, also because I tried this with several small clips that are only like a minute or two.

Since I captured with Divx5, is it a better idea to use H.263?
I'd like to capture with XviD, but it seems to be too slow.

[Matthaeus]

Sorry, I thought you want to watch your movie on an MPEG-4 player device, not on a computer. 'Enable interlacing' keeps the frames interlaced and it looks better on a TV than a PC.
You mentioned that you are useing a 12000kbps Divx5 for capture. Use Quality based mode instead with Quantizer lower than or equal to 2. It will always give you high quality.
If you want to capture with Xvid: try lowering the motion search precision, disable VHQ, chroma ME, qpel, b-frames and other advanced stuffs and enable interlacing.

When Encoding:
- Use exactly the same options in both passes.
- Do not use Modulated and Modulated HQ matrix.
Use MPEG or H.263: make tests on an 1000 frames source using
1pass Quantizer 2 with each matrix, and choose the matrix which
creates smaller file (I bet on MPEG, but who knows...).
- Trellis quantization is an experimental feature and works with
only H.263 atm. It's safe to leave it unchecked.
- If you still have blocky key-frames use Min I-quant 2 and
Max I-quant 3.

If the problem still exists:
- your XviD is probably installed incorrectly:
download uManiacs latest developer build, install it and push
'Load Defaults...' in codec settings
- you have an SSE2 Pentium4 CPU: Disable SSE2 in debug tab
- you might have an old directshow filter (old ffdshow)
load your encoded video in vdub and check the key-frames (shift+right
to snap to key-frames). if they are not blocky the problem is in the
directshow filter. Download ffdshow-alpha from Sourceforge or encode
your video with DIVX or DX50 FOURCC code ('General options').

[Enrico Ng]

Originally posted by Matthaeus
[You mentioned that you are useing a 12000kbps Divx5 for capture. Use Quality based mode instead with Quantizer lower than or equal to 2. It will always give you high quality.
If you want to capture with Xvid: try lowering the motion search precision, disable VHQ, chroma ME, qpel, b-frames and other advanced stuffs and enable interlacing.


ok, I'm gonna try capturing with xvid
I'm setting quant to MPEG

now you said quality based and quantizer lower <=2
if I do qulaity based, I set a percenage 0-100%
if I do quantizer, I set the number 1-31

which one should I use.
were you refering to some other quantizer setting? which one?
there are several in the quantization tab

[Matthaeus]

No! Use Quality based only if you want to capture in Divx5 (use the quality/percentage slider)!
When you are capturing with XviD use strictly 1-pass Quantizer 2, since Quant 1 is buggy atm.

[Enrico Ng]

When I do quality based in Divx5 I get dropped frames.
I have to set it really low.

With XviD I can do Quality based at 95% without dropped frames but I get dropped frames with Quantizer set to 2 , even 3

[Matthaeus]

Hint: Quality based with XviD depends on Min P frame quantizer and Max P frame quantizer:

Average quant = MaxP - Quality/100 x (MaxP - MinP)

So if you have the default settings:
Min P quant: 2
Max P quant: 31

and Quality set to 95%, you get an average quant of: 3.45
3.45 = 31 - 0.95 x (31 - 2)

3.45 average quant not provides good quality for re-encoding. Try lowering Motion search precision and increasing Quality (if you set Min P quant to 2 and Max P quant to 12 you can change quantizer by 0.1 with every 1 % of quality, so 100%= 2.0 quant 95 % = 2.5, 90 % = 3.0 ...)

You must capture Quant <= 3.0 ! If it drops frames decrease the Motion search of the XviD codec (or increase Performance/Quality in Divx5), do not decrease quality! (A faster CPU may solve the problem too)

[Enrico Ng]

Originally posted by Matthaeus
Hint: Quality based with XviD depends on Min P frame quantizer and Max P frame quantizer:

Average quant = MaxP - Quality/100 x (MaxP - MinP)

So if you have the default settings:
Min P quant: 2
Max P quant: 31

and Quality set to 95%, you get an average quant of: 3.45
3.45 = 31 - 0.95 x (31 - 2)

3.45 average quant not provides good quality for re-encoding. Try lowering Motion search precision and increasing Quality (if you set Min P quant to 2 and Max P quant to 12 you can change quantizer by 0.1 with every 1 % of quality, so 100%= 2.0 quant 95 % = 2.5, 90 % = 3.0 ...)

You must capture Quant <= 3.0 ! If it drops frames decrease the Motion search of the XviD codec (or increase Performance/Quality in Divx5), do not decrease quality! (A faster CPU may solve the problem too)

Are you refering to XviD or Divx5 or both?
I'll give it a try. I had motion search set to 4, I'll try lowering it.
I have an Athlon (Tbird) 1400

[Matthaeus]

I don't get it! You cannot capture realtime with a T-bird?
Maybe your Hard Drive is slow or its DMA is disabled?

I was refering to XviD when I wrote about how to calculate quantizers but
the quantizer <=3.0 rule is true for DivX too.

If you want to go for maximum capturing speed set I frame interval to 1, or disable Motion search (only key-frames, no delta frames).

You can get more speed with discarding every other line of the source and capture in 720x288 or 720x240 (Instead in 720x576 or 720x480). It will be faster and you don't need to deinterlace it because of the halved Y resolution. After capturing you can simply resize it back with a Bicubic filter to its original size. You will lose some quality of course.

If it drops frames too, use a capture codec like PicVideo or Aware MotionWavlets. They are faster than XviD.

I don't have any good idea.

[Enrico Ng]

with everything shut off, I can capture with Quant 2.9 without dropping frames. I have not tried this with a really long capture yet.