Macroblocks in Water of mpeg-2 encode

[-y]

Background Info:

MiniDV cam Sony DCR TRV17 (500 lines)

My PC:

Dual P3 933Mhz
1 GB of PC800 RDRAM
Supermicro P3DR3 Mobo i840 chipset
35GB Ultra 160 SCSI HD
75GB Ultra 160 SCSI HD
Geforce 3
SB live 5.1
Hauppauge DV Wizard Pro Firewire Card

Encoding Software:
Premiere 6.02
CCE 2.62
TMPGEnc Plus v2.55

I have captured a DV clip of a swim meet in a swiming pool.
The capture video quality is great.

I have exported the movie in premiere using no filters and uncompressed at 720X480 (huge file)

Now when I encode using CCE with 9pass VBR Max bitrate = 9000 Ave = 7000 and Min = 4000. I used the settings in the CCE guide on this website. The mpeg movie has some blocks around the swimers.

The same happens with 2 pass in TMPGEnc.

Any help to get rid of these blocks???

Thanks

[TRILIGHT]

MiniDV technology is MPEG2 compression. This compression occurs within the camera. It's at a much higher bitrate and you are using CCE so I would assume that you would not see such artifacts. However, it also depends on the quality of the card you are using to capture it. I looked at the specs of the Hauppauge card but could not find any real details on whether it does any YUV->RGB conversion or not. Also could not find any details on the chipset or true technical specs; just the usual marketing stuff. Another thing you should check and be sure of is that you're not recompressing when you export from Premiere. Check your export settings and make certain you do not have a checkmark on "Recompress". This will add futher artifacts to the already compressed video.

I've never personally worked with MiniDV so I can't say for sure but usually compression of video that was already compressed at one point, will always introduce some sort of artifacts. The bitrate on MiniDV is so high, I just don't see how it would matter that much with CCE. I'm not sure. If I had to guess, I would say your problem is somewhere in the capture or editing stage which causes the video to be slightly more difficult to compress in the final stage. Of course, water, especially in motion, is always a very difficult piece to compress well. Sorry I couldn't be of more help. Perhaps someone with more MiniDV experience can assist you.

[-y]

Thanks for your reply. I have done a lot of searching and found out that moving water is very high motion and I am going to try mb1's matrix and a few GOP tweeks. I made sure the recompress is not ckecked in premere. How would YUV to RGB cause this blocky problem? or my capture card? I have tried many capture programs i.e. motoDV, iuvcr etc. and I have tried to compres the DV.avi file straight to MPEG-2. I have tried to tweek both CCE and TMPGEnc in every way possible. I going to figure this out some day.

Thanks

[taudule]

The only way to really avoid blocks is to deinterlace your video.
I tried the mb1 tricks, and with SVCD bitrate, all that i've got is blurred video.
Try using GreedyHMA in an avs script, and resize so you can add 16 pixel border. Then encode with tmgenc CBR 2520, standard SVCD, High quality.
Just try...
.

[mb1]

Of course miniDV cannot be compressed to svcd bitrates with quality. taudule, for that you should give 1/2 D1 a try.
My matrix is developed for high bitrates, not for svcd bitrates.
Best is always to increase bitrate - if your player allows (or decrease resolution).

@ -y
First don´t do 9pass with cce. Max should be 3pass for dvd bitrates (maybe up to 5 for low svcd bitrates) as beyond that point encoding quality reaches its limit and drops down again.

Use tv overscan (and macroblock) optimization to save bitrate for visible area.

For dv encoding I always achieved better results with tmpeg in 'highest quality' for motion search precision (of course 2pass vbr).
That takes a long time but especially on your dual p3 should work well (as tmpeg is optimized for dual processing if set in the options).

Go up to the maximum dvd bitrates (9.8 Mbps) with max setting.


If the problem exists after following these hints please upload some seconds of the native dv stream anywhere if possible (maybe 10 - 20 seconds).

Don´t deinterlace (f.e. with greedyhma) as you will loose smooth movings and especially sharpness in your sports video.

[-y]

Mb1 thank you for your suggestions I am going to try to use them tonight.
I think your right about TMPGEnc; it is better at DV than CCE I have searched every forum on VCDhelp and Doom9 about very high motion and DV to Mpeg-2 DVD. I am not very web savvy but the DV file is only 8 sec long and 31MB in size. (This is just my test clip to get my settings right before I encode the whole 1 hr movie.) I do not have broadband yet and would not know how to upload the 31mb file. I have Media cleaner 5 and uncompressed the DV file and used the adaptive Deinterlace filter. It helped, but now the video is not as sharp. I know there is a solution for this complex video I will not give up until the MPEG-2 file looks as good as the DV file.


One more question MB1 what is TV overscan? (Is this croping or resizing the video to 704 x 480?) Also what is macroblock optimization to save bitrate for visible area? How do I do these procedures?

Thanks

[mb1]

TV sets are based on analog technique. The displayed picture changes from cold picture tubes to warm picture tubes.

Therefore technicians decided to introduce an overscan area. The picture always goes beyond the visible borders (if it follows the tv standards).

Tests have shown that tv overscan area is around 6-10% of the whole picture.
This lets you save the same amount of bitrate for equal quality or improves your quality for this amount.
The next thing is: you can´t see the whole recorded picture (look at the border scenes of your dv.avi on pc and then compare the visible picture if you connect your cam to the tv set and play it there).

So we do a little interlaced resizing to bring the whole picture to the visible area of your tv set and fill the borders with pure black (which needs almost no bitrate).

Macroblock optimization means firstly to bring the pure picture (coded film pixel) which is the area where a picture can be seen (for dv this is almost the whole area of f.e. 720x480 pixel except maybe a black pixel line or more on the border) into multiples of 32.
Mpeg encodes with macroblocks which is a size of 16x16 pixel.

I live in PAL world where dv size is 720x576. The above optimizations allow to resize the picture to centered 688x544 and fill up with black borders with 16 pixels size around. So a lot of macroblocks are simply black and need not to be encoded.

For NTSC world this would mean 720x480 (or the pure picture, f.e. 720x478, crop black lines away) can be resized to centered 688x448 and filled up with black.

But for the sake of correct aspect ratio you have to resize correct, so you can't do 720x480 directly to 688x448.

For correct resizing values I recommend FitCD. These values then can be set f.e. in Tmpegs advanced settings under 'clip frame' and the 'arrange setting'. Video arrange method has then to be 'center (custom size)'.

But I don´t know if 448 vertically is full screen on ntsc tv sets (sure 688 horizontally is) as I cannot test that in PAL world.


Another note:
On a clip with only 8 seconds you cannot test vbr bitrate properly as the encoder has no chance to calculate variability over an appropriate movie length.
For tests you then can encode with cbr at the highest possible bitrate (to get best quality for your 8 sec clip).

A correct value for aspect ratio would be reached if you crop your 720x480 dv.avi to 718x468, then resize to 688x448 and add black borders to reach 720x480 again.

Maybe you do a little mpeg and tv theory research in the web or some forum search (good search engine here).

I can´t explain all theory in english which is not my native language.

[-y]

MB1 that info was great. I will begin my reasearch today; got a T3 line at work and a lot of free time. I live in the NTSC world I will try your resizing methods. And use CBR for the test.

Thanks Again for your Help

[24hourloop]

Do not resize vertically interlaced DV to interlaced MPEG2. It is very important to retain the same number of lines otherwise this is going to create interlacing problems (It will be very jerky). You must stay at the full resolution. (Or reduce horizontally for SVCD).

I have had the best results not with CCE but TMPGEnc out of Premiere using the Videoserver plugin. If you go to DVD (8Mb) you can get excellent quality that is as good as DV (DV is 3.6MB (notice the difference in bits and Bytes)). However, DV has only intraframe compression and is thus ultimately inferior at comparable data rates.

You will always lose some quality but at DVD data rates you will hardly notice.

[mb1]

Never had any problem with vertical resizing of interlaced material.

Be sure to do it correct:
1. don't crop or crop only even numbers (2,4,6,8,...) to obtain field order
2. separate fields
3. resize as you like
4. weave fields together

Tmpeg does it all correct if you're cropping the right way (and if you have chosen 'interlaced' for input and output).
Avisynth does it also correct if you use the right syntax.

So there are no problems in vertically resizing interlaced material.

[taudule]

@-y:
Sorry i read too fast and didnt see the bitrates you used, and i made all my tests at svcd bitrate, or close. (maybe, as you've got time, you could give a try deinterlacing ).

@mb1 :
i made many tests, and when i keep interlacing at 2520kbps, the resulting video has artifacts (edges with ghosts, blocks on fast moving scenes f.e. sea or swimming pool) and overall all occurs as if there were a heat source between my eyes and the television set.

When i deinterlaced the heat source disappears, and there are much less blocks and artifacts. This is the only method i found that imho significantly increases the result quality.

I wondered if this could not come from my player or TV, but when i whatch my video using an mpeg card (margi DVDtoGo), i get the same phenomenon.
I tried 352x576 interlaced with very low bitrate matrix, but i found the result less good than 480x576 deinterlaced. Is 352x576 half D1 ?
.

[mb1]

> Is 352x576 half D1
Yes.

> I tried 352x576 interlaced with very low bitrate matrix
Don't use matrices with values of 99 in it for interlaced material. Result will be slight colour distortions especially red shifts (maybe also your ghost edges).

Furthermore the very low bitrate matrix uses zig zag scan order. So it is developed for progressive material. Use a matrix with alternate scan order (which is optimized for interlace material).


Try that:
1. Crop your dv.avi to 712x572, resize to 336x540, add borders to 352x576 (do it all within Tmpeg for easy use; but avisynth gives better quality and speed)

2. Encode with Tmpeg 2.56 with the highest possible bitrate you want to use or your player is able to handle.
Let's assume you want standard svcd bitrate highs:
- convert your audio to 44.1 kHz with 160 kbps; video 2.558 cbr
- or convert audio to 48 kHz (not svcd standard but dvd standard) with 192 kbps; video then 2.526 cbr

3. Use also the following settings (in Tmpeg):
vbv 224, interlace, 10bit, highest quality, video source interlace, field B, 4:3 625 line PAL,
do your resizing settings under clip frame (values from 1.) chosing center (custom size); GOP 1/3/2/1/15; detect scene change; mb1 interlaced DV matrix; soften block noise 4/5

If resulting quality is to low in your eyes only increasing bitrate (f.e. to 3.500 kbps) gives improved results.
Water always is extremely difficult and askes for highest bitrates.

Please report your result.

[taudule]

Thank you mb1 for your quick reply.

I just finished encoding my test video. i did the following :
- Cropped and resized with avisynth (bicubic resize, 0.75) following your figures.
- Encoded with tmpGenc 2.54, CBR 2600 audio 44 Khz 128Kbps, with all parameters as you said.

I've no TV set right now to do the testing. I'll try at home tonight, with my standalone DVD player (which needs strict SVCD compliance).

Viewing the result on my PC with winDVD or my mpeg card, it seems the result is better than what i've had before as interlaced, but i'll tell you more when seen on my standalone.

Anyway, i think i'll go and buy some cheap XSVCD/mini DVD capable player, i don't know which one yet (advices welcome, for europe).

Regards
.

[-y]

I took the video and...

Used Media Cleaner 5.02
Uncompressed avi
24 bit RGB
720x480
Deinterlace Filter (Adaptive blend)

Then Used TMPGEnc 2.56 plus

Video tab:
Stream type: MPEG-2 Video
Size: 720X480
Aspect ratio: 4:3 Display
Frame Rate: 23.976
Rate control mode: CBR 9800kbps
Profile and level: MP@ML
Video format: NTSC
Encode Mode = Progressive
YUV format = 4:2:0
DC component precision: 10 bits
Motion search precision: Highest quality

Advanced tab:
Video Souce type: Non-interlace
Field order: Bottom (for DV)
Source aspect ratio: 4:3 Display
Video arrange method: Full Screen Keep aspect ratio

Filters:
IVTC 24 fps

GOP Structure Tab:
Number of I pictures: 1
Number of P pictures: 3
Number of B pictures: 2
Output interval of header: 1
Max GOP : 18

Quantization Matrix Tab:
Quantization matrices: MPEG standard
Output basic YCrCb: Enabled

I don't know why the progressive selection made a difference but I think it did. I also used 3:2 puldown so Scenarist won't complain.

This gave me the best results so far.

If anyone has any more ideas or changes Please Let Me Know.

I'm always seeking more advice to problems.


Thanks for all the posts.

[taudule]

@-y

I think i've got it !
The key word is BLEND.

I used to deinterlace using a simple blendfields() in avisynth and was fairly happy with the result until i discovered GreedyHMA.

By comparing frame by frame the deinterlaced video, i found that GHMA do not mix the fields when movements occurs. GreedyHMA keeps a clear view of the moving object, frame after frame.

Instead, Blend method mix the fields (please have a look Here : http://nickyguides.digital-digest.com/interlace-test.htm).

So i changed for GreedyHma. When movement occurs, objects don't blur.

But clearness has a price called "blocks" when you encode, and deinterlaced video with Blend method is less blocky than one deinterlaced with GreeedyHMA(0,0,0,0,0,0,0,0), particularly with tmpGenc, which adds blocks around fast moving objects.

@mb1

I watched the results and in fact your parameters increase the quality of the interlaced video, and we are very near to what can be obtained by deinterlacing, with the advantage of smooth movings.

Yet, i still find video deinterlaced with GreedyHMA (PAL DV source) very sharp and clean (i mean artifacts free), at SVCD bitrate.

Thank you for your time.

[Nogami]

But just to clear up a misconception, DV formats (MiniDV, DVCAM, DVCPro) aren't MPG-2 compression. MPG compression has a problem where it uses I, B, and P frames to make up a group of pictures for playback. I-frames have the least compression, however B and P frames are predictive heavily compressed frames.

DV uses the same compression on every frame of the incoming video for a constant bitrate. This also allows easy frame-accurate editing without needing to line up on I-frames, or jump to an I-frame, then decode the progressive frames to get to the correct edit position.

Snipped from a webpage:

----

The sampled video is compressed using a Discrete Cosine Transform (DCT), the same sort of compression used in motion-JPEG. However, DV's DCT allows for more local optimization (of quantizing tables) within the frame than do JPEG compressors, allowing for higher quality at the nominal 5:1 compression factor than a JPEG frame would show.

DV uses intraframe compression: Each compressed frame depends entirely on itself, and not on any data from preceding or following frames. However, it also uses adaptive interfield compression; if the compressor detects little difference between the two interlaced fields of a frame, it will compress them together, freeing up some of the "bit budget" to allow for higher overall quality. In theory, this means that static areas of images will be more accurately represented than areas with a lot of motion; in practice, this can sometimes be observed as a slight degree of "blockiness".

DV video information is carried in a nominal 25 megabit per second (Mbps) data stream. Once you add in audio, subcode (including timecode), Insert and Track Information (ITI), and error correction, the total data stream come to about 36 Mbps.

A good site on DV formats: http://www.adamwilt.com/DV-FAQ-tech.html

Don't want to start an arguement, but it's a good learning resource.

N.