5.1.1 - nasty (macro?)blocking

[allynm]

i've noticed that encodes with divx 5.1.1 get some very nasty blocking. this happens in dark scenes, also in scenes where there was blue lighting involved, and also after scene changes. this is not just a small area either, i've seen it effect the entire scene. it almost appears as if a scene change is not being handled properly in some cases. here are some examples:
star trek motion picture - encoded with gknot, 2 pass, 1400 meg (2 cd):
first shot (no issues):
http://scirocco.dhs.org/st-1.jpg
immediately before scene change (notice slight blocking near bottom right):
http://scirocco.dhs.org/st-2.jpg
after scene change:
http://scirocco.dhs.org/st-3.jpg
second scene change back to blue:
http://scirocco.dhs.org/st-4.jpg
third scene change back to guys (change occurs without fault this time):
http://scirocco.dhs.org/st-5.jpg

second example, with a scene from one of the star trek ds9 dvd extras, set gknot to average 950k/sec for the bitrate:
initial uncorrupted frames:
http://scirocco.dhs.org/ds-1.jpg
http://scirocco.dhs.org/ds-2.jpg
http://scirocco.dhs.org/ds-3.jpg
corruption after scene change:
http://scirocco.dhs.org/ds-4.jpg
http://scirocco.dhs.org/ds-5.jpg
http://scirocco.dhs.org/ds-6.jpg
http://scirocco.dhs.org/ds-7.jpg
corruption continues through another scene change (about a second later), and fades over the next few frames:
http://scirocco.dhs.org/ds-8.jpg
http://scirocco.dhs.org/ds-9.jpg
http://scirocco.dhs.org/ds-a.jpg

i'm using default divx settings, was using slow psych on the ds9 stuff, but no psych at all on the trek movie, yet still got this nasty blocking. i have seen times where i get macroblocking only in a small area (say 30% of the scene), and on another occasion i saw the entire frame get macroblocked during a smooth motion sequence (blue lighting was involved that time).

the system is a dual xeon 2.4 ghz, not overclocked, fairly fresh install of windows xp. every time i saw the macroblocking i did the encode a second time and verified that the blocking occured again. i even switched over to a w2k3 dev partition, did a clean install of the encoding stuff, and reproduced the same blocking there as well.

i've tried playback with media player classic (ffdshow+divx 5.1.1), and even with powerdvd (to do the captures, as i cant seem to get mpc-ffdshow to want to capture frames for some reason).

so, thoughts? opinions? whats the deal here? i'm tempted to just go back to divx 5.1, as this never happened there, heck, it didnt even happen with the 5.1.1 betas that i tried.

[DigitAl56K]

If you create a file "C:\framelevelcontrol.txt" the encoder should fill it with data while encoding.

Could you do this while running a test clip through for us and then send me the framelevelcontrol.txt file to digital56k[at]btinternet.com , together with a list of frame numbers in the output that show the blocking? (use VirtualDub or VirtualDubMod to get these).

You don't need to send the output AVI.

Thanks.

[allynm]

Originally posted by DigitAl56K
"C:\framelevelcontrol.txt"

sweet, didnt know that trick.

here is the framelevelcontrol.txt (http://scirocco.dhs.org/framelevelcontrol-ds9.zip)

664-4th ds9 pic (above)-macroblocks appear to occur right at the scene change (and a keyframe)
693-6th pic-next scene change, not detected though, macroblocks continue, although there should have been enough of a change to fix this
766-8th pic-next scene change, this one IS detected (keyframe), but the macroblocks continue. also, this is a light scene, so i really dont get why its happening.
781-793-9th/10th pic--blocks dissipate during these frames. the directors name slides into the bottom of the screen, and this seems to convince the codec to use more bitrate, side effect being that the scene-wide blocks are fixed.

looking at the framelevelcontrol.txt myself, i notice the quantizer, which doesnt usually get very high, and only gets high during high motion sequences, is pegging at 31 during this set of frames (675-770). this is very weird, since there is very little motion during these scenes, yet the codec is quantizing the crap out of it. i cant seem to figure out whats triggering it either. the value jumps to 10 earlier on in the encode (367-371), but this is correct, as there was a small explision on the screen at the time.

the vob for this file is small (~100 meg), and i have saved the .avs and the other stuff as well. i can recreate this at will, with my current gknot settings and such. if you need me to run some test builds or anything, just lemme know.

[temporance]

@allynm

Are you sure you're using the correct nth-pass logfile? In my experience trimming just one frame from the start of the video in the nth pass that you didn't trim in the 1st pass can cause this sort of stuff.

[allynm]

Originally posted by temporance
@allynm
trimming just one frame from the start of the video in the nth pass that you didn't trim in the 1st pass can cause this sort of stuff.

i'm letting gknot do all of the file handling, so i cant imagine why the frames would be different between the 1st and 2nd passes, or why the logfiles would be incorrect. the video is going through ivtc, but its the same vob and ivtc applied to both passes, so there should be no difference in the number of frames.

here are all of the gknot log files/avs/etc for this encode (http://scirocco.dhs.org/framelevel-other-logs-ds9.zip)

[Tingletangle Bob]

Hi!
I just wanted to let you know that I have the exact same problems. Often (but not always) that macroblocking happens when fading in a scence from black to "normal". I'm currently trying to encode the X-Files; i get this in almost all episodes (each about 40 mins long) and not always at the same position. The whole blocking takes about 5 seconds and then the quality is good again. Some normal movies of about 1.5 hrs never showed such a problem, however.
I even tried 3 passes without luck. Somehow it seems to me, that this problem is combined with the length of the clip.

I also already did a complete reinstall without success. I will try that Logfile thing this evening.

[allynm]

Originally posted by Tingletangle Bob
Hi!
I just wanted to let you know that I have the exact same problems.

glad someone else has the same issues. perhaps some more attention will be drawn to it. hey, i can hope, can't i? :)

are you using IVTC? if so, are the number of frames different between the 1st and 2nd passes? (going on what someone suggested above).

[Tingletangle Bob]

No, I don't use IVTC as my source is 25 fps PAL. I have not checked if there are differences, though. I will look it up tonight.

Guess it is something with the Nth pass strategy planning, which has been modified for DivX 5.1. I'm not really sure about this, but I think I already had the problem back then. :confused:

[DigitAl56K]

All: The team is back from the winter break now, I'll make sure we get on it. Thanks for the information allynm - Q=31 would cause bad blocking and shouldn't occur in your movie so far as I can see.

[allynm]

Originally posted by DigitAl56K
All: The team is back from the winter break now, I'll make sure we get on it. Thanks for the information allynm - Q=31 would cause bad blocking and shouldn't occur in your movie so far as I can see.

Thank you very much DigitAl56K
please let me know if you need more troubleshooting info/files. i can make the whole .vob available if necessary (100 meg). this is a very bad graphical bug and i would like to help fix it any way possible.

also note: in the past week i was able to recreate this blocking with the same files using divx 5.1. the blocking was slightly different (not as severe), but it was there (so was the quant of 31).

[allynm]

Originally posted by DigitAl56K
[B]All: The team is back from the winter break now, I'll make sure we get on it.

i imagine everyone is back by now. do you guys need any extra info/help reproducing this one???

[CyBorx7]

I too get these errors, in DS9 as well as a matter of fact. I used the DivX EKG to relocate more data to those frames. Well those frames got fixed, but right after the next frames got even worse, much worse. I havent found a solution to this yet... :( .

[allynm]

Originally posted by CyBorx7
I too get these errors, in DS9 as well as a matter of fact. I used the DivX EKG to relocate more data to those frames.

i've had this happen in stuff other than DS9, particularly where the video is mostly dark, with the occasional bright scene. i've noticed this happening within seconds of a bright flash of some sort. the codec properly handles the bright part, but seems to overcompensate in the opposite direction before or after the bright part.

[DigitAl56K]

Investigation of/fixes for this issue are definately planned for the next release. I can't tell you when that will be, but thanks for all the information you have provided! :)

-Al

[allynm]

Originally posted by DigitAl56K
Investigation of/fixes for this issue are definately planned for the next release. I can't tell you when that will be, but thanks for all the information you have provided! :)

-Al

my only concern is to know if the cause of the issue has been identified. i need to know how long i need to stick around to offer assistance in identifying / nailing this one down.
Al

[CyBorx7]

Encoded 4 more episodes of DS9, out of which 3 came out messed upright in the beginning. One of them was fixed by just letting the final filesize get bigger, for 2 others i enabled psychovisual enhancements (fast) and they came out very well.

[SeeMoreDigital]

Can you confirm the source please allynm. DVD, captured from TV etc?

Cheers

[allynm]

Originally posted by SeeMoreDigital
Can you confirm the source please allynm. DVD, captured from TV etc?

Cheers

source is telescened ntsc tv video, from the star trek DS9 season dvd's (encoding from direct rip of dvd). i have also had it happen with wide screen movies, but they were also telescened ntsc. in both cases i was using IVTC via gordian knot / avisynth, so divx only saw progressive video as its input.

the only common factors i have seen was the source video was dark, with some momentary bright areas (blue in nearly all cases). most of the other stuff was documented/posted earlier in this thread (about quant going to the maximum for a few seconds for no apparent reason).

if any further info/testing is needed, please dont hesitate to ask.
Al

[freezer]

I have encountered this problem with macroblocks when encoding my own DV (MJPG) material. First, as already has been said it occurs just after a fade between scenes. Second for me it also occurs in the middle of scene without much changes but extrem colors. I have filmed a guy walking out of a foggy sunrise and in post-production did a gain to the colors resulting in an extremely red and saturated orange/yellow. The colors are broadcastsafe PAL, saturation 80% max, ranging from 0 to 235.

Download the example here. (http://www.loom.at/stuff/Macroblock-Problems-DivX511.avi)

[allynm]

Originally posted by freezer
I have encountered this problem with macroblocks when encoding my own DV (MJPG) material.

...so it appears that it happens with any source, so long as the video content does whatever it takes to cause this crazy macroblocking to occur.