Bad frames randomly appearing in Huffy AVIs (MeGUI)

[Dr.Khron]

I'm doing some batches of Simpsons episoides, using x264 and MeGUI. I usually check the the "Add pre-rendering job" box since I have plenty of hard drive space, and becuase my AVS scripts only run at 12-13 FPS.

I've noticed a single bad frame in two different episodes.
This one here is from the end credits, but the other one was in the main show.

http://img508.imageshack.us/img508/2656/badcreditsdr3.jpg

Tracing back, I find that the bad frame exsists in the huffyUV AVI, so it has nothing to do with x264. Sometimes the entire frame is distorted, sometimes just a piece of it, but its always a single frame: the frames before and after it are just fine.

When I found the first error, I assumed that since I was using the computer duirng the encode, maybe there was a memory leak or something. So, I re-ran the encode over night while the PC was undisturbed, and the result had no bad frames. However, I ran one last night that had a bad frame, so my theory is crap. I'm guessing that these bad frames are completely random, becuase I can't re-create them. But what could be causing them?

The only workaround I can think of is to run the pre-render job first, inspect the entire huffy AVI for bad frames, and then encode normally with the huffy as input.

[DarkZell666]

Anything looking "random", I'm afraid, can be caused by some (and combinaisons) of the following :
- poor quality/defective/old RAM
- overclocked and/or unstable mobo/cpu (overheated ?)
- some batard(tm) multithreading implementation (just throwing ideas in right ? ;)).
- memory leaks indeed, you'd need to do your encode after a fresh reboot, and try another encode of another episode with the same codecs/programs straight afterwards and see what sort of garbage might be lying around in the memory. If the first encode runs fine, but the following one doesn't (and you reboot and have this "1st one ok, 2nd one b0rked" pattern each time), then chances are you have discovered a memory leak.

What's your avs script like ? Are you using the latest version of every plugin in it ? Are you using some MaskTools/MVTools somewhere ? Do you happen to have a dual core cpu ? Maybe huffyuv wasn't thoroughly tested on such cpu's ... (I don't have a clue right ? if it has, forgive me for being ignorant :p)

But heh, before wasting hours doing some encodes, better double check your system stability first, it wouldn't be the first time unsuspected hardware starts being naughty ;)

[Dr.Khron]

Hmmmmm.... Well, good point. My PC is heavily overclocked.
Its an old Athlon XP (thoroughbred), and I've got it running at a 25% OC. However, its been rock stable at this OC for four years, with never a crash or any wierd heat related issues (I've got some big noisy fans in there).
I suppose a thorough spring cleaning to the filters, fans and heatsinks would be a good idea.

But, I think I'd like to blame Mencoder and the pre-rendering job. You need to update to a newer version of mencoder just to make the pre-rendering jobs function normally... so maybe the reason that the creators of MeGui havn't included the lastest version of Mencoder is because it conflicts with something. Anyway, I'll re-run the encodes without the pre-render job, and see if I still get bad frames. Should only add maybe 45 minutes to the job.

What's your avs script like ? Are you using the latest version of every plugin in it ? Are you using some MaskTools/MVTools somewhere ?

Well, since I cant reproduce the bad frames with AvsP, or by re-running the huffy AVIs, then I doubt it could be anything in my scripts. I did a clean OS install recently, and I took pains to add in only the newest of the various plugins.
And yes, I'm using part of MaskTools, as I'm using LimtedSharpenFaster.

[[P]ako]

The only workaround I can think of is to run the pre-render job first, inspect the entire huffy AVI for bad frames, and then encode normally with the huffy as input.

If it is a single frame you can freezeframe it. No need to pre-render again.

[Dr.Khron]

If it is a single frame you can freezeframe it. No need to pre-render again.

Good point. But how do I FIND those frames?


I cleaned out my PC (including all filters and heatsinks), and declocked it to stock speeds. It seems to have worked, as my last three encodes came out fine, with no bum frames in the Huffy AVI.

However... I've got 6 episodes in the can that I encoded in a batch, back before I discovered the problem. They all might have bad frames, but I'd rather not sit there and watch them all.
Is there a way I can automatically scan through my MKV (or huffy AVI) files and look for bad frames? Since they are only single frames, it seems like there would be an easy way to filter for them.

If I have to sit and watch them all the way through, I'd rather just re-encode them first, and then watch them.

[foxyshadis]

You can try, for instance, this avisynth script:

prev=deleteframe(0)
next=duplicateframe(0)
prevdiff=mt_lutxy(last,prev,"x y - 4 / 2 ^")
nextdiff=mt_lutxy(last,next,"x y - 4 / 2 ^")
writefileif("D:\video\bad.txt","AverageLuma(prevdiff) > 100 && AverageLuma(nextdiff) > 100","current_frame")

mv method:
f=MVAnalyse(pel=1,idx=1,overlap=4)
b=MVAnalyse(pel=1,idx=1,overlap=4,isb=true)
prev=MVCompensate(last,f,idx=1,scbehavior=false).removegrain(20)
next=MVCompensate(last,b,idx=1,scbehavior=false).removegrain(20)
removegrain(20)


Open the script in vdub, use run video analysis pass, close, and check out the file. It'll give you a list of frames you can at least check. You can plug any that are definitely bad into fizick's BadFrames (http://avisynth.org.ru/badframes/badframes.html) plugin.

I totally made up the 50 though, you might need 10 or 100, I don't know. It's just a quick and dirty little untested thing.

[Dr.Khron]

Ah, very cool, thank you very much!

I kinda knew how to do this in theory... its just that my coding skills are weak. I had no idea how to output a text file, thats huge for debugging issues like this. I'm assuming that the value in question is the threshold of difference (in average luma for the frame). I'll fool around with it and see what happens.

BTW, declocking my PC to stock settings produced a performance hit of about 25-35% for encoding jobs!!!
Ouch.

[squid_80]

FWIW, the frame you posted is definitely indicative of a glitch in the huffyuv compressed stream. The reason it looks like it does is because of the way huffyuv works; pixels are predicted downwards so once a decoding error happens it messes up everything below.

[Dr.Khron]

Wow, you are exactly correct, Squid. Most of the frames had much less corruption, but it was always ran all the way to the bottom of the frame, regardless of where it started.

Hmmm... I can't really get that script to work, but I havn't had time to fool with it enough. I'll take a crack at it this weekend when I have more time.

[foxyshadis]

Actually tested this time, and removed a syntax error and the silly makediff that I wasn't thinking through. :p Also added a mv method to replace prev/next for more accurate results in scenechanges and motion, much slower so it's up to you.

[[P]ako]

Good point. But how do I FIND those frames?

Easy as sit and watch through the episode.

If I have to sit and watch them all the way through, I'd rather just re-encode them first, and then watch them.

One episode takes you 20-23 min to watch. If you are not doing any filtering other than deinterlacing (if there is any) and resizing, then you are better off re-encoding and then watching. On the other hand, if you are doing some heavy filtering... it could be faster to watch through the HUFFY.

[Dr.Khron]

Well, I got it working... but it doesn't catch the bad frames. It catches about 30-50 frames per run, and they are mostly high motion shots.

How can I adjust the sensitivity? I'm using the first method, and changing either of the 100's, makes it return no frames.

Also, I made a spreadsheet to write an AVS for me from the suspect frame list, so I can just easily check all the suspect frames with AvSP. It makes a script like this:

trim(186,-1)+
\ trim(745,-1)+
\ trim(746,-1)+
\ trim(1832,-1)+
\ trim(6178,-1)+
\ trim(6181,-1)+
etc...

I tried to get fancy and edit the WriteFileIf command so that it outputed an AVS file with the trim command as above... but I couldn't figure out the syntax to get AVIsynth to write the lines in that format.