Saving time with slow filters

[wingphil]

I've been wondering. When using gknot, does it apply all the filters you use twice, once for each pass? i.e. could I save time by doing the following:

- Create the avs file in gknot with whatever filters, load it in vdub and encode the output using huffyuv to an avi
- Load the resulting avi in gknot, comment out most of the new avs and encode as normal

I think this would save a lot of time, especially with one or more slow filters, e.g. i am wanting to use pixiedust followed by c3d/moviehq to squeeze 3 hrs of a comedy series onto one cd. If you did this, you would only need to apply the filters once, before you start any encoding, and the total time would be shorter.

Or does gknot do this anyway? I don't know.

[manono]

Hi-

does it apply all the filters you use twice, once for each pass?

Yes it does.

Interesting question. GKnot will still run 2 more passes after loading the Huffy .avi, so you're running 3 passes in all; one really slow one encoding to Huffy, and 2 real fast ones since you're recompressing an .avi with no filters added. So it depends on if you have enough HD space for the uncompressed Huffy, and how long the original Huffy encoding takes with all the filters. But, if you're using both PixieDust and C3D, I think it'll probably result in a considerable time savings by doing it the way you're suggesting.

This is a well known method for encoding SVCDs. If you're doing 4 passes with CCE, and you're slowing the encoding process with some filters (filters nowhere near as slow as your combination), then it's often worth it to make a Huffy .avi first, and then run that through CCE. Although Huffy is uncompressed, as I understand it, it's not lossless, but I don't think you'll notice any loss in quality, particularly after using your filter combination.

You might try and encode a small portion of the video first to get an idea of the relative encoding times using the two methods.

[hakko504]

@manono

Huffy is compressed, and it is almost completely lossless. The losses you get from huffy are usually in the order of rounding errors, but there are some special cases that can produce larger errors. The only time you'll get measurable errors is when you convert from RGB to YUY2 or vice versa.

[wingphil]

cheers for your help guys, but i don't understand why it wouldn't always be quicker. surely huffyuv is pretty quick to encode and decode, so you'd almost always be doing less work, wouldn't you? because you'd also be saving time on the mpeg2 decoding, deinterlacing/IVTC and resizing that you'd have to do only once instead of twice, as well as any other filters you might use?

[hakko504]

There are a few reasons for this: Huffy is a very slow codec to work with: it requires a lot more processing power than DivX (per frame) - but it may be fast enough for your purposes You need a manual interaction stage between the compression of the Huffy and the start of the actual DivX encoding - You can't leave it unattended over the night and expect all three parts to be done in the morning. And the main reson not to do it this way is usually HD space - can you spare 50~60GB for a 90 min movie?

[wingphil]

really? i thought it would be quick. probably still quicker than pixiedust+c3d tho.
good point but it was going to take about 50 hrs (estd.) to finish one pass on my celeron 900 :scared: so i'm not too worried about that
its going to be only 320x240, so think i'll give it a go.


i'll let you know how i get on.

[hakko504]

50 hours! No wonder you want to speed things up. I mean, when I say a lot slower I meant maybe 50% of actual encoding time, not counting the overhead from pre-processing. (Think of it like this: the difference in time between DivX and Huffy when doing 1h of video will be constant no matter how much filtering you do. Thus, there is a point when the overhead from filtering will be greater than the speed loss from using huffy plus having to do a third pass. Finding that point isn't always easy though, but in this case you not only have found it, you've passed it by a considerable margin. And at 320x240 the HD space needed can't be more than 20GB/h. Even I could find the space for something like that with my mere 230GB of HD... ;)

[wingphil]

i'm not arguing or anything *bows down to superior knowledge* :) but this guy (http://math.berkeley.edu/~benrg/huffyuv.html) seems to reckon its very fast, like faster than real time. am i missing something? its a pity you can't break a process's cpu time down by the dlls its using, that would be interesting.

edit: and just a thought, what about using divx at 100% quality instead of the huffy?

[jonny]

edit: and just a thought, what about using divx at 100% quality instead of the huffy?

Doing this you'll probably going to lose a lot of video information (and with divx it will take more time imho)

[wingphil]

so is huffyuv everyone's favourite lossless codec then?

[hakko504]

Originally posted by wingphil
so is huffyuv everyone's favourite lossless codec then? Not to mention the only lossless. And while fast enough for realtime captures of 704x480@29.97 at a reasonable computer, it doesn't mean that something else can't be faster. Huffman compression is somewhat complex, and a MJPEG or 'blind' (no motion search) DivX capture would be faster but since they rely on (slightly/very) lossy compression they are not really suited for top quality captures or as temporary storage in this case. One limiting factor is often the huge amount of data that Huffy produces: A normal huffy AVI often produces data at a rate close to maximum of DMA-33 enabled drives. That means that at a DMA-100 device, Huffy will spend 1/4th of a second writing data for each second of the AVI produced. For DivX (@10000kbps) the same time is 1/10th of a second, assuming the full 10000kbps is used every second. So it's not only a matter of pure encoding, there are other limiting factors as well.

[wingphil]

update: looks like the huffy is going to be about 23 GB (3 hr movie) and run for about 40 hrs. and thats after taking the c3d call out. sheesh! :(

[ohliuv]

hmm... 40hrs for 1st only or for both passes?

[wingphil]

there is only one pass when encoding to huffyuv ;)