Frame serving over a network

[yokem55]

I have two systems, a p3733, and a p4 1.8, and I was wondering if it was possible to have the 733 doing all the mpeg2 decoding, filtering, resizing, serving up the frames, and having my P4 access the frames over my 100 mbit network and do the divx encoding.....now some simple math tells me that 640x352 full true color frames is going to over saturate my network by about 30%, so even if it were possible I doubt it would be fast enough to be worthwhile....am I right about this, or is it worth persuing.....

[department-7]

Originally posted by yokem55
I have two systems, a p3733, and a p4 1.8, and I was wondering if it was possible to have the 733 doing all the mpeg2 decoding, filtering, resizing, serving up the frames, and having my P4 access the frames over my 100 mbit network and do the divx encoding.....now some simple math tells me that 640x352 full true color frames is going to over saturate my network by about 30%, so even if it were possible I doubt it would be fast enough to be worthwhile....am I right about this, or is it worth persuing.....

640 x 352 = 225280 pixels
x 24bit (3 byte) = 675840 bytes (660Kb)
x 25frames/sec = 16.5 Mb/sec

100 Mbit/sec = 9.6 Mb/sec

This gives an estimated 72% overload for me.

Even 30% overload gives you a network related frame rate of
aprox. 17-18 fps. PIII 1.1 speed - hu?
And I did not include protocol overhead and processor load
through heavy network traffic in my calculations.
And which nic does reach the magic 9.6 Mb/sec limit?
OK - one could argue: Turn up the MTU and use jumbo frames.
BTW - do you have switched network? fdx 100Mbit?
But 100Mbit does not do the trick.

Get some gigabit nics or start up the dvd-player on the PIII
and feed the analog video to the P4 through a tv-card and
use the virtualdub custom capturing mode with divx in realtime.

happy hacking - =)

d-7

[droolian01]

Hi there.
Just a quick point. I will be upgrading to a second computer in a few months and have been (briefly) looking at networking solutions.
Why not use firewire as a distributed encoding medium? It is suppossed to support up to 50 MB/sec speeds and is cheap to implement.
Look forward to any explanation, thanks.

[yokem55]

the benchmarks I've seen though of firewire networking though has been that it's generally slower than 100mbit ethernet....I looked at this as well....I likely will be getting a pair of gigabit ethernet boards in order to try this....

[madoka]

Originally posted by yokem55
the benchmarks I've seen though of firewire networking though has been that it's generally slower than 100mbit ethernet....I looked at this as well....I likely will be getting a pair of gigabit ethernet boards in order to try this....

This (http://www.pcbuyersguide.com/solutions/networks/ieee1394_networking.html) seems to say that IP over 1394 is faster than 100BT, though not exactly 4x the speed.

[yokem55]

hmm....I'm not sure where I saw the benchmarks that I had seen....I think I had just done a quick google search....anyway 14200kbyte/s still isn't quite enough for 640*352*24fps for real time decoding an network frameserving(you'd need 15824 kbytes/sec transfer rate for just realtime, faster than realtime would require even more bandwidth)....when you have a powerful machine that when it isn't burdened by anything else can encode faster than real-time, being limited to just below real-time by the network isn't worthwhile if you would be going just as fast if the mpeg-2 decoding were done locally....hence gigabit ethernet, or a dual processor system (pick your poison) is what you need to get encoding performance to the next level.....

[Blight]

I think your math is wrong...

Most AVISynth processing is done in YUY2 which isn't 24bit. I think it may be 16, but not sure about that.

[Nic]

Hi Blight,

Ive seen YUY2 described (on the net) as:
"YUY2 colourspace (A 16-bit YUV colourspace
variant)"

But Ive also seen it listed as 24-bit (???) I've dealt with YUY2 and because of the packing it is not as easy to determine the bits as RGB....If you do find a definitive answer please post it :)

Thanks,
-Nic

[Guest]

YUY2 has 32 bits per two pixels: 8 bits for the Y (luminance) of the first pixel, 8 bits for the Y of the second pixel, 8 bits for the U color difference for *both* pixels, and 8 bits for the V color difference for *both* pixels. So it is 16 bits per pixel and the chroma is subsampled by a factor of two. (Note this is not the byte ordering of the data; that is: Y1 U Y2 V)

[TactX]

Read more about YUV formats (http://www.webartz.com/fourcc/fccyuv.htm) here.

[yokem55]

So....640*352*16*23.976=~10.3mbytes/sec so long as everthing stays in yuv mode over the network....this may make firewire feasible...based on the above benchmark of the firewire networking, the firewire connection should be able to handle ~32 fps of uncompressed yuv frames....but any higher resolution, and you're still taking a nosedive...with gigabit though..you've got 10x the bandwidth, and the bottleneck is not the network....hmm....

[yokem55]

What would be helpful with this situation would be if it were possible for avisynth to be able to output huffyuv compressed frames. I realize the architecture of avisynth doesn't really support this, but would it be possible to do it as in the form of a colorspace conversion plugin? Huffyuv compression (which as far as I can tell is pretty fast)would cut the bandwidth requirements of network frameserving of 640*352 yuv frames in half....sometimes I think it would simply be better to just get a dual processor system to get the performance boost I am looking for.....oh well.

[drZymo]

Originally posted by department-7


640 x 352 = 225280 pixels
x 24bit (3 byte) = 675840 bytes (660Kb)
x 25frames/sec = 16.5 Mb/sec

100 Mbit/sec = 9.6 Mb/sec

This gives an estimated 72% overload for me.

Even 30% overload gives you a network related frame rate of
aprox. 17-18 fps. PIII 1.1 speed - hu?
And I did not include protocol overhead and processor load
through heavy network traffic in my calculations.
And which nic does reach the magic 9.6 Mb/sec limit?
OK - one could argue: Turn up the MTU and use jumbo frames.
BTW - do you have switched network? fdx 100Mbit?
But 100Mbit does not do the trick.

Get some gigabit nics or start up the dvd-player on the PIII
and feed the analog video to the P4 through a tv-card and
use the virtualdub custom capturing mode with divx in realtime.

happy hacking - =)

d-7

You say you don't calculate the overhead and such, but still you say 100mbit = 9.6 MB/s ?? I don't get it. 100mbps = 12.5 MB/s to me :D

But well , i was thinking about this way to frameserve to, last day. But it's just not feasible with the current 100mbit. The only thing that can be tried is a to apply a simple huffman compression to the frame, but takes away a little speed. Perhaps it's worth it and we gain a bit of speed increase compared to the standard 1 system encoding. :P

[Richard Berg]

I do this all the time - the weak link is rarely network bandwidth, even when feeding dual XP's. The computer with the WinTV card has its capture drive mapped on other machines to the same drive letter as it is locally; I can open any of my avisynth scripts on any computer and have them work automagically.

[Kedirekin]

Umm, you're talking about file serving, not frame serving. There is obviously no problem with file serving - that data is compressed.

[OUTPinged_]

some thoughts on that:

1. raw data can be lossless compressed (huffyuv). that brings us down 2.7x times in terms of traffic usage.

2. the best usage for second pc frameserving is when using very slow filters (spartial soften+temp.soften or SSHQ), and you are getting 3-4 fps anyway.

[dividee]

I think file serving is the way to go. With frame serving you're limited to 2 computers anyway.

What we need is an helper app to automate the process. Since Virtualdub (Nandub) can be controlled by another app this should be feasible.

This is for divx3 but it could be easily(?) adapted for other codecs:
You have a master app on one computer and slave apps on the others. The master splits the movie in chunks with size proportional to the encoding speed of the different computers. Each chunk will have a keyframe but you could have interpenetrating chunks. The master app wakes up the slaves which encodes the first pass for their chunk. When they are done, the master merges the stats chunks and process the bitrate curve. If needed, the chunks can be recalculated to avoid intra-scene KF since we now have scene change info. The second pass is done like the first one. The master just has to merge the final avi chunks (taking care of interpenetrating chunks if needed).
The only downside is that the rate control is less precise; you multiply the max. size error by the number of chunks, but this is a small price to pay.

Anyone up to the task ? :devil:

[eng3]

would the network speed be a problem?
I'm fairly new to this compared to you guys, but if a video is 16.6MB/s, that is if you tried to play the video real time. my encoding on my computer does not go at 30fps, it goes at like 3fps because I am using a avisynth filter, etc.
the data rate would be much much slower.
I have two computers also, it would be nice if the load could be split somewhat. Any improvement in speed would be nice.

I would think it would be something like, send one frame to my computer, send the other to the next computer, at the end, put them together at the end.