TCPDeliver 0.2 - Technology preview.

[sh0dan]

TCPDeliver 0.2 - Technology preview.
- What is this -
This will enable you to send clips over your network. You can connect several clients to the same machine.

- Disclaimer -
This is pre-alpha software!

- Expect bugs.
- Expect future compatibility to be broken.
- Expect less than optimal performance.
- Read Bugs section.

- Syntax -
Server:
------
TCPServer(clip, int "port")

This will spawn a server thread on the current machine running on the specified port. Port default is 22050.
You will get output in the application you open your script in, but the server will only be running as long as the application (vdub for instance) is open.

Example:
Colorbars(512,256)
TCPServer()

will start a server.


Client:
------
TCPSource(string hostname, int "port")

This will connect to the machine with the given address (IP-number for instance) to a server running on the given port. Port default is also 22050 here.

Example:
TCPSource("127.0.0.1")
Info()

This will connect to the local machine, if a server is running.

- Bugs / Todo -
- Don't report bug yet. This is still highly experimental. Bug reports are likely to be ignored.
- Parity information is not getting passed.
- Not very efficient on multiple clients (Frame requests block net transmission).
- Status GUI, on both Server and Client.
- Simple, fast, lossless Image Compression.


Comments, experiences and feature requests are however welcome. :cool:

Edit: Removed links.

[Si]

Example works fine on a W98 machine.

Great work - start of a new era I feel :D

regards
Simon

[sh0dan]

Posted v0.2

- Client now prefetches the next frame. Much better efficiency.

[DvdKhl]

mhh interesting new feature. thx sh0dan :D

I wonder if you can gain some speed when one PC does the hole filter thing and the other the encoding.

I will do some testing to find out.

[sh0dan]

@DvdKhl: If your filter chain is long enough - yes. ;)

For simple scripts, where you decode MPEG2 content, crop, resize and send it to XviD I don't think - your network will be too saturated. But when doing filtering you are likely to get a gain.

Multiple clients are not the optimal situation - but hopefully that'll change before 1.0.

Edit: Silent updated the file above. Updates resulting in better client efficiency. Server should not be updated.

[trevlac]

I have not had time to try this, but hopefully a post of encouragement adds value to this thread ....

THIS is a VERY cool feature. :D :D

[Mug Funky]

i'll just jump in and say VERY COOL.

[mf]

Oh dear, now I'll have to get myself an AVISynth cluster with gigabit ethernet cards! :scared:

I presume forwarding is also possible?
for instance:
Clustermember 1:
AVISource
Deinterlacer
TCPServer

Clustermember 2:
TCPSource
Sharpener
TCPServer

Clustermember 3:
TCPSource
- client app -> video codec -> final file

[bill_baroud]

eheh mf, a computer for each filter :p
you're crazy =)

you should look to IEEE 1394 networks, it's quite cheap and fast (400Mbit/s...) and damn easy to setup (in WinXP, plug-in and enjoy).
Last time i tried the laptop harddrive wasnt't fast enough to keep with it :)

[timecop]

Hmm Yeah...
FW800 cards at each processing machine and a short run of firewire cables would be really nice.

Now to wait for PCI-Express boards + firewire cards, since current PCI bus doesn't have enough bandwidth for FW800.

...
Now just wait until some Lunix fanatic sees this thread and you'll have to fight them off from making a beowulf cluster of filtering machines.

[sh0dan]

@mf: Yes - that's the concept.

Regarding network speed I think a 10mbit network should be enough for mf's needs :D

[mf]

Originally posted by sh0dan
@mf: Yes - that's the concept.

Regarding network speed I think a 10mbit network should be enough for mf's needs :D
10mbit/s = 1.25mbyte/s
I'd need... 7*24*8=1344mbit/s network :D

[vion11]

Are there any "silent clients" out, means no/minimal gui
and work as clustermember 2 in mfs script above?
Interesting is only fps and cpu load, I think.

I like server accepts more than one client.
Avisynth can broadcast now, to bad I'm behind
firewall, otherwise open alternate tv station
and start unfriendly overtaken of local pal provider.

"movies for masses, now!"

[hanfrunz]

sh0dan this is so cool!

I have some ideas for you:

If somebody uses this architecture for previewing purposes, you could add a lossy jpeg compression this should gain network-speed a lot!

For lossless compression you could use standardformats like png, tiff or maybe jpeg2000 so you could use external libs for that and maybe write faster routines later. Or maybe there are TCP/IP-protocolls that compress data for you, like in analog modems :)

I just made some tests with uncompressed PAL(YUY2) and i only get 28f/minute (100mbit Ethernet) encoding to HuffYUV on the client (P4 2,4GHz CPU)

I'm looking forward to the next version with compression :)

hanfrunz

[Popovk]

Originally posted by bill_baroud
eheh mf, a computer for each filter :p
you're crazy =)

tsss, don't be so suprised, mf IS crazy, you just have to get a look to his script to understand that.
I'm almost sure he'll be able to make an insane script with incredible result that'll put 3 computer with P4c 3ghz processor using this network tool to a impressive speed of 1 fps each ! :devil:


Nice new, Shodan, but you should really avoid to be so close with mf, I fear what we'll be able to do with this sweet stuff ;)

[sh0dan]

@hanfrunz: I'm planning on adding compression. It's in the "protocol", but not implemented. I'm currently considering a huffyuv-inspired implementation. Main consideration is speed - if it cannot compress >100mbit/s it's of little use. Another cosideration is simplicity - I don't want the protocol to become bloated with various compression shemes.

The implementation is almost complete now, and I've just cleaned it up a bit. Next big project is a popup GUI - so I guess I'll have to study GUI-sample a bit more! :)

I don't plan lossy compression. After all reduceby2 reduces your data load by a factor 4. :)

[hanfrunz]

mmh another idea:
you could use it as a previewing system on one computer:


Serverscript:
a=AVISource("somefilm.avi")
b=a.reduceby2()
c=a.flipvertical()
d=a.bicubicresize(800,600)

TCPServer(a,22050)
TCPServer(b,22051)
TCPServer(c,22052)
TCPServer(d,22053)

return d

Clientscript1:
TCPSource("127.0.0.1", 22050)

Clientscript2:
TCPSource("127.0.0.1", 22051)

...



just an idea...

[Richard Berg]

If somebody uses this architecture for previewing purposes, you could add a lossy jpeg compression this should gain network-speed a lot!

When I did some testing a few years ago, the CPU overhead (jpeg and png) was worse than the network speedup. Things may be better with today's CPUs, especially if we use code like HuffYUV that's already designed for real-time video.

[avih]

very cool. sh0dan, how about using MPEG4 for compression? ;)

[TheXung]

Can server and client be the same machine? thus allowing filtering and encoding to be split on dual setups with non-multithreaded encoding applications.

[sh0dan]

@TheXung: Yes - just connect to 127.0.0.1

[bill_baroud]

hmm, but that would cause a little delay, even through the local interface no ?


tsss, don't be so suprised, mf IS crazy, you just have to get a look to his script to understand that.

well, i wanted to say _really_ crazy ;)
i know mf and his scripts from some time now :scared:

[mf]

Pfff. :p

[esby]

Well i wanted to add:

For normal users, there will be probably no benefit of using multiples instances over a network,
the average speed you get on one computer can be superior
to the speed obtained with network transmission, and its delays...

But for users like mf, which proceed at very slow fps no matter
the numbers of computers used :D,
the data rate needed could be low enough to support the network speed,
and produces a gain of time...


esby

@Avih:
That's a nice idea, including encoders in the filter chain,
how about merging vdubmod and avs , in order to completely implements the idea :p

[Richard Berg]

No, I think the gains are very real for average users, if you Trim() the source into different regions for each computer.

[phrentec]

using the example scripts I get

TCPClient: Did not recieve expected packet (SERVER_SENDING_AUDIO)

So I guess it won't ignore that there is no audio stream and keep playing the video instead of displaying the error.

on the side, there is no way I can go through firewalls on opposite sides. not that I intend to use it through that.

Great job, it's at least workin.

[sh0dan]

@phrentec: That error can mean a bunch of different things, and may not have anything to do with audio.

[esby]

...
TCPServer(a,22050)
TCPServer(b,22051)
TCPServer(c,22052)
TCPServer(d,22053)

return d

Clientscript1:
TCPSource("127.0.0.1", 22050)

Clientscript2:
TCPSource("127.0.0.1", 22051)



mmmmm

Supposing a,b,c & d are trimming parts...

with on client side

(lets suppose we assign a b c & d with the proper tcpsource)
return a+b+c+d

the application will probably request frames of a, then frames of b,
then frames of c and finally frames of d...
meaning a
tcpserver(a) will work, then
tcpserver(b) will work, then
tcpserver(c) will work, then
tcpserver(d) will work...

Multithreading & trimming here does not help at all...

Of course if you use selectevery, that could work...

One other thing that bother me,
Is avs multithreaded yet, or

because if tcpserver(a) consults the filterchain
while tcpserver(b) is already accessing it...
What will happend?
(probably nothing since it's read only, but i'm asking just in case)
Or maybe tcpserver(a) & tcpserver cannot work together at the same time...
And then again, you don't get all the potential speed you would have in theory... right?

esby

[timecop]

I tried it.
I hate DivX 5.05 and higher versions. But I have a video clip encoded with 5.05 that 5.02 decoder doesnt decode.
So I wanted a way to avoid installing CrapX 5.05 on my encoding machine. So I thought this would be the solution. I loaded DivX 5.05 on a throwaway test machine and installed AviSynth 2.54, and TCPDeliver.

On the crap server, I had AVS script to load the avi, trim and resize it. really basic, nothing special. (AVI had no audio).

On the encoding machine, I had a simple TCPSource() or whatever is the function name to receive data from the server.

Loaded up VirtualDub on the crap server, noticed port 22025 was being listened on, made a one-line script with TCPSourcE() on the encoding machine, and hit encode in vdub. Network usage was about 500-700k/sec, CPU utilization on both machines was quite low, and the encoding was going at <= 0 fps. After about 5 minutes the estimate remaining time finally calculated and said it was going to take 13 hours to encode this 25 minute clip.

I guess this still needs work.
That, or I missed some way to make this superfast(tm).
Maybe it only works when source material is pirated DVD content?

[morsa]

That´s the idea, when you release a beta version you wait for opinions and testing to discover problems, bugs, etc.
I wonder if you know what I'm talking about and if you know what communism is....
How old are you, kid?

[Richard Berg]

Stay on topic.

[timecop]

Originally posted by morsa
That´s the idea, when you release a beta version you wait for opinions and testing to discover problems, bugs, etc.


Yes, and I posted my experience.
What is so wrong with that?
Hopefully the developer(s) can analyze the environment and come up with fixes.

[mf]

Originally posted by morsa
I wonder if you know what I'm talking about and if you know what communism is....
How old are you, kid?
timecop hates everything, everybody and his pet. Don't take it too serious :D.

[morsa]

OK :D

[sh0dan]

Thread closed.