servers not in synch

[Doom9]

as written in the news... I have 2 servers in roud robin mode (so you never know which one you'll get) and one of them is giving me crap when I try to store something to it resulting in one server being 2 days behind in terms of news. It will be fixed when it's fixed and everybody bugging me about it will receive no christmas gifts whatsoever :P

just letting you know....

[Doom9]

and an update on the situation.

It looks pretty grim. Here's the facts:

Since 72h I can't make any updates to server1.doom9.org. At first I just got write errors, since 48h my ftp login doesn't work anymore. When I tried to contact the webmaster I got back an email from another guy telling me that the guy I wanted no longer works at that particular company and they didn't have a forward address. And since the server is running in the net of that particular company I would presume that sooner or later my whole stuff will just disappear.

I have 3 servers in total, one I've never really used for several reasons: 1) For months now I've tried to contact the webmaster with no success
2) the vhost isn't properly set up
3) there's no properly working caching mechanism meaning when somebody accesses a certain page I can't replace it with a new version. And as I can't contact the admin there's no way to get this fixed which makes the whole thing unusable..

So.. looks like I need another hosting. Before you start suggesting free services let's have a peek at what's needed: 300-400GB traffic a month for starters... that already excludes all free public hosts. And about commercial hostings... you probably know about the tight spot Digital-Digest is in financially.. bandwith is really expensive and the amount of donations I get atm (like 2 this months so far) would mean I'd have to pay it all out of my own pocket, a thing which I'm a) not willing to do b) not in any way capable to do as I'm just a student.

And for the rest of the requirements: vhosts (so the server will respond to doom9.org and can be used in round robin mode so the load can be shared). ftp access to upload.
php and mysql would also be nice but I could live without if.

If anybody out there has too much unused bandwith let me know. I can't just double the load on the remaining working server or I run into risk of being kicked out there and that would leave me with no hosting at all (and you can imagine what that would mean for the future of my page).

[cofferscuffs]

Sorry about your position but for now, can't you get a CGI script which limits the number of visitors on the site?

[b0b0b0b]

What do you study?

[b0b0b0b]

There are bandwidth throttlers for apache, but I think we have to remember that these servers are out of doom9's hands and their admins might not want to install extra software on their servers.

[Doom9]

you know.. limit number of users and you get into trouble.. lots of people bitching.. we tried before.. the results were disastrous. People don't understand why they can't get onto your page and will not come back. And I don't need any mods.. I need other hosting.. it's that simple. You can't cut traffic in half without pissing people off and with 23'000 daily visitors there's just a lot of traffic.

[Absimilliard]

Hi Doom

contact me, you should have my email!

I may help you!

[mrbass]

Suggestion: Forum and doom9.org are on separate servers which is great. Split up the (a)html and guides (images) and (b)all the downloads of software on another server. This way people can still get to the guides and find other means of obtaining the software perhaps (not always though). Also perhaps any software over 1MB or 1.5MB don't host it. Instead provide a link to it on tucows. Just suggest to the author to submit their program to tucows and they have mirrors around the world. Something must be done to cut down that bandwidth of 400GB/month.

[Doom9]

actually.. it's more like between 500 and 600gb a month in total.. and the forum is starting to pull a lot of traffic, too.. 21gigs in the first month.. so you can see.. even html pages create a lot of traffic.

And.. most progs are not really welcome on most regular sites... take ac3 decoders for instance.. dolby doesn't really like them.. ripping progs... really hot items. And direct linking to other people's files is not good stile and usually forbidden to prevent leechers (sometime I have the same problem)... so that wold require that each and every program author maintains his own webspace.. which usually means some public service which means a lot of kicking.. and the story goes on. The reason why my site and digital digest run their own archives is that people don't have to go to twenty sites to get all the soft they need for a dvd rip.

[Absimilliard]

I think the undeclared situation in europe makes it difficult to host files of this area.

Files that host on tucows or other "anonymous" services give a bad taste to the (i´m sure) legality of the software you use for personal backups.

[pinoy2201]

Doom9 go ask Baldrick the WebMaster of www.VCDHelp.com he had the same problem you are in rite now...he did alot of stuff to compress his pages to lower his Bandwitdh usage..he might help you

here is his Email: baldrick@vcdhelp.com

[DarkAvenger]

I just wanted to suggest something similar: Optimize the webpage (ie: show less days of news on the front page, people coming regularly will know yesterday's news...), kick out large graphics (esp your articels with divx comparisons...) or subst them with highly compressed versions (is jpeg2000 free for use?). Find out which pages are clicked most often and esp. try to optimze them. And so on...

Esp the html code of the forum looks very ugly, but that's porbably something you can't do much about it.

I furthermore think it isn't too bad to *not* host software locally. Unless there are many complaints, that a site is down it is just about three clicks more to get a soft from the official home. YOu could tryx to use a link checker and if a site is down make the soft available locally in a temp. manner. (Furthermore one could tell the authors to code and compile more efficiently, as I just helped DSPGuru a bit with compilation. I just thought about this ac3fix programm for example: The dl archive was about 200 to 300kb, and when I compiled it it came down to about 32kb...)

[dkarl]

Ok guys, time for the formal apology...

I just noticed what was going on earlier today and have cleaned up the situation. Doom9, I shot you off an email a few hours ago... Didn't see the link to this message until just now, but thought I should let the world know:

1. I'm still alive.
2. Although Swan laid off most of their American work force in a blind cost-cutting move, the server is hosted by my other job (the one I can't get rid of without dying : ) So the site's future is safe
3. Things were down 'cuz I was stupid and let the machine run out of disk space when I added more RAM and the swap file expanded (oops.) Although this happened several days ago and was corrected, the FTP server's user file got blown away and screwed everything up. In any case, it's all back up for use.

4. Those <nasty name>s at Swan DO TO HAVE A FORWARDING ADDRESS for me. They had us all turn in other names the day we, err, "left" and told us ALL our mail would be forwarded. That was a blatant lie, as obviously they are reading the mail coming in but not sending it back out to me. Nasty part is I know the idiot who's doing that - and even if HE doesn't have the list, I KNOW the guy down the hall does, 'cuz I talk to him every other day or so.

<rant>It's probably better off that I don't / can't work for them anymore, the company is a lost cause. They have a good idea for a product (Electronic Software Distribution) but the implementation absolutely sucks. Like alot of companys they promise the customer TONS and deliver NOTHING. Look at www.swan.fr - they still list Wrigley, St Laurents Cement, and Ford as customers - but they dropped the product long ago. They doctor the books, make unauthorized press releases (Bank of America specifically said "Don't make one" but there it is, right on our site, and on Easdaq's!). Announe revenue before the account is even signed - and apparently our client can "install itself automatically on all the workstations on your network". Now, besides the fact that most of us know "That's almost impossible, but I can come up with a way to do it", Swan didn't even bother to come up with a way. Their answer? The NT domain login script. Yet this is marketed as a multiplatform product. Idiots. </rant>

Anyway, sorry again for the trouble I've caused. I'd like to try to keep a close eye on the server, but obviously trying to figure out how to pay my rent and car and support a wife and two babies right at Christmas has been taking up most of time....

Hope I can be forgiven. I'd love to keep running the mirror...
Sincerely,
Dave

[dkarl]

Oh one more thing - as far as bandwidth consumption goes, I know it's part of PHP but I think there's also a separate filter that can be installed to automatically compress the pages being fed out to browsers that support that... PHP uses zlib I think... It won't make alot of difference, but might be something I can turn on...

Let me know.
-D

[mabe]

Hi,

i was having a quick look over the html code. You use return and spaces for better reading of the html code. but this just makes it bigger. its not that much but the start page before got 27 980 bytes and after 24 780 bytes, without returns and spaces. Another thing when the translations are finished you can put them on seperated servers.

greetings martin

ps.: thanks for the good work

[cofferscuffs]

I got another idea, it's a bit extreme and might not be welcomed much but, don't allow guests on this board, only registered members. Even though this will turn the board into a closed community it will save some (a lot more like) bandwidth. Just by looking at users online you'll see that the number of registered users online = the number of guests online. And after all it is only a temporary measure and can be reversed at anytime.

[resonator]

You could cut a lot of traffic if you'd get rid of recurring tags in the html code, like f.e. <font face="Verdana, Arial, Helvetica, sans-serif" color="#333333">. Use a sytlesheet instead. For example the news page has a lot of them. Applying a simple <table style="font-family: Verdana, Arial, Helvetica, sans-serif; color: 333333;"> would help a lot, as all the tags within that table tag would use that font-family.

I also noticed that the external stylesheet (doom9.css) could be "compressed" from 379 to 315 bytes. BODY{FONT-SIZE:10pt;COLOR:#333333;FONT-FAMILY:Verdana,Arial,Helvetica}A{COLOR:#b75a3e;TEXT-DECORATION:none}A:link{COLOR:#b75a3e;TEXT-DECORATION:none}A:hover{COLOR:#d59380;TEXT-DECORATION:none}A:visited{COLOR:#b75a3e;TEXT-DECORATION:none}TD{FONT-SIZE:10pt;COLOR:#333333;FONT-FAMILY:Verdana,Arial,Helvetica} That code doesn't win a beauty-contest, but it works. At 37,000 hits a day these 64 bytes alone would cut 60megs of traffic a month! Imagine that.

Also, you could use some "HTML Compressor" proggie. All they do is strip empty spaces and returns, as mabe suggested, but they do it automatically, so you could keep your own code for editing at home, but before uploading speeqze some bytes out. I'm pretty sure that save some 100 megs traffic a month.

Another thing I thought about is your plan to implement some sort of PHP-Content Managment system. My suggestion would be that by all means don't use a database. Use a system that generates html-files instead.

The other thing is, you could at least make people aware that they could also use the alternate download locations, as for example the authors website, to get their stuff.

Don't mistake this for critizism, I just wanted to make some suggestions. I see a lot of possibilities. Need help or one to do the work? Lemme know.

[DarkAvenger]

@resonator

Hehe, I thought the same, but I didn't want to go too much into detail. Furthermore I dunno if Doom9 can play around with the forum soft easily.

[resonator]

Well, VBB hacking can be a tremendous pain in the ass. I was talking about the html code only though.

[Doom9]

about html optimization. When I first switched to dreamweaver I ran the 100odd pages thru the optimization tool and already got rid of a lot of bad code.. but the stuff remaining would require manual tinkering.. on 229 html pages... and I don't see that happen any time soon. And I think the tables would require additional css tags or they'll look different (apparently html treats normal paragraphs differently than table content)... anybody up for the job of manually optimizing my site for less bandwith consumption?

[mabe]

there is a nice little programm which cuts out unused returns and so on. i will post it if you want. furthermore i can have a look at the code. i will contact resonator maybe we can work together.

martin

[Doom9]

sure.. post it (presuming it's legal). and it certainly would be great if somebody could have a look at the code. I thought dreamweaver creates rather clean code but you never know.. and there might still be a lot of nasty things from the import from frontpage even though I spend hours and hours manually editing out stuff that just didn't look right.

[mabe]

Here is the url,

but it is shareware. I like it because it works on multible files.
http://gallery.uunet.be/Jacobs.Jan/htmlcomp/index.htm
I will have a look at the code next days.

so long

martin

[resonator]

You know what? I'll see what I can do. Could take a while though, but worth a shot I guess.

[Kaizen]

There are many ways of reducing bandwidth on a forum; but almost all increase server load.

I could try the style idea. I dont see why it wouldnt work; yet.

I could remove all the images and replace them with text ?

What u think ?

[MxxCon]

what about distributed file sharing?
edonkey2000 looks like a workable solutions..you can give url link directly to specific file. there's linux version of edonkey2k server and client so they could run on web servers..

another solution would be same as deviantART's..when you go to their page you get small popup asking if you want to join their distributed network. it caches images and pages on user's hd and then create something like gnutella network.

update: got url for that popup (thanx google:))
http://www.deviantart.com/shared/swoosh-install.php

ya, let's try out text only style on msgboard

[mabe]

text style would be ok. if it is more simple it is also faster. we can also think about (for the future) to put the translations on extra servers. What about this ????


greetings martin
5cm of snow and spain is in chaos :p

[InSOMniA]

i have 10 MB space (providet by t-online.de, as you
see in germany) i could upload some
tools like virtualdub (must be legal sorry) and images ...

traffic is no problem .. no traffic limit on this
account >:-)

if you are interrested mail me ...

InSOMniA

[Kaizen]

Originally posted by InSOMniA
i have 10 MB space (providet by t-online.de, as you
see in germany) i could upload some
tools like virtualdub (must be legal sorry) and images ...

traffic is no problem .. no traffic limit on this
account >:-)

if you are interrested mail me ...

InSOMniA

I am willing to bet a lot of money that they close your account if it takes up a lot of bandwidth but thanks for offering.

[InSOMniA]

Originally posted by kaizen


I am willing to bet a lot of money that they close your account if it takes up a lot of bandwidth but thanks for offering.


do you use clonecd ? download server 3 is space such i have .. and clone cd creates a lot of traffic ;-)

the newer t-online accounts have a traffic limit of 2 gigs per month .. my not :)

but if you want we can try it with some small apps which will not produce multible gigs of traffic ...

InSOMniA

[Kaizen]

Ask doom about this....

I only manage the forum.

[InSOMniA]

i've mailed him ...

he answerd i should post my offer into the forum (what i've done) ... don't know why ;)

InSOMniA

[mabe]

this could have advanteges. If there are several servers and one is down the rest would be not afected. the bad part is that are a lot of responsible persons.
mabe

[Doom9]

@insomnia.. I'm unable to find such a mail..

and while 10mb is nice.. could you imagine having 100 software files on 100 different servers?

[cofferscuffs]

Hmm that would be me who told him to post in the forum, the email should be in your indox now Doom9, sorry for the slipup.

[InSOMniA]

Originally posted by Doom9

and while 10mb is nice.. could you imagine having 100 software files on 100 different servers?


it's only 10 mb sorry .. but for programs which are not beeing updated so oftem (like the divx4 codec .. i'm currently getting grey hair during waiting for the 3dnow optimations ;) ) it is ok imho.

on my website i used an anti leech script to manage all download urls,
the dl was always called in such way ...../cgi-bin/dl.pl?lame or so .. its some time ago i used it ...

all dl links were stored in a file and could be changed very fast.


InSOMniA

[SometimesWarrior]

I was just testing the graphics used for the guides, to see if they could be compressed further.

I used Fireworks 4, which has given me good results in the past. Just about every graphic lost 1Kbyte in size simply from being re-saved with Fireworks, even though the GIF image itself was not altered in any way. I'm guessing that Macromedia's software saves the GIF's with a more efficient header?

I also lowered the size of the palettes on the graphics with gradients and icons, and used some dither so the image quality didn't degrade much (average savings of 2K per image). And on the images with grainy windows from needless dither, I modified the palettes (cutting filesize by over 50%!).

So far I've done the Gordian Knot and VirtualDub guides' graphics, and decreased the total graphics files from 230K to 171K.

My question is: shall I continue to optimize all the guides' graphics (it would only take me a couple of hours) and then send them to Doom9? I'm betting it could cut down bandwidth by over 10% for the guides, and if someone else added HTML compression, it would complement this effort.

I've already made my donation, so rather than give away more of my meager student-budget money I thought I could help in some other way.

-SometimesWarrior

[MxxCon]

check out image optimizer over at xat.com
it is the best image compression program i ever used.
.png 256 colors blows away most other formats and look as good as jpg at 99% quality.
you can even do batch processing on all images.

[SometimesWarrior]

Very nice program! I'm trying out the demo now.

Its files are a bit smaller than Fireworks', even. The palette adjustment wasn't quite as good as Fireworks', but even when I raised the palette size to remedy the color problems, its files were still a bit smaller. Impressive! Even its dither looks a bit better than the one in Fireworks. Not bad for a piece of software that costs 1/5th as much...

[Doom9]

well.. image and html compression is a start.

and btw.. servers are not in synch yet again.. actually one is pretty much down (or completely overloaded). admin has been notified.

[SometimesWarrior]

After going through all the Divx4 and Flask 3.11 guides, I reduced image file sizes from 1.93MB to 1.1MB... that's a 40% reduction. The quality of the images was largely unaffected... most of the gains came simply from optimizing the palette and by using (or not using) dither.

Depending on how much of the site traffic is from the guides, this kind of optimization might make a noticeable difference on total site bandwidth. If nothing else, it should make the guides load a bit faster for modem users.

I sent a link to the submission email address, where one of the siteops can download the re-compressed images. The email's header is "Compressed graphics"... Tell me what you think.

-SometimesWarrior

[Refthoom]

So far I've seen several options aimed at the technical side, which could really help:
- optimising html code:
Making 'smarter' use of CSS and tags
- comressing html code and images:
Using tools
- placing softs on different servers:
By regrouping the softs and placing them per group,
would avoid having to maintain hundreds of servers

Now, all of these together would greatly reduce bandwith natuarally. But something else could also help: improved navigation.

atm every click leads to a page opening. On that page you click to the next page, etc. By using unfolding menus that follow the structure of, for instance, the guides, one could get directly to the page of interest. This would be relatively easy to do, since it only involves the left menu-frame and it could greatly reduce the load on the servers, without having to change anything to the rest of the site.

If this looks interesting, let me know and I'll see what I can do to help.

[mat813]

I currently admin 1 of the 2 servers, to give you some stats, here are transfer generated on my server :
august : 160GB
september : 159GB
october : 185GB
november : 171GB
december : 177GB
which leads to about 5.5GB/day

I've switched http transfers to ftp transfers on my server, it will lower the load on apache.

[mat813]

and for all of the optimizations proposed, it's of no concern, what was making my server crawl was downloads, http is not designed for download, ftp is, since I've switched, the load got back to 0

[Refthoom]

I'm glad the load on the servers is better now. But changing http to ftp did nothing for the amount of data being transferred, which was one of doom9's worries from a cost perspective. Or am I overlooking something?

doom9:
Besides generating less traffic, I would still *like* improved navigation with i.e. unfolding menus ;-)

[mat813]

ok, let me explain why putting downloads via ftp solves my (which is not bandwith but load).
to serve /
you need to serve :
/top.html, /left.html, and right.html, that's 4 requests (not counting the counter) just to get the home page, these requests takes less than a second to be served (the whole) but each needs 1 free httpd.
if you have 1 guy leeching 1 .zip file with some crap software like gozzila or something, it will be divided into 4 or more parts to go quicker (and so, use 4 httpd just for 1 real request).
Now, the problem :
at the time I'm writing this, there are about 40 downloads on the doom9 ftp. which would be 40 httpd in the best case to 160 in the worst and there are also about 60 requests to web pages on doom9 (and about 80 for the rest of the web sites being served), that's 120 httpd, which is easily handled by the server (dell poweredge 2550), but, if you add the other part it goes up to 300, which would be 300 process at the same time (not verry crowdy tonight), at about 600 process, which have occured a lot of time recently, the server just pops off...
now, another thing, a ftp process takes a bit less than 1M of memory, an httpd, between 4 and 6M (count the difference when you have 80 ftp downloads instead of 80 http).
that's http download that kills everything, not bandwith, really not :)
hope that this was clear enough, it's late, and my english is getting worse and worse

[mpucoder]

@mat813 What server software are you using that takes so much resources per thread? Sounds like you should look around for better software.

[smiller667]

@mpucoder:

Originally posted by mat813
[...] it will lower the load on apache. Guess he's using Apache ...

[Doom9]

about improved navigation: can somebody do that for me? I'm not really experienced in dhtml.. and keep one thing in mind: even though more than 80% are using IE the site still has to work in Netscape and Opera. We don't want to exclude people using an alternative browser. It's bad enough that the Netscape / Solaris combination doesn't work for the forum.

[Refthoom]

I'll be glad to help in making the menu.

I guess we'll have to exchange some more infos before I can make the menu. Do you want to do that through this forum or do you want to use email?

[Doom9]

email please

[Refthoom]

doom9:
I mailed using your ifrance account. Not sure if this is the right one, but recieved no reply yet. Otherwise, pls send me an email at refthoom_nospam_@yahoo.com (remove _nospam_).
tx.

[Doom9]

just a small update: there's now 3 production servers with another one standing by and I'm looking into more options. Also.. I've added a failover service so if a server won't return a html page upon requesting the index page it will be removed from the round robin.