Operation Enduring Downtime

Update 02-12-2001

As some of you may or may not know, we've apparently been being read by the US secret service as a way of keeping tabs on citizens who voice opinions which are "too dangerous." At least that's what the poster claims. I'll try and verify this once we have our various bits and logs off of the raid array. If it is true, I urge that if you are a citizen of the US, you encourage your leaders to look to other countries that live with terrorism (such as Northern Ireland), and teach them how those people can fight terrorism without also fighting civil liberties.

In similar news, it seems that despite the objections of many respected politicians, the Canadian government has passed similar liberty limiting measures, and may pass more.

-- Inoshiro

More: Ok, I didn't really want to get into this while the site was down because it's likely to cause some (probably undue) concern, and probably isn't all that big a deal.

I've talked to Lee, and believe that he's not making this up. The comment in question was this one, attached to the smallpox story. (Google cache is the best). Basically, Lee was visited at work by a couple of Secret Service agents, who asked him some questions about the comment, in an attempt to ensure that it was not a credible threat. He was a little frightened, but I don't think anything came of it.

While I have great doubts about the wisdom and cost-efficiency of following up a comment like this, given that I have a really hard time imagining what kind of person could actually read that as a threat that warranted investigating, it doesn't seem like Lee's civil liberties were violated in any way, and he wasn't dragged off to some secret chamber for the third degree. Our government is doing a lot of things we Americans should probably be worried about, but I think this is just dumb, not actually evil.

My point is, I don't think this is anything you need to be worried about. More specifically, at least one person has already emailed asking if all their comments and whatnot can be removed,. The answer to that is no. I'm planning to add a "remove me" button in the very near future, which will remove all of your personal information from the site (including retroactively anonymizing all your contributions), but we won't be mass-removing comments or stories. It's just a bad idea.

This is already way too much detail for a temporary status page, so I'm going to leave it at that. Please, please, try to hold off until the site is back, at which time I promise to have a full report, for general discussion. If you all email me individually about this, I probably won't reply anyway. :-)

--rusty

Update 01-12-2001

The new machines are in at Voxel.net. For those of you interested in technical details, the new network will consist of: That's the initial setup. The plan is to eventually put a lighter machine in front of the whole thing, as running LVS doesn't require a lot of power, and move hex into the Scoop tier, and also to add one or two more database servers, which will act as read-only slaves via MySQL replication. So everything is more or less on-track then, although we don't have the data from bubba yet.

If you're bored, MetaFilter is doing a special issue today, focused on World AIDS Day as part of Link and Think. It's worth a look.

-- rusty

Update 29-11-2001

I'd normally post a diary, but K5 seems down at the moment. Any K5 people wanting to meet Inoshiro and who live in Seattle or San Francisco will get a bit of a chance to see me on the 13th and 18th of December. I'll be flying through on the way to a bit of a family re-union (seeing dad for the first time in 8 years). I'll keep the updates in my diary from now on ;) K5 will hopefully be back around next week once we get the DB yanked off the RAID.

-- Inoshiro

Update 11/28/2001 (#2):

The verdict is in. Bubba is dead. Let us all have a moment of silence.

 

There. So what happens now, you ask? First, we get the disk array from bubba back to Compaq HQ, where our ever-helpful friends figure out how to get the data off of it. No, we don't really have useful backups. No, this will not be a mistake we ever make again. The data is fine, it's just locked inside a RAID array that we can't mount anywhere else. Compaq HQ will have the right gear though, so that's ok.

Meanwhile, we proceed full-speed ahead with the transition to the new host. They should be ready for us by Friday, at which time we'll begin the configuration of the new machines for maximum K5itude. Early next week, we expect to have the data back from the late bubba's disks. As soon as we're more or less satisfied that things seem to be working, the site will come back up.

I don't want to write checks I can't cash, so allowing for problems, I'm going to estimate next Friday as the return date. It ought to be sooner, but something always goes wrong. Until then, I will update here as warranted.

-- rusty

Update 11/28/2001:

Hi everyone, rusty here. I wanted to let you all know what the status is.

The good news is, right now, Jason from VHosting is at the datacenter with new processors for bubba. If they work, we'll be bringing bubba back up immediately, and you'll all have K5 back as soon as DNS propagates.

The bad news is, if they don't work, we won't be back until, probably, the middle of next week. Ultimately, this will work out for us -- here's the situation. I was contacted by Voxel.net a few days ago, with an offer to take over hosting. We have no problem with Jason or VHosting, so I doubted it would be a good idea. But as it turns out, Voxel is willing to set us up with three new servers as well as 500Gb/month of bandwidth, in exchange for the same small ad space as currently taken by VHosting. This was simply too good to pass up, so we will be changing hosts next week when the new machines are in place. This is not a reflection on VHosting -- this downtime is not Jason's fault at all, and we've always been thrilled with the level of personal support and all the hard work he's put in keeping us up. As I said above, the new offer was simply too good to pass up. I think Voxel will be a good move for us, and will be able to help us ensure a more reliable K5 down the road.

The new K5 server network will therefore consist initially of hex (our previous K5 server, a dual P3 700) running a static Apache and taking all outside connections, which will proxy Scoop requests back to two dual P3 866's which will be load-balanced and failover capable, and run Scoop and the mod_perl Apaches. These, in turn, will fetch data from a database server, another dual P3 866 with RAID storage.

So, from the start we'll be a lot more fault-tolerant, with redundancy at least in the middle layer of machines. But it gets better. As many of you know, certain subversive elements within Compaq have been supporting us with hardware for a while. We're negotiating right now for official support from Compaq's Linux division, who will provide us with a beefier database machine (quite possibly an Alpha), and maybe some Scoop servers as well. Any new hardware, in the new setup, will automatically be redundant, meaning that we should very soon have a fully redundant system which would require several machines to fail before K5 service was interrupted at all. Needless to say, this will be sweet.

Getting back to the present, if bubba fails to come back, we're planning to abandon it entirely and go straight to getting the data off bubba's disk array, and bring the site back up on the new network. We should be ready to start testing and configuring the new setup by Friday of this week, which means we could probably go live with it by the middle of next week.

So that's where we are right now. We'll either be back in a few hours, or definitely next week. I'll post another update when I know for sure. Also, current subscribers will have however many days we end up being down added to their subscriptions.

Meanwhile, several people have asked me to link to some other fine Scoop sites, for your enjoyment during this horrible time. I would recommend Satanosphere and Anarchy Online for general discussion, Smokedot if this downtime is driving you to substance use, and Adequacy.org for those of you with a very strange sense of humor. More Scoop sites can be found here. Just don't go getting all comfortable elsewhere, now. We will be back before you know it. And thanks for all the fish.

-- rusty

Distraction link of the day: the DrFun archives! Thrill to the Farside meets Dillbert of the comic :)

Kuro5hin has been down for a few days, and will probably continue to be down for a few more for the reasons I'll reveal here. First off, we've had some colo problems. VHosting is great, but VHosting can't control it when the people who actually have the rack/cage space decide to move (without telling anyone) a hundred or so of their customer's servers to another facility. Naturally, it takes a few days for all the BGP routing to get adjusted; even longer for those whose ISPs filter BGP broadcasts out.

But it gets better! Bubba is a sensitive fellow. We've had problems before from bad ram, and an unwillingness to bootup somedays (depending on Bubba's mood). Since the hardware isn't within kicking distance, I (Inoshiro) can anly suggest things to Jason or whatever local colo tech person is handy. It's not the best situation. And it's worse since the move as whoever moved it did not take everything out carefully at one place, move it, then carefully put it back together. They just moved the entire thing while it was in one piece. So now, Bubba doesn't work. We've narrowed it down to the CPUs, but we can't do anything about it unless we get some plugs for the slots that'll let us test which CPUs are bad.

In the mean time, Jason has offered some great support. We love VHosting :) But until we can arrange for a new machine, K5 will be on a bit of a siesta. Mail and DNS are fine (those machines are at my house, so I can always slap 'em around and make them work). If we had the "username@kuro5hin.net" thing setup for people to buy, it'd still be working fine -- so don't fret about it :) Services will also be getting a secondary backup machine soon which I'll configure to mirror everything.

In a few days, all of VHosting's machines will be moving to a new colo facility/provider who won't play games with their customer's boxes.

-- Inoshiro

PS: Try "Acts of Gord" for a nice distraction :) Alternately, you could always donate something like this bad boy to replace Bubba ;)