[Nordskogen] Not-Quite-Catastrophic Server Failure

[Michael Scott Shappe]

It seems that I moved the Gallery none too soon.

The server at Itasca.net that hosts Northshield, Nordskogen, Caer
Anterth and Windhaven's web presences (among many others), their
mailboxes and aliases, and all the mailing lists, has developed a
severe disk problem. The disk is not yet dead, but it is dying, and
some files are already unreadable or error-prone.

As a result, I have spent much of today off from my day-job working to
salvage the services the system runs and move them to other systems.
This migration has been planned for a long time, but kept getting put
off or done piece-meal. Now, needs must and all that.

Since it seemed to be the most effected, mailing list service is the
first thing I focused on cutting over. If you receive this message,
then the cut-over has been mostly successful. I imagine it will still
take several hours for the domain name changes I made to accomodate
this to percolate everywhere, but the basics seem to be working.

As part of this change, mailing list service and web services are
being separated, to help balance out the load.

The big consequence at the moment is to the archives. The archives
were one area of the disk that seemed to be suffering a lot of damage,
and so, I have chosen to restore them from Sunday evening's backup
rather than attempt to copy what might be corrupted data on the disk.
Any messages since Sunday will likely have been lost from the archives
for good.

If this is working, my next task will be to migrate web services for
all Itasca's customers on still using vhost1, and then lastly (because
it's a big pain to do so I'm putting it off as long as I can, and I
admit it), mail services.

List owners of other Itasca-hosted lists should feel free to copy this
message to their lists so they know what's going on. I'm only posting
this to the lists I can think of off the top of my head.

/m