Given a failed upgrade from the vendor on (gs) Grid.Cluster.2’s storage segment, and then a failed rollback attempt, we either had to work to repair the systems so customers would have their “live” data — or recover from backup, potentially taking several days to get fully back online and rolling back some customers to their “last backed up” date. Though neither option was “pleasant”, we had good confidence that the “up to date” data was safe and accessible with the appropriate vendor involvement — this was seen as the best overall customer outcome.  And so after several hours of troubleshooting, we have have managed to repair the systems preventing a restore situation.

We are continuing to validate the auxillary systems of the (gs) — database, database container, containers — at this time. All basic services such as web, email, and FTP have been restored.

We understand how frustrating situations like these can be and we sincerely appreciate your continued patience and understanding.

This entry was posted on Saturday, December 1st, 2007 at 10:13 am and is filed under Electrical Systems Maintenance Notice - Nov. 30th. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.