Forums downtime Dec. 2 - 4, 2006

Tek7

CGA President, Tribe of Judah Founder & President
Staff member
First, let me thank everyone for their patience while hescominsoon, SirThom, and I worked to attempt to repair damage to the Christian Gamers Alliance forums database. I hope everyone had a restful weekend and had time to rest their typing fingers. Now that the forums are back up, I expect many of our members will be looking to make up for lost time.

Second, let me assure everyone that no forums were altered. With the exception of two accounts (my former account, now labeled Tek7 (Legacy), and hescominsoon), none were altered in the course of events.

Third, many of you are likely wondering why the forums went offline suddenly and stayed offline for approximately 48 hours. What follows is a sad, but true, story.

Late Friday evening, hescominsoon was working on account management and accidentally clicked to prune his old posts. Approximately 3,000 of hescominsoon's posts were deleted from the forums. Approximately 1,300 of his posts were transferred to my account, then labeled Tek7 and now marked Tek7 (Legacy). Among those posts deleted were essential forum documents such as the forums Terms of Service.

When the problem became apparent, we immediately closed down the forums. If there was any need to roll back to the latest backup, we did not want users to make any posts that would disappear after the rollback. We searched the dedicated server on which the forums run and could not find any backups of the forums database dated after Oct. 30, 2006. It turns out that the automated backup script that had been scheduled to run each night had been bombing out since then and had not given any warning or errors.

We were left with a difficult choice: Leave 3,000 posts deleted and 1,300 transferred to another account or roll back the forums one full month.

SirThom, hescominsoon, and I attempted to create a third option: save the Dec. 1 database, roll back to the Oct. 30 database, and "graft" on posts added between the two dates. The theory had merit; SirThom and hescominsoon invested several hours each this weekend trying to make this post "graft" a reality.

Unfortunately, after several experiments, a day and a half of downtime, and a support ticket to vBulletin developers later, we concluded that the post "grafting" was not possible.

Left with two poor choices, we decided on the "lesser of two evils," and decided to lose 3,000 of one user's posts rather than lose one full month of community activity. It was a terrible choice to make, but I can assure everyone that every effort was made to explore alternative options.

Again, no other user accounts or forums were altered. The only difference most users will see is that I am now using a new Tek7 account starting with a zero post count today. All my old posts were retained, but are now credited to the account Tek7 (Legacy).

Other than that, all is well. I apologize for the extended downtime and hope all users can forgive the inconvenience. Rest assured that scheduling a reliable daily backup for the forums is a top priority. Once we configure a daily backup test, and in case of database problems, we will be able to roll back the forums a single day instead of having to choose between "living with" the damage or rolling back a month or more.
 
Last edited:
Back
Top