Jump to content

Data Loss Announcement

Admin

5,640 views

First, so you don't have to read through all my rambling, the website database is set back to November 2010, and there is no way around it.

 

I was working with the forum software on Sunday, 4-17, and I was just very pleased with how things were working. I saw there was an update for the bug tracker application, and the change list just mentioned some bug fixes, so I felt okay installing it. The new version removed custom issue fields, which we use extensively for listing hardware, drivers, operating systems, etc.

 

It's not very easy to revert to previous software versions. I had performed a full website backup earlier that day, so I figured the best way to revert the tracker app would just be to have the IT admin revert the server to the backup I had made. I'd downloaded it to my local hard drive and it was intact, and also still available on the server. I also had an earlier full website backup I made on April 2. I figured this would be a minor inconvenience that would take the forum offline for a few hours.

 

Here's the interface in CPanel:

Image2.png

 

I also talked to the IT admin and they assured me that the full website backup does includes MySQL databases. I sent the following message to them Sunday afternoon:

There is a backup file on my server called “__________.tar.gz”. I would like my domain www.leadwerks.com restored to the state it was at the time this backup was created.

 

Please note we have several MySQL databases and some complex forum software installed. I need the backup to restore everything it to its previous state at the time the backup was created. I chose “Download or generate a full website backup” in CPanel > Backups to generate this file.

 

My caution in this email might indicate I had a suspicion something was wrong. Did I somehow know something would go wrong? Should I have? This thought is nagging me right now.

 

I've been on the phone with tech support for a couple days, but you don't need the whole story. The outcome is the server was erased and repopulated with the contents of the backup archive. All files were intact, but the databases were not saved when the backup was performed. The previous backup I had was from April 2, which would not be a terrible loss. It appears the same occurred then. In fact, since I have been dutifully saving full website backups since November, thinking I was making extra effort to be on the safe side, the databases were never being saved in the archive. So the last copy of the forum database I have is from November 2010. I've been quite diligent with backing the site up, but that doesn't mean anything if the backups aren't working. The host automatically backs up sites under 10 gb every 24-36 hours, but that size excludes us.

 

Although it was easy to bring the forum software up to date, all forum data since November 2010 is gone forever. The loss of this is staggering. To me, and to others here.

 

You can get your screenshots here in a single package. They're randomly named, but if you really need something you can find it, and it's divided up by month:

http://leadwerks.com/werkspace/index.php?/topic/3098-screenshot-archive/page__pid__28584#entry28584

 

I still have all the attached files from posts, blogs, downloads, etc., but they are randomly named and I don't know how possible it is to go through them.

 

The thing that really hurts is the lost documentation that was created after November. That's going to take time to recover from. We'll recover, but the loss is still sinking in for me. Jorn worked really hard on that at a reasonable pay rate and I feel really bad about it.

 

I have records of all registered users. If you registered for your Leadwerks account after November of 2010, your account will be recreated and your password will be set to your registration key you received.

 

The wiki and old forum are fine.

 

The new forum skin is fine, but not installed at the moment.

 

This must never happen again.

 

First, a better backup protocol is needed. The most important thing is that the backups actually be valid backups. Obviously, I've learned that MySQL databases must be backed up individually and that a full website backup from CPanel is not reliable. Data is saved to a 2 TB external hard drive. Additionally, I am opening a safe deposit box where code and site data are regular deposited. I already thought I was doing everything right, so at this point I feel like I can't be too careful.

 

Second, I am locking the forum software at the current versions, unless a critical vulnerability is discovered. In 2009, all we had was an installation of PHPBB. The idea for Werkspace was something Annika came up with and gradually sold me on. We listed all the features we wanted and found forum software that would allow it. Customizing the look and feel of the software to fit in with the theme of Leadwerks was a long process, but we finally got a suitable skin made. I am satisfied with the features and functionality of the system, and now we can just leave it be, indefinitely. The data loss does not affect the skin we had developed. The skin will continue to get minor improvements, but those kind of changes are easily performed and rolled back, if need be.

 

Third, semi-annual tests are needed to make sure data can be restored successfully to a test server. Backups don't help if you wait until you need them to find out they don't work. A test server will be used to install the latest backup on every six months, or before any restoration takes place. Maybe this is overkill, but nothing was supposed to go wrong before, and we found out otherwise.

 

It seems redundant to say I'm sorry about this occurrence. I know we'll get back to normal, but it hurts pretty bad. The key point is my data backup technique was flawed, so all the backups I've been performing were worthless.

 

-Josh



41 Comments


Recommended Comments



It's really my fault. For something this important, I should have assumed the backup system would fail.

Share this comment


Link to comment

You should install a test server where you can play around and test untested things like backups and restores. And you should have a fallback server too, which has always the last good state. Even a $100 netbooks are good enough to act as test and fallback server.

 

And not to mention that with Domino this could never happen, since there data and design is seperate, and you can just switch take a backup of a design, replace it with another and switch back to the last good design anytime. It wouldn't happen with a self-made PHP/MySQL app either, since you are in control what is changed, but these ready-made boards are all ****. Also CPanel is ****, and I've never understood why it's even needed, as you can do much more and much easier things directly with SSH command line, or you can make your own CPanel with PHP or LotusScript.

 

And I wouldn't trust any application level backups. Always take a OS file system level backup.

Share this comment


Link to comment

You should be able to restore the sql databases through phpMyAdmin with your full backup.

Create blank database the same name and import the .sql file for the database from the mysql folder in your full backup, also import the mysql.sql from the root of the backup this adds the user privileges.

You may be able to import into the the current databases your using now but I have always started from and empty one.

Share this comment


Link to comment

You should be able to restore the sql databases through phpMyAdmin with your full backup. Create blank database the same name and import the .sql file for the database from the mysql folder in your full backup, also import the mysql.sql from the root of the backup this adds the user privileges.You may be able to import into the the current databases your using now but I have always started from and empty one.

I should, but the full backups aren't full backups. They are all missing the databases, even though they are supposed to include them. Otherwise this would be no big deal.

 

All developers' accounts have been reinstated. Email support at leadwerks dot com if you cannot log into your account using your registration key as the password (you can change this to whatever you want).

Share this comment


Link to comment

Yes. I have the files that were uploaded, but they take some effort to identify because they are just randomly named with no extensions. The application is reinstalled and I can set everything back, but the data in the system is gone. I know, when I realized what had happened I was shocked and devastated. Now you guys are all feeling that.

 

I am looking through the backup archives, but since the most recent two did not contain databases, it seems likely CPanel just never saved them.

 

If anyone can recommend a fast .tar.gz viewer, I'd appreciate it. I'm using WinRAR to view them and it's really slow.

Share this comment


Link to comment

Josh have you tried 7zip ( http://www.7-zip.org/ ) it can open all archives very quickly, just remember to set the manager so it is associated with the archive.

 

If you need a hand going though the archive I could always help if possible :)

Share this comment


Link to comment

ah that would explain why they are all slow thats one heck of a archive :) I dont think there is anything that will open then quick as the index is going to be huge.

Share this comment


Link to comment

I am copying files off a USB hard drive right now. It will take hours, and it is 1 AM in California, so I am going to sleep. If I am able to locate any copy of the database it will be good, but it seems very likely to me the CPanel full site backup has been failing to save databases this whole time.

Share this comment


Link to comment

Opening a tar.gz of 12 GB shouldn't be any problem with WinRAR or 7Zip. The initial loading/opening might take 30 minutes (if not more). But afterwards you'll browse through the files quite easy.

Share this comment


Link to comment

It’s odd I use the fullbackup option when transferring new clients sites to my server’s, and restore them through my WHM panel, you can’t do it through cpanel, but never a file as large as yours. I have seen cpanel backup dump database tables into individual binary files but has always restored correctly through phpMyAdmin, I have my fingers crossed for you.

Share this comment


Link to comment

All UI packers are slow, but the command line is fast. So if you just search for some certain file pattern, like *.gmf it will be very fast to list all gmfs in a .tar.gz or any other archive format.

Share this comment


Link to comment

Tell your host to restore a previous backup. Most hosts do daily, weekly and monthly backups of the whole server. Sometimes they'll charge a small fee for the restoration.

 

If they don't do that, drop your cheap host, and get a real one.

Share this comment


Link to comment

I wouldnt say its all your fault josh - as a user we need to expect that the software is configured properly. In case of a backup i too would expect the default setting would be "all-and-everything".

Got me tempted though and i might have to put a bbord up myself, just to work with it.

I hope its not too bad for lw beginners ... at least there is the wiki, the set of tutorials, the user-manual and the C api manual from sep2010.

Share this comment


Link to comment

Josh, I think that you should first of all stop thinking about the past and turn toward the future. It seems to me that it is not worth wasting your valuable time to searching the archives. Probably the biggest pain is the loss of documentation for LE2. The rest is not critical. Asset store files, screens, blogs, posts, etc., many of these things are made by individuals from community, and perhaps some may be (in some cases) restored by their authors (if this will be really needed).

 

I hope, that this minor setback with Werkspace, don't affect your great work with development of Leadwerks Engine (I believe that the engine sources are backed in a better way). Anyway, chill out and don't worry too much. :)

Share this comment


Link to comment

Josh, I think that you should first of all stop thinking about the past and turn toward the future. It seems to me that it is not worth wasting your valuable time to searching the archives. Probably the biggest pain is the loss of documentation for LE2. The rest is not critical. Asset store files, screens, blogs, posts, etc., many of these things are made by individuals from community, and perhaps some may be (in some cases) restored by their authors (if this will be really needed).

 

 

 

Actually that's quite insulting to the community I think, which is made up of many individuals who have spent many hours of their valuable time contributing to it, with "Asset store files, screens, blogs, posts, etc." and much more, but as none of that was critical I am sure they can now take it easy, stop uploading files, writing blogs, posting, helping with technical support, writing tools, headers and tutorials. :)

Share this comment


Link to comment
Actually that's quite insulting to the community I think, which is made up of many individuals who have spent many hours of their valuable time contributing to it (...)

I had no intention to offend anybody - just my poor English is not able to express themselves more subtly. Josh seems to be very worry about this accident. This is obviously a sad thing and I just wanted to give him some encouragement.

Share this comment


Link to comment

Yep that's quite disturbing but "s..t happens". Like any good community we have to help a bit to solve problems together as much as possible, to upload again what we can upload :)

 

Especially it is disturbing for you guys who have contributed a lot, I am grateful to you. I am relatively new member, but I'll try to contribute more in the near future.

 

Something Positive :

Deja vu :) Looks like Diving in the past :) Jane Croft site + release date :)

 

I'm young again :)

Share this comment


Link to comment

A few questions (and sorry if these were covered in some form):

 

What do we do about previously uploaded assets? It sounds to me like you're going through them and will see how reasonable it is to reupload them. Is that right? I ask because I have a saved copy of a file I uploaded (Leadwerks Joystick/Gamepad Example). I assume I should hold off on uploading to not mix things up.

 

Why is the documentation lost? Was there only an online copy? I would have imagined that Jorn would have worked on it offline and then uploaded it when he was ready to, so he would still have a copy offline which he could reupload. But it sounds like this isn't how it worked. Just curious. You made it sound like it needs to be recreated pretty much from scratch (for entries created after November).

 

Thanks and best of luck with painlessly restoring as much as possible!

Share this comment


Link to comment

In my case all is OK Josh, so this happennning at each of us i thinking, and many times.

Share this comment


Link to comment
Why is the documentation lost? Was there only an online copy?

It was created and edited online, but I've been making lots of backups. The problem is the backups don't do what they say they do. There are no options in CPanel to include only parts of a full backup. It's explicit about what it includes, and tech support assured me on two separate occasions the databases would be saved. So I felt pretty good about using it. Even if one archive somehow got corrupted, I had full backups going back for months.

 

The user data lost is very important. This isn't just a forum. People rely on it to host a lot of images, files, and their writing. I hope the protocols I enact will convince them to trust the site again with their data. I'll talk to my uncle about what steps should be taken.

 

My hosting company is actually very highly rated and not cheap, and they've done everything else very well. I won't mention their name for now, but they seemed remarkably unconcerned about this. Their reaction was "Hmmmm, that's interesting, I don't know why the databases aren't being saved."

Share this comment


Link to comment

Well, sometimes this sort of thing just happens. Data is lost and that's it. Hopefully this serves as a reminder to rest of us to backup our stuff.

 

*cough* stuff... I really must set up my own backup system.

 

Hmm, what is a good way to backup mysql stuff?

Share this comment


Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...