Jump to content

Data Loss Announcement

Admin

5,921 views

First, so you don't have to read through all my rambling, the website database is set back to November 2010, and there is no way around it.

 

I was working with the forum software on Sunday, 4-17, and I was just very pleased with how things were working. I saw there was an update for the bug tracker application, and the change list just mentioned some bug fixes, so I felt okay installing it. The new version removed custom issue fields, which we use extensively for listing hardware, drivers, operating systems, etc.

 

It's not very easy to revert to previous software versions. I had performed a full website backup earlier that day, so I figured the best way to revert the tracker app would just be to have the IT admin revert the server to the backup I had made. I'd downloaded it to my local hard drive and it was intact, and also still available on the server. I also had an earlier full website backup I made on April 2. I figured this would be a minor inconvenience that would take the forum offline for a few hours.

 

Here's the interface in CPanel:

Image2.png

 

I also talked to the IT admin and they assured me that the full website backup does includes MySQL databases. I sent the following message to them Sunday afternoon:

There is a backup file on my server called “__________.tar.gz”. I would like my domain www.leadwerks.com restored to the state it was at the time this backup was created.

 

Please note we have several MySQL databases and some complex forum software installed. I need the backup to restore everything it to its previous state at the time the backup was created. I chose “Download or generate a full website backup” in CPanel > Backups to generate this file.

 

My caution in this email might indicate I had a suspicion something was wrong. Did I somehow know something would go wrong? Should I have? This thought is nagging me right now.

 

I've been on the phone with tech support for a couple days, but you don't need the whole story. The outcome is the server was erased and repopulated with the contents of the backup archive. All files were intact, but the databases were not saved when the backup was performed. The previous backup I had was from April 2, which would not be a terrible loss. It appears the same occurred then. In fact, since I have been dutifully saving full website backups since November, thinking I was making extra effort to be on the safe side, the databases were never being saved in the archive. So the last copy of the forum database I have is from November 2010. I've been quite diligent with backing the site up, but that doesn't mean anything if the backups aren't working. The host automatically backs up sites under 10 gb every 24-36 hours, but that size excludes us.

 

Although it was easy to bring the forum software up to date, all forum data since November 2010 is gone forever. The loss of this is staggering. To me, and to others here.

 

You can get your screenshots here in a single package. They're randomly named, but if you really need something you can find it, and it's divided up by month:

http://leadwerks.com/werkspace/index.php?/topic/3098-screenshot-archive/page__pid__28584#entry28584

 

I still have all the attached files from posts, blogs, downloads, etc., but they are randomly named and I don't know how possible it is to go through them.

 

The thing that really hurts is the lost documentation that was created after November. That's going to take time to recover from. We'll recover, but the loss is still sinking in for me. Jorn worked really hard on that at a reasonable pay rate and I feel really bad about it.

 

I have records of all registered users. If you registered for your Leadwerks account after November of 2010, your account will be recreated and your password will be set to your registration key you received.

 

The wiki and old forum are fine.

 

The new forum skin is fine, but not installed at the moment.

 

This must never happen again.

 

First, a better backup protocol is needed. The most important thing is that the backups actually be valid backups. Obviously, I've learned that MySQL databases must be backed up individually and that a full website backup from CPanel is not reliable. Data is saved to a 2 TB external hard drive. Additionally, I am opening a safe deposit box where code and site data are regular deposited. I already thought I was doing everything right, so at this point I feel like I can't be too careful.

 

Second, I am locking the forum software at the current versions, unless a critical vulnerability is discovered. In 2009, all we had was an installation of PHPBB. The idea for Werkspace was something Annika came up with and gradually sold me on. We listed all the features we wanted and found forum software that would allow it. Customizing the look and feel of the software to fit in with the theme of Leadwerks was a long process, but we finally got a suitable skin made. I am satisfied with the features and functionality of the system, and now we can just leave it be, indefinitely. The data loss does not affect the skin we had developed. The skin will continue to get minor improvements, but those kind of changes are easily performed and rolled back, if need be.

 

Third, semi-annual tests are needed to make sure data can be restored successfully to a test server. Backups don't help if you wait until you need them to find out they don't work. A test server will be used to install the latest backup on every six months, or before any restoration takes place. Maybe this is overkill, but nothing was supposed to go wrong before, and we found out otherwise.

 

It seems redundant to say I'm sorry about this occurrence. I know we'll get back to normal, but it hurts pretty bad. The key point is my data backup technique was flawed, so all the backups I've been performing were worthless.

 

-Josh



41 Comments


Recommended Comments



It's really my fault. For something this important, I should have assumed the backup system would fail.

Share this comment


Link to comment

You should install a test server where you can play around and test untested things like backups and restores. And you should have a fallback server too, which has always the last good state. Even a $100 netbooks are good enough to act as test and fallback server.

 

And not to mention that with Domino this could never happen, since there data and design is seperate, and you can just switch take a backup of a design, replace it with another and switch back to the last good design anytime. It wouldn't happen with a self-made PHP/MySQL app either, since you are in control what is changed, but these ready-made boards are all ****. Also CPanel is ****, and I've never understood why it's even needed, as you can do much more and much easier things directly with SSH command line, or you can make your own CPanel with PHP or LotusScript.

 

And I wouldn't trust any application level backups. Always take a OS file system level backup.

Share this comment


Link to comment

You should be able to restore the sql databases through phpMyAdmin with your full backup.

Create blank database the same name and import the .sql file for the database from the mysql folder in your full backup, also import the mysql.sql from the root of the backup this adds the user privileges.

You may be able to import into the the current databases your using now but I have always started from and empty one.

Share this comment


Link to comment

You should be able to restore the sql databases through phpMyAdmin with your full backup. Create blank database the same name and import the .sql file for the database from the mysql folder in your full backup, also import the mysql.sql from the root of the backup this adds the user privileges.You may be able to import into the the current databases your using now but I have always started from and empty one.

I should, but the full backups aren't full backups. They are all missing the databases, even though they are supposed to include them. Otherwise this would be no big deal.

 

All developers' accounts have been reinstated. Email support at leadwerks dot com if you cannot log into your account using your registration key as the password (you can change this to whatever you want).

Share this comment


Link to comment

Yes. I have the files that were uploaded, but they take some effort to identify because they are just randomly named with no extensions. The application is reinstalled and I can set everything back, but the data in the system is gone. I know, when I realized what had happened I was shocked and devastated. Now you guys are all feeling that.

 

I am looking through the backup archives, but since the most recent two did not contain databases, it seems likely CPanel just never saved them.

 

If anyone can recommend a fast .tar.gz viewer, I'd appreciate it. I'm using WinRAR to view them and it's really slow.

Share this comment


Link to comment

Josh have you tried 7zip ( http://www.7-zip.org/ ) it can open all archives very quickly, just remember to set the manager so it is associated with the archive.

 

If you need a hand going though the archive I could always help if possible :)

Share this comment


Link to comment

ah that would explain why they are all slow thats one heck of a archive :) I dont think there is anything that will open then quick as the index is going to be huge.

Share this comment


Link to comment

I am copying files off a USB hard drive right now. It will take hours, and it is 1 AM in California, so I am going to sleep. If I am able to locate any copy of the database it will be good, but it seems very likely to me the CPanel full site backup has been failing to save databases this whole time.

Share this comment


Link to comment

Opening a tar.gz of 12 GB shouldn't be any problem with WinRAR or 7Zip. The initial loading/opening might take 30 minutes (if not more). But afterwards you'll browse through the files quite easy.

Share this comment


Link to comment

It’s odd I use the fullbackup option when transferring new clients sites to my server’s, and restore them through my WHM panel, you can’t do it through cpanel, but never a file as large as yours. I have seen cpanel backup dump database tables into individual binary files but has always restored correctly through phpMyAdmin, I have my fingers crossed for you.

Share this comment


Link to comment

All UI packers are slow, but the command line is fast. So if you just search for some certain file pattern, like *.gmf it will be very fast to list all gmfs in a .tar.gz or any other archive format.

Share this comment


Link to comment

Tell your host to restore a previous backup. Most hosts do daily, weekly and monthly backups of the whole server. Sometimes they'll charge a small fee for the restoration.

 

If they don't do that, drop your cheap host, and get a real one.

Share this comment


Link to comment

I wouldnt say its all your fault josh - as a user we need to expect that the software is configured properly. In case of a backup i too would expect the default setting would be "all-and-everything".

Got me tempted though and i might have to put a bbord up myself, just to work with it.

I hope its not too bad for lw beginners ... at least there is the wiki, the set of tutorials, the user-manual and the C api manual from sep2010.

Share this comment


Link to comment

Josh, I think that you should first of all stop thinking about the past and turn toward the future. It seems to me that it is not worth wasting your valuable time to searching the archives. Probably the biggest pain is the loss of documentation for LE2. The rest is not critical. Asset store files, screens, blogs, posts, etc., many of these things are made by individuals from community, and perhaps some may be (in some cases) restored by their authors (if this will be really needed).

 

I hope, that this minor setback with Werkspace, don't affect your great work with development of Leadwerks Engine (I believe that the engine sources are backed in a better way). Anyway, chill out and don't worry too much. :)

Share this comment


Link to comment

Josh, I think that you should first of all stop thinking about the past and turn toward the future. It seems to me that it is not worth wasting your valuable time to searching the archives. Probably the biggest pain is the loss of documentation for LE2. The rest is not critical. Asset store files, screens, blogs, posts, etc., many of these things are made by individuals from community, and perhaps some may be (in some cases) restored by their authors (if this will be really needed).

 

 

 

Actually that's quite insulting to the community I think, which is made up of many individuals who have spent many hours of their valuable time contributing to it, with "Asset store files, screens, blogs, posts, etc." and much more, but as none of that was critical I am sure they can now take it easy, stop uploading files, writing blogs, posting, helping with technical support, writing tools, headers and tutorials. :)

Share this comment


Link to comment
Actually that's quite insulting to the community I think, which is made up of many individuals who have spent many hours of their valuable time contributing to it (...)

I had no intention to offend anybody - just my poor English is not able to express themselves more subtly. Josh seems to be very worry about this accident. This is obviously a sad thing and I just wanted to give him some encouragement.

Share this comment


Link to comment

Yep that's quite disturbing but "s..t happens". Like any good community we have to help a bit to solve problems together as much as possible, to upload again what we can upload :)

 

Especially it is disturbing for you guys who have contributed a lot, I am grateful to you. I am relatively new member, but I'll try to contribute more in the near future.

 

Something Positive :

Deja vu :) Looks like Diving in the past :) Jane Croft site + release date :)

 

I'm young again :)

Share this comment


Link to comment

A few questions (and sorry if these were covered in some form):

 

What do we do about previously uploaded assets? It sounds to me like you're going through them and will see how reasonable it is to reupload them. Is that right? I ask because I have a saved copy of a file I uploaded (Leadwerks Joystick/Gamepad Example). I assume I should hold off on uploading to not mix things up.

 

Why is the documentation lost? Was there only an online copy? I would have imagined that Jorn would have worked on it offline and then uploaded it when he was ready to, so he would still have a copy offline which he could reupload. But it sounds like this isn't how it worked. Just curious. You made it sound like it needs to be recreated pretty much from scratch (for entries created after November).

 

Thanks and best of luck with painlessly restoring as much as possible!

Share this comment


Link to comment

In my case all is OK Josh, so this happennning at each of us i thinking, and many times.

Share this comment


Link to comment
Why is the documentation lost? Was there only an online copy?

It was created and edited online, but I've been making lots of backups. The problem is the backups don't do what they say they do. There are no options in CPanel to include only parts of a full backup. It's explicit about what it includes, and tech support assured me on two separate occasions the databases would be saved. So I felt pretty good about using it. Even if one archive somehow got corrupted, I had full backups going back for months.

 

The user data lost is very important. This isn't just a forum. People rely on it to host a lot of images, files, and their writing. I hope the protocols I enact will convince them to trust the site again with their data. I'll talk to my uncle about what steps should be taken.

 

My hosting company is actually very highly rated and not cheap, and they've done everything else very well. I won't mention their name for now, but they seemed remarkably unconcerned about this. Their reaction was "Hmmmm, that's interesting, I don't know why the databases aren't being saved."

Share this comment


Link to comment

Well, sometimes this sort of thing just happens. Data is lost and that's it. Hopefully this serves as a reminder to rest of us to backup our stuff.

 

*cough* stuff... I really must set up my own backup system.

 

Hmm, what is a good way to backup mysql stuff?

Share this comment


Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Blog Entries

    • By Josh in Josh's Dev Blog 4
      Here are some screenshots showing more complex interface items scaled at different resolutions. First, here is the interface at 100% scaling:

      And here is the same interface at the same screen resolution, with the DPI scaling turned up to 150%:

      The code to control this is sort of complex, and I don't care. GUI resolution independence is a complicated thing, so the goal should be to create a system that does what it is supposed to do reliably, not to make complicated things simpler at the expense of functionality.
      function widget:Draw(x,y,width,height) local scale = self.gui:GetScale() self.primitives[1].size = iVec2(self.size.x, self.size.y - self.tabsize.y * scale) self.primitives[2].size = iVec2(self.size.x, self.size.y - self.tabsize.y * scale) --Tabs local n local tabpos = 0 for n = 1, #self.items do local tw = self:TabWidth(n) * scale if n * 3 > #self.primitives - 2 then self:AddRect(iVec2(tabpos,0), iVec2(tw, self.tabsize.y * scale), self.bordercolor, false, self.itemcornerradius * scale) self:AddRect(iVec2(tabpos+1,1), iVec2(tw, self.tabsize.y * scale) - iVec2(2 * scale,-1 * scale), self.backgroundcolor, false, self.itemcornerradius * scale) self:AddTextRect(self.items[n].text, iVec2(tabpos,0), iVec2(tw, self.tabsize.y*scale), self.textcolor, TEXT_CENTER + TEXT_MIDDLE) end if self:SelectedItem() == n then self.primitives[2 + (n - 1) * 3 + 1].position = iVec2(tabpos, 0) self.primitives[2 + (n - 1) * 3 + 1].size = iVec2(tw, self.tabsize.y * scale) + iVec2(0,2) self.primitives[2 + (n - 1) * 3 + 2].position = iVec2(tabpos + 1, 1) self.primitives[2 + (n - 1) * 3 + 2].color = self.selectedtabcolor self.primitives[2 + (n - 1) * 3 + 2].size = iVec2(tw, self.tabsize.y * scale) - iVec2(2,-1) self.primitives[2 + (n - 1) * 3 + 3].color = self.hoveredtextcolor self.primitives[2 + (n - 1) * 3 + 1].position = iVec2(tabpos,0) self.primitives[2 + (n - 1) * 3 + 2].position = iVec2(tabpos + 1, 1) self.primitives[2 + (n - 1) * 3 + 3].position = iVec2(tabpos,0) else self.primitives[2 + (n - 1) * 3 + 1].size = iVec2(tw, self.tabsize.y * scale) self.primitives[2 + (n - 1) * 3 + 2].color = self.tabcolor self.primitives[2 + (n - 1) * 3 + 2].size = iVec2(tw, self.tabsize.y * scale) - iVec2(2,2) if n == self.hovereditem then self.primitives[2 + (n - 1) * 3 + 3].color = self.hoveredtextcolor else self.primitives[2 + (n - 1) * 3 + 3].color = self.textcolor end self.primitives[2 + (n - 1) * 3 + 1].position = iVec2(tabpos,2) self.primitives[2 + (n - 1) * 3 + 2].position = iVec2(tabpos + 1, 3) self.primitives[2 + (n - 1) * 3 + 3].position = iVec2(tabpos,2) end self.primitives[2 + (n - 1) * 3 + 3].text = self.items[n].text tabpos = tabpos + tw - 2 end end  
    • By 💎Yue💎 in Dev Log 5
      The prototype of a four-wheeled vehicle is completed, where the third person player can get on and off the vehicle by pressing the E key.  To move the vehicle either forward or backward, is done with the keys W, and the key S, to brake with the space key.  And the principle is the same as when driving the character, a third person camera goes behind the car orbiting 360 degrees.

      I don't think the vehicle is that bad, but I'm absolutely sure it can be improved.  The idea is that this explorer works with batteries, which eventually run out during the night when there is no sunlight.
      Translated with www.DeepL.com/Translator
       
      Mechanics of the game.
      I'm going to focus on the mechanics of the game, establish starting point (Landing area), after the orbiter accident on Mars where all your companions died, now, to survive, you will have to repair your suit, oxygen runs out, good luck.  This involves replacing the oxygen condenser that is failing and the suit is stuck.

      On the ground and performance.
      The rocks, the terrain and the vehicle kill the SPF, but there is a solution, and everything is related to the chassis of the vehicle. That is to say that if I put a simple collision bucket for the vehicle, the yield recovers, something that does not happen if I put a collider of precise calculation for the car. This has the advantage of better performance but is not very accurate, especially when the car crashes with an object in front, because the horn of the car has no collision. And the solution to this, is to put a sliding joint, as was done with the area in which the player climbs the car and descends from it.


       
      On the rocks, I am trying to make them with the slightest polygons and the most distant from each other. 
      Obviously on Mars I can not create canyons, high mountains, is because the terrain does not produce shadows on itself, that's why the terrain tries to be as flat as possible, simulating a desert with dunes. 

      That's all for now.
       
    • By 💎Yue💎 in Dev Log 9
      The prototype is finished, and the mechanics of the game can be given way.  It has established a desert terrain in the form of dunes, this implies that there are no cannons or anything similar, because Leadwerks does not allow a terrain to cast shadows on that same terrain and this looks visually rare.
      So the terrain is like low-slope dunes. On the other hand, I think the texture of the terrain is already the definitive one, with the possibility of changes and suggestions on the part of those involved in this project.
      On the other hand we have taken the model of a habitat of the nasa, which certainly looks very nice. 
      The next steps, are to establish the starting point of the player, this must start near the capsule return to Mars somewhere on the map of 2024 x 2.
      And think about the first thing you should do, repair your suit? Seek a shelter? things like that.  


×
×
  • Create New...