Community: Serverausfälle bei mehreren Firmen
Mehrere britische Tabletopfirmen melden schwere, eventuell irreparable Schäden an ihren Webservern.
Hervorhebungen durch Brückenkopf Online
Den Anfang machen Hasslefree:
Hi Everyone
So the short version, as always, is up here, then I will go into detail below for those who are interested in the why’s and wherefore’s of such things.
We are currently closed. We’re not 100% certain when we will re-open but it will be at least 1 week and likely more.
Just to be absolutely clear, even though some people will ignore this part, we are NOT closed down 🙂
Something behind the scenes has happened with the server and it is some level of catastrophic, we are working on finding out ‚how‘ much we need to fix and while we do so we are closed for new orders.
Existing orders will be fulfilled. However you may be contacted by us to double check shipping addresses etc. as while the server is melted we currently have no way of checking if you wanted a different shipping address to your billing address.
This does not affect the Kickstarter, it’s only related to our webstore.
So for right now, if you have ordered from us and not received a despatch notification, please keep an eye out for an email in case we need to ask you something.
If you need to contact us, obviously our domain name is down so please use BWARTEMIS at GMAIL dot COM.
Also please make sure you are following our FB Page as I cannot send out newsletters every day 🙂
The Long and Scary Explanation
Ok folks, strap in, because I’m typing this and even I don’t believe it.
(If anyone out there buys into the whole ‚we are living in a simulation‘ stuff, this will not help disprove it).Yesterday our website went down. It’s happened before, it’s not usually a big deal. This one is.
I got an email from our web guy that included the word ‚catastrophic‘, not normally a good sign. Our server is toast, and backups or recovery are currently in ’status unknown‘ territory.
A mysql update broke, let to a number of steps (a rollback, a reboot, a corrupted file system etc.) that got worse and the reboot corrupted the whole thing. (Just for clarity purpose, nobody’s data is at risk or anything, this is more akin to a hard drive failure)
This is beyond terrible timing obviously, I am just back from illness and have been working on a number of things to get the business back up and running. This is like a kick to the genitals as far as that is concerned.
However, it gets worse! This happened the literal day that our web guy is leaving the country for a ‚cruise‘, not even a normal holiday, a ‚boat in the middle of the ocean, try working online from there‘ cruise!
So we are currently sans website and more importantly, sans database.
Obviously we have emails for all payments and orders etc. But that means over the next few days I have to manually go through all order emails, make a spreadsheet, mark off everyone’s order that has been shipped using the physical sheets of paper we use for that and then contact people from their payment details to ship outstanding orders.
Can we do that? Yes.
Is it a metric f&*%ton of work that takes time away from the plans to release new product and get the KS out the door etc. that I have been working hard on these past weeks since returning from illness? Also bloody yes.I’ll get a holding page up over the next day or two and link it to this newsletter on our FB page. I can’t even do that easily as …no server.
The bad luck of it both happening, and the timing of the things around it is ludicrous. I stared at the email for a little bit wondering if I was still asleep to be honest.
But, all we can do is soldier on. No point in panicking or freaking out. We’ll get the orders out, do a full manual inventory and then depending on how long it takes us to get the full site back up we will likely start up our backburnered Ebay store. We will keep you all informed about each step as we go but priority number one is making sure we get outstanding orders out the door so people don’t start freaking out when they can’t check on the website.
Arty.
Auch Vanguard Miniatures sind betroffen:
ANNOUNCEMENT
SERVER DEADExtremely bad news from my website host, the hosting server is dead and it is quite possible that my website is going to be unrecoverable, both business email accounts could also be lost. Although an attempt to recover the site will be happening it will not be able to be carried out for at least another 12 days or so.If recovery is impossible then that means a complete rebuild will be required, the site could be out of action for several weeks or more its hard to say at this time. Needless to say this is a disaster of the highest magnitude for me and my business, I’m now looking at possibly spending days and days rebuilding ever product entry which is over 600 plus items of stock. Not to mention the best part of a month of lost sales which I can ill afford.Anyway that’s the situation folks Vanguard Miniatures is closed until the website is either recovered (not sounding likely) or rebuilt.John
Genau wie Too Fat Lardies:
Righty ho. Well, I am not really sure where to begin, but let’s give it a go.
The TooFatLardies website is likely to be down for two weeks. And when it reappears, I’ll be in the US for Historicon. Now, there’s a double whammy.To provide a bit of context, we, along with several other wargaming companies, use the services of the man I always refer to as the web elf. It’s great to deal with him and he’s a wargamer and consequently it makes life really easy when we release products and he understands exactly what we are doing. He’s also a bloke who goes the extra mile and often works round the clock to make sure things are running as well as they can.HOWEVER, he’s just had a catastrophic failure of the server he keeps everything on and all the web sites he manages are kaputt, for now. That probably isn’t what has actually happened, as there is all sorts of technical reasons this stuff kicks off, and it’s all beyond me.HOWEVER (yes, another HOWEVER!) its his 60th birthday tomorrow and his wife has booked a cruise. They leave in the morning. So, that means that until, he returns, nobody can access our web site and, as it is likely it will have to be rebuilt from scratch, which rather compounds the issue.Now, there are moments in life which test one’s patience and, frankly, this is one. But the reality is that I could jump up and down and scream and shout all I want, but it won’t change a thing. What’s more, the poor bloke has got to go on holiday and look like he’s enjoying it. So, with that in mind, I have told him to have a great holiday and not to worry about it until he returns.It’s obviously a huge blow for us as our cash flow is now completely shut off. It’s equally frustrating that the business email doesn’t work and if you ordered in the last 36 hours we can’t process that. Sadly, I am not here providing any solutions, just informing you of the issue we currently have. I do hope that you can all bear with us until we can get this resolved.
Quelle: Hasslefree Newsletter
Quelle: Vanguard Miniatures bei Facebook
Quelle: Too Fat Lardies bei Facebook
TLDR
Server Kaputt
Backups gehen nicht?
Weil warum?
Ist so eine Angewohnheit von Computerproblemen, dass man länger braucht, um zu herauszufinden, was da passiert ist.
Bei der Anhäufung innerhalb kurzer Zeit frage ich mich allerdings ob das ein Angriff sein könnte.
TFL sagen doch:
Das ist EIN Typ der für mehrere Firmen arbeitet und alles auf EINEM Server hat, und dieser ist abgeraucht…
Klingt nicht nach Angriff, eher nach shit happens der worst case Sorte.
Liest sich so als wäre die Häufung einfach weil die verschiedenen Hersteller auf der gleichen Kiste laufen, weil eben die Welt klein ist.
Wenn die Kiste dann ausfällt sind halt alle kaputt.
ich drücke doll die Daumen dass irgendwie noch eine Datenrettung gelingt.
Danke. Habe ich überlesen. Dann wünsche ich Glück.
Viel Glück. Ich hoffe das regelmäßige Backups erstellt wurden.
Wenn der Admin was taugt war er faul.
Faule Admins automatisieren das und dann ja
DIE FRAGE ist aber wo wurden die Backups gespeichert?
Auf der gleichen Maschine vielleicht noch im gleichen Raidcluster?
Auf einer Tape Libary an einem anderen Standort?
Und wir alt sind die, also wie regelmäßig wurden full und incremental Backups erstellt.
Gab es ein Full vor dem DB Update da alles zerfetzt hat?
Wir wissen es nicht …
Backups auf derselben Maschine wäre ja maximal doof.
Eigentlich gehört sowieso auf ein eigenes System, idealerweise räumlich komplett getrennt mit eigenem Brandabschnitt usw
Schon klar
meine Backup Server stehen auch in verschiedenen Rechenzentren.
Das war zwar damals teuer in der Einrichtung aber ich muss nach drei ransomware angriffen, einer toten maus im server und einem feuer das die bauarbeiten im keller neben an ausgelöst haben, bin ich sehr froh das mein Admin da damals sich mit mir bis an den Rand der Kündigung gestritten hat bis er bekommen hat was er wollte. Und ja sein Jahres Bonus war angemessen Motorisiert.
Bei dem da oben klingt mehr nach nebenbei mal eben dafür günstig für kleine buden gemacht.
Aber da räume ich durchaus ein, dass das durch die darstellung in der Meldung verzerrt sein kann.
Also will ich nix unterstellen und hoffe mal auf das beste
Okay jetzt hatte ich zeit das zu lesen
Und mein Admin sitzt hier und schüttelt den Kopf (nein er lacht nicht).
Wenn das passiert ist was er glaubt dann wurden die Backups nie getestet bevor ein Major Update für was auch immer eingespielt wurde. (Hier MySQL). Und es wurde ein Roll out gemacht bevor jemand sich länger nicht um die Server kümmern kann. Das macht man halt nicht. Keine Rollouts vor Wochenenden, Feiertagen oder bevor der Verantwortliche in den Urlaub geht.
Das geht immer schief (irgendwann)
Drücken wir mal die Daumen das da nicht zu viel zerstört wurde.
Da drücke ich fest mit! „Computer“, und „Internetz“, sind halt Neuland. Da muß es sich halt noch einruckeln. Wenn sich das überhaupt durchsetzt.
https://www.karwansaraypublishers.com/blogs/wss-blog/server-down-websites-out
Falls es ok ist das hier zu verlinken. Eine Liste aller betroffenen Firmen mit Kontaktmöglichkeiten. Auch wenn das meiste als kleine UK-Klitsche für uns hier weniger relevant sein dürfte.