There was a FIRE at the EEVblog WebNX/Gorillaservers data center in Ogden Utah and the EEVblog website and forum has been down for days now. This is an update.
https://www.theregister.com/2021/04/06/webnx_data_fire/
#WebNX #DataCenter #Fire
Subscribe on Odysee: https://odysee.com/ @eevblog:7
EEVblog Web Site: http://www.eevblog.com
The 2nd EEVblog Channel: http://www.youtube.com/EEVblog2
EEVdiscover: https://www.youtube.com/eevdiscover
Support the EEVblog through Patreon! http://www.patreon.com/eevblog
AliExpress Affiliate: http://s.click.aliexpress.com/e/c2LRpe8g
Buy anything through that link and Dave gets a commission at no cost to you.
Donate With Bitcoin & Other Crypto Currencies!
https://www.eevblog.com/crypto-currency/
T-Shirts: http://teespring.com/stores/eevblog

Hi, just a quick, uh, update a video because you might have noticed that the Eevblog website is down. That means the center of the Universe is down. Which is the Eev blog forum. Um, everyone's getting withdrawal symptoms.

They're having to check themselves into clinics and all sorts of things because, um, yeah, they can't get their fix of, uh, the Eev blog forum. unfortunately. And the website and my email has also, uh, been down since as of approximately uh, the fourth of the fourth at uh, 2100 hours? Uh, it's been down for quite a few days now. What is it? Currently it's the 6th of April.

Um, almost midnight on the, uh, yeah, so almost the 7th of April now. So it's been down for like three days or something. Um, yeah, and it's kind of a big deal, So I just wanted to, uh, update you on what's happening. Not that I know much.

Uh, we're waiting for it to come back because, uh, what happened is there was a fire basically at the Uh Data Center in Ogden, Utah at Gorilla Servers, which is the company I use to host the dedicated servers. Uh, for the Evblog website, the Ev Blog forum, Eevblog.org is also down where I do the podcast. uh, host the podcast videos like the streaming 720p podcast version, and stuff like that. It's all down their entire, even their own website does not work.

I'll refresh that now. Nope, Gorillaservers.com is down. So they do actually have an update, uh, here on webinx. I guess it's one of their other companies that's still up because it's ironically hosted somewhere else.

So let's have a quick reader list. Now that we have a better understanding of what happened, we would like to give everyone an update. One of our old generators that have has worked for years and was recently load tested had a mechanical failure and caught fire, resulting in power being cut to our core routers and fire suppression system controlling the fire. So I guess the automatic sprinklers and everything else.

So a certain fire suppression. They probably might not use water they might use, you know, some gas or some data centers you know, fill up with inert gas and things like that. which, uh, extinguish the fires. Not sure which.

um, uh, unfortunately. uh, the fire department opted to cut power to the rest of the building as a precaution even though the power systems were independent. So it's even though this backup generator fire failed, the fire was controlled. Uh, this data center could have still operated including uh, presumably my Uh servers.

So yeah, they, um, but the fire department just cut power to the whole complex. It's a massive complex in the middle of the desert somewhere. Um, yeah. We are currently waiting for an emergency inspector to arrive.

give the all clear so and bring back most of the servers in Ogden back online. Some servers will have an extended outage as those may require rebuilds due to some water damage. Okay, it looks like they do have sprinkler system or that might have been the fire department. Who knows, Um whose builds have high probability that data is intact.
We'll like to thank you for your patience. blah blah blah blah blah. So uh yeah. their uh, latest update on Twitter? Uh, their current uh, Gorilla Servers Inc.

Um, Host Vision Follow Host Vision on Twitter. That's gniff on the Eev blog forum. He's the one who maintains the Eev blog Uh Servers and Gorilla Servers current updates project that close to 90 to 95 of all Guerrilla Servers hardware. Well make it bigger.

Experience: Zero damage. What's the bet? Even money that the Eev blog servers are the ones impacted. Anyway, electricians are working and now to restore power to the Ogden Utah Data Center and an estimated powershell return at approximately 3pm Mdmdt Either I think that's mountain time or something tomorrow. So that time has passed.

but since then, uh, last night, my emails started to come back. So my email server I've got a Cpanel email server which handles email and a bunch of other uh stuff and that's all working again because if you don't know, we have uh, three servers for the that handle all the Eev blog infrastructure. One is, as I said, the email server and the Uh which handles some other stuff as well. and also uh.

two redundant dedicated servers. These are all dedicated boxes. They are shared, so I've got actually three dedicated boxes. There's Xeon something or other.

Anyway, two redundant Xeon database servers which handles all of the forum and the uh, Wordpress website and all my online store and stuff like that. Yes, my online stores down. I might still be able to ship some uh stuff if you've got an order in, because then I use a third-party shipping system so the orders might have been automatic. I haven't checked today.

we're going to ship some today, but they should automatically have been imported in, so hopefully uh, those will still go out even though this is all down. So yeah, so I've got so they're a redundant Uh server. so if one fails, it takes like the database is mirrored and shared and it's all complex. Uh, Genif has set all this up.

It's all you know. Advanced Penguin, uh, stuff. So I don't know too much about it. But yeah, so that's all safe.

And by the way, all the data's safe. All the backups if for some reason, uh, the server, the data center has actually lost the data on the drives in the server machines. We do have full daily backups in two separate Uh locations, professional backup services. and at most we might and uh, genius said that.

um, it's he did check and it was during the backup time is when the whole thing went down. So at worst we're going to lose like a day's worth of posts on the forum or something like that. if my uh server is one of the ones affected. So there you go.

Which Murphy, that's it. Like yeah, it's going to be back. It's interesting that my Cpanel email server is back, but the others aren't but they are on different subnets. They could be fine.
It's good chance it's fine, but they're within different subnets in the data center. So yeah, so they bought one subnet back up and the other ones? Yeah, they're just separate. So we we just don't know. We just got to sit back and wait.

Um, they have said that they can't uh uh, like give dedicated. You know, say oh yeah, this um, clients servers will be back up and running or whatever time. But anyway, they are past the time when it's supposed to come back up and the website hasn't come back up yet so we're just going to have to wait. I don't know how long it's going to be.

So yeah, that raises uh, some interesting uh questions. and we just put up a dedicated uh gene if just got this dedicated page hosted somewhere else and redirected the Dnses. So at least it shows you something and this you'll get the updates. There's no more update.

I don't think he's updating anything in the background. No, no, we're just going to have to wait. He's monitoring it and it'll eventually come back online. But that raises some interesting questions.

I know there's going to be one million responses down below of how we're doing it all wrong. and we should be using a cloud infrastructure. We should be using a more reliable, uh, data center. and uh, come on right.

And a fire in a data set? A fire in a generator. This could have happened to any data center, really. And I guess you can make an argument that it should be on, you know, Amazon cloud or whatever. But hasn't that gone down in the past? I mean, it's just like Murphy's gonna get you right.

We've had no issues with, uh, this current host for I don't know. I mean, five plus years at least. I think. uh, we've been on these dedicated uh boxes at Gorilla Servers and we haven't had an issue apart from, uh, like technical stuff.

you know, like it's yeah, it's not like they were unreliable, it's just other things. Uh, sort of have taken the website down here or there for like, you know, hours or a day or something like that. Um, but yeah, that wasn't really. Um, the server's their actual fault.

So anyway, um, yeah, I don't know. Flame away down below. Yeah, got caught with pants down. Data's safe, but yeah, um, it's been out for three.

probably four. Could be longer days that everything's down. At least my email is back. And I guess that's a lesson that I need to set up some sort of redundant email exchange server thingamabob.

I do actually use Uh Gmail for all of my gmail is actually my interface for email. So if you send a email to Dave at Eevblog.com then it actually just gets forwarded to Uh Gmail. And it actually replies I've got. like the reply to section in there is Davidev Blog.

So you don't know that I'm actually reading all my email on Uh Gmail, so it's mostly there. But anyway, it is back up. But um, yeah, because a lot of stuff was tied to Dave at Eevblog.com Um yeah, I couldn't do a lot of stuff like I couldn't get like do email confirmations for doing, uh, various online activities and stuff like that. So yeah, um, it's a bit.
I hate. Gmail has gone down before, so you know, yeah, it's Murphy's going to get you at the worst possible time it will happen. Um, so I don't know. Yeah, everyone will flame away down below.

Hey dumbasses Dave for using that. You know, just hosting it on a couple of little dedicated boxes in the middle of Utah somewhere. But no, there's there's reasons why we do that. A lot of people ask why we don't host in Europe or something like that.

The majority of, uh, the viewers and the forum users and stuff like that are sort of like Us-based U.s is probably the best place to actually host it in terms of like latency and you know all that, uh, sort of stuff. So you know Yeah, it could be hosted here, in Australia, could be hosted in the Eu, could be hosted in, you know, some little, uh, I don't know, Tin Pot Island somewhere in the Pacific or something. But I yeah, Anyway, there you go. Just wanted to give you an update.

Uh, so I can't do the keysight, uh, drawing. by the way, because the forum's not available. I, you know, I don't have ability to. I run my Wobulator program and all that.

but I'll probably do the drawings for the other Uh ones probably today. So yeah, anyway, there you go. Just wanted to give you an update. It's down, could be down for a while.

I got no idea. um, if we do have to, if my servers were impacted by the, uh, water damage or whatever and the hard drives were lost. Yeah, we might lose like you know, half a day or up to a day at most worth of uh, forum posts or something like that. So that does happen.

Um, sorry. It's uh yeah. There's probably reasons I think we did look at doing like a live database backup somewhere. we already run a redundant system.

It's just that the whole data center has gone down. So it's like, you know, so we just do a daily uh backup. So yeah, we don't have a third off site redundant, live, redundant synced database server. It's all complicated when you try right to multiple databases multiple times we've actually had uh, technical issues regarding on on the forum regarding actual uh, the use of having the multiple databases because you've got the sync issues with it.

I don't know the exact you know technical stuff behind it, but you know it can get complicated when you try and share all of your databases. You know they're trying to all sync and be live and stuff like that. We have image problems on the forums for attachments they weren't linking correctly and stuff like that. So if all your databases aren't synced, I'm sure Gdif can inform people down below if you've got technical questions about the whole thing.
But there you go. Yep, still down. I don't know when it'll be back. Sorry, got no idea.

Um, we'll just have to wait like everyone else, but my email is back Davidevblog.com so that allows me to do some stuff. Yeah, Flame away. catch you next time.

Avatar photo

By YTB

23 thoughts on “Eevblog webnx data center server fire!”
  1. Avataaar/Circle Created with python_avatars Matthew Miller says:

    There is no cloud, its just someone else's computer…one time we had problems paying our mortgage because the cloud-provider the bank used went down for a day…

  2. Avataaar/Circle Created with python_avatars Dan Baker says:

    fire dept cuts the trikety all cool. hacker subverts the server 20 yrs jail 100s of million in damages kittens die puppy forget how to lick. data center life.

  3. Avataaar/Circle Created with python_avatars notadream1999 says:

    Went down right when I had a question for the forum…. just my luck 🙃🤦 I'll just have to play around and figure it out.

  4. Avataaar/Circle Created with python_avatars John Athanasiou says:

    I've tried to help customers who had whole website data completely lost from local hosting providers due to hardware crashes at hosting sites & their local techs not properly securing nor verifying their hosted web servers backup integrity…I won't mention names!.
    I've also dealt with service providers whose previous employees held domain keys of entire domains & whose techs were trying to financially exploit their customers to the tune of thousands of $$$$ to get the key back.
    Their excuse was they were fired but oh how they shot themselves in the foot as a result of their foolishness

  5. Avataaar/Circle Created with python_avatars John Athanasiou says:

    Dave, you should look at Datto backup solutions also.
    Cloud servers going down is a fact of modern technological life & it's basically like Russian Roulette in that eventually it'll hit you too some day.
    All those other suggestions are great, but, no matter what you implement be it a RaspberryPi, Windows or Linux server it can also experience severe hardware issues like backplane faults & what most never think about is protecting these backup devices with (to borrow from Sorin!) a "properly calibrated firewall" ie FortiGate etc. to help protect for more prominent security vector targets ie famous people like Dave, Celebs, Prominent companies against black hats trying to exploit a vulnerability because they had some sort of "issue" with what you said, or produced.
    Wishing your CSP a speedy recovery & working in the IT & electronics space I really feel for you Dave as IT people can only be sympathetic to your cause.
    This is a lesson for all of us, not just the cloud service providers that the 5 nines promised by sales teams doesn't always live up to its claim

  6. Avataaar/Circle Created with python_avatars Sevil Natas says:

    So much for their five 9s up time. Hope you are getting a credit. I am not sure their is a "should" in this situation. You provide valuable information to the public for zero cost to them. The only "should" is that you SHOULD do it the way you want and bugger the nay sayers.

  7. Avataaar/Circle Created with python_avatars Nathan L. says:

    I HAVE BEEN TRYING TO REGISTER TO EEVBLOG FORUM FOR MONTHS NOW AND NEVER GET THE REGISTRATION EMAILLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL

  8. Avataaar/Circle Created with python_avatars eidodk says:

    It's a website.. You could host it on the international space station and the latency would be irrelevant. Unless you use a critical SQL database that is remote from the server, latency is a completely and utterly useless measure. It's not online gaming, it's a wordpress website.

  9. Avataaar/Circle Created with python_avatars Gadget Review Videos says:

    Redundancy is a beautiful thing. It's even better when the servers are located in different data centers to prevent outages from natural disasters and other reasons like generator failures.

  10. Avataaar/Circle Created with python_avatars Chasing Dragons says:

    I'd ask why their servers got wet… actually I would also ask where their generators were located and if it was an electical side fire or an engine side fire. Different maintenance implications. Sounds to me like they took shortcuts that bit them in the ass.

    This is as bad as that AT&T Christmas bombing causing outages across multiple states because some idiot decided to stick all the generators in the basement… below several floors of water based fire suppression systems that flooded the basement (aka the generator room).

    This was the moment AT&T services went down, NOT went the literal bomb went off beside the building.

    Actually a LOT of things were wrong with that data center from both a security AND BCP/Disaster Recovery perspective but that decision was possibly the worst of them.

  11. Avataaar/Circle Created with python_avatars SurvivalSquirrel says:

    Corona, Ever green, burning Datacenters… Some would say, these arent coincidences happen at the same time

  12. Avataaar/Circle Created with python_avatars isettech says:

    This is why many corporate servers are Geo-Redundant Colocation stored. One entire data center can go offline to a disaster and the alternate location takes the load. EV Blog is not at the level to spend the money for Geo-Redundant servers. Most fortune 500 corporations, the stock market, and major banks all use redundant servers for this reason.

  13. Avataaar/Circle Created with python_avatars xDR1TeK says:

    It's so surreal how we live synthetic and artificial lives. Imagine when drives go bust, how we feel the loss?

  14. Avataaar/Circle Created with python_avatars Ivan Stamenkovic says:

    I am doing programming professionally, trust me you are better of with a dedicated box rather than using the cloud for this. And it's cheaper. What you could do because you have the redundant databases, is to have redundant wp and forum software and load balance (or some other strategy like in case of offline…) between those.

  15. Avataaar/Circle Created with python_avatars Travis Parker says:

    My job had grid power fail, UPS banks kicked in, generators started… well, two out of three generators started. A few minutes later the UPS providing ride-through dies and we lose one feed. Racks stayed up but cooling did not! We also lost environmental monitoring (genius idea to put the monitoring on the same feed as the aircon!) so nobody realized until I thought to check the datacenter after we noticed we lost power to scattered lights/cubicles/networking gear… opened the door and got hit by the heat wave. Or course, as soon as we get everything shut down, sweating buckets the entire time… grid comes back.

  16. Avataaar/Circle Created with python_avatars kd1s says:

    Well Dave – I found out when I worked in a place that relied heavily on Google services that even they go down sometimes.

  17. Avataaar/Circle Created with python_avatars Railgap Esoterica says:

    This is why you use data centers with dual utility feeds AND generators. There's at least one in every major city and they cost more. Clearly: you get what you pay for. Looks like WebNX dropped some 9s. 😉

  18. Avataaar/Circle Created with python_avatars Railgap Esoterica says:

    re; the key frame… I worked for a "big data" AI research outfit briefly, and they sent me a very serious server box for the time period, loads of cores and RAM all in a 1U, took two circuits to run it, and of course its hostname was WOPR. It never caught fire, but my house wiring should have. But didn't. ^_^

  19. Avataaar/Circle Created with python_avatars Code Blue says:

    you would be suprised but most DC's ive been in just use water, at least for the switchgear and generation side. of course if it activates there is alot of gear to replace.

  20. Avataaar/Circle Created with python_avatars RN1441 says:

    OMG didn't Chernobyl teach us that safety tests are the biggest risk to high value facilities!? (/sarc)

  21. Avataaar/Circle Created with python_avatars Siana Gearz says:

    OVH also seemed pretty good until it caught fire just a couple weeks back. And then it turned out they're made out of wood and don't even have sprinklers. You'd think one of the world's largest serverfarm companies should be pretty serious about their business? Yours is better equipped for sure.

    I'd say you've had a reasonable enough service track record, you have a good redundancy and backup regimen, and congrats on getting mostly back up.

  22. Avataaar/Circle Created with python_avatars Jan-Jonas Sämann says:

    Interesting. Even if You had hosted one box in france at OVH for backup (wich also burned down recently) also could have lost everything. Maybe OVH inspired fireman to do it the hard way in the first place. OVH lost nearly the entire datacenter worth of data in that accident.

  23. Avataaar/Circle Created with python_avatars Cubeist Games says:

    Well, if it was worth it you could mirror your site using a DIFFERENT provider, with either a load balancer on the front or just cut over using DNS. It’s a cost/benefit business decision.

Leave a Reply to Chasing Dragons Cancel reply

Your email address will not be published. Required fields are marked *