Blog Disaster: Choopa-Vultr (Network Panda) Server Outage

I decided to turn off the WiFi router and stay offline for almost the whole week. With that move, I am able to read and finish lots of chapters of my review books. I feel like the week is so good to me that I am in pace and understand a lot compared to the previous days. Since I am absent in social media and as  well as here in my blog, I am not able to monitor things 24/7.

Update on this article:

Oct 2 This blog is no longer hosted by Network Panda due to numerous problems that arise a month after the transfer. More details about it here: Network Panda Web Hosting Review.

Network Panda Server Outage

The unfortunate discovery happened last Friday. I found out that my blog was down for more than 34 hours already! WTF. Is this for real? Three different uptime monitoring service reported that my site is down. Cloudflare returns an Error 521 (Web server is down).  I immediately login to my hosting provider’s member panel to find out what’s wrong. Right after logging in, an announcement in red font color welcomed me in cold. Numerous servers were taken down by one of the upstream provider of my host due to violation of terms of service. This act is due only to one single website that uploaded an illegal content. Network Panda (my current host) is renting servers from Choopa/Vultr to provide hosting services in some locations. Sadly, the location where my blog resides is included on the suspended servers. Netherlands (my blog’s location) together with NewYork 3, United Kingdom and Germany servers are the ones affected. This is really a huge disaster after knowing that the backup resides on Choopa’s network too. I’ve been with Network Panda for almost 3 months now and this is the only major downtime ever since.

Network Panda is losing hope on reactivating the said servers or at least recovering the backup of thousands of sites they serve. Choopa does not respond on messages sent to them. With that, Network Panda is encouraging their clients to provide a copy of backup that they have for their websites to get restored. Sadly, I do not have a latest full backup of my site. The only copy I have is the one generated last June. And it is corrupted. Great. But still, I am lucky that I downloaded a .xml file of my blog before being offline. Yes! At least I have this copy of manuscript of my articles.

Restoration

After sending a ticket, Network Panda immediately transferred my account to a new server in France. They also added a 1 year free extension of hosting service in my account as a compensation for the disaster that just happened (This compensation looks good but i want my site better.) Since I do not have a full backup of my site, I installed a fresh WordPress copy and start from scratch. I manually uploaded everything and updated all the articles. This is so tiring and time-consuming. A price of not having a backup.

Today is the third day of outage but Choopa still refuses to provide the backups of network panda’s servers. Network Panda had this feeling that the servers were already formatted and recycled. I think this is not a good way how Choopa is treating their clients, whatever the case is. With the fact that right after discovering an illegal (gambling related) content in one of the servers, they immediately taken down all the servers without giving Network Panda the time to fix or delete files on the affected server. Network Panda said that the service was shut down only after two minutes of Choopa’s notification. My host did not mention any history of notices or anything dealing with Choopa/Vultr that maybe the reason that led to the termination. After all, I still believe that a customer has the right to secure files before the closure of the account.

What I Learned

I guess I learned the hard way. It is still best to secure a backup locally. And for Network Panda, they should also learn from this big disaster – it is to place the backups on a provider different from the main network and regularly check client’s accounts for content/s that may violate terms of service.  Right now, the server I am with is provided by OVH and it seems like it is as fast as Choopa/Vultr. I am more positive that the incident won’t happen again since OVH is a well-known and praised hosting provider. Network Panda also promised to improve their backup/restoration plan.

Info: Choopa/Vultr is a service offered under Choopa LLC.

Oct 2 The chance I had given to Network Panda ended today and I finally decided to leave the service.

About the Featured Image

Since I consider this unwanted event as a big disaster, I remembered the disaster happened to the airship Hindenburg in the 1930’s. Hindenburg is full of Hydrogen gas which provides lift for the aircraft. It is a way of transportation before but Hydrogen was never used again for this purpose after the disaster. Hydrogen is a flammable gas and that makes Hindenburg to catch fire with passengers and crews still on board. Hindenburg crashes  and burst into fire quickly as Hydrogen is burned out. Helium, a non-flammable gas replaced the use of Hydrogen when something is needed to be “lift in air or float” just like those balloons in a children’s party.


— RAZ RECOMMENDS —
zalora-bap-15-offBrand Ambassador Promotion: Discount code is for new customers only.


  • Hands

    Network Panda can’t blame Choopa/Vultr for this problem. Having backups on the same network is not a backup. Real backups have to be remote. What would Network Panda do if Choopa/Vultr data center is destroyed by a flood, fire or a plane crashes into it?

    There is no excuse for an IT company not having remote backups, redundancy or a disaster recovery plan. All the affected sites should have been up and running on different servers within a couple of hours at most.

    Sounds to me like the people who run Network Panda are learning their trade as they go along which doesn’t inspire confidence.

    • Hi Hands, thanks for the input. I am surprised too with how Network Panda handles these “fragile” backups and just put them on the same place where all other things reside. It’s like putting all the glasses with the spare ones on the same box!

      This blog has been down for about two days before I am able to restore it. But what can I do about it? All I do now is to secure my own backups and do not rely on any hosting company’s “backup” . For real, I am disappointed with Network Panda but I am still giving them another chance as they promised to improve their backup/recovery method. After all, they are kind and easy to deal with for the past 3 months I am with them. But, better to be safe, I am testing another host now.

      Many sites are still down (or empty) until today because of lack of backups. Many really lose confidence to Network Panda due to this event. Regarding with Choopa/Vultr, I still believe they should provide at least the backups or just terminated the server where the illegal content is. Not the whole account. Anyway, I do not know the whole (or true) story.

      • Hands

        I don’t believe for one minute that Choopa/Vultr terminated their services with 2 minutes notification. The real story is probably this incident was the last straw of a number of incidents. It doesn’t take a genius to work out Network Panda don’t know what they are doing.

        It’s absolutely ridiculous that they don’t have any working experience or knowledge of managing remote backups and redundancy. It’s basic IT. It makes you also wonder how secure their servers are? If they don’t understand remote backups they probably don’t know how to harden servers and configure firewalls.

        • Maybe that 2-minute span is the notification of termination and the notice/warning for termination happened 2 days beforehand. This time frame will be sufficient for Vultr to just terminate the account. Of course, Network Panda will not tell their customers they do not take action of securing backups (or moving all the sites) to other provider despite of knowing this. They will not also tell the customer that they secure backups on the same server/provider either. MAYBE. That will make more customers run away from them and probably ask for refunds.

          It is hard to judge any party about this since this is also not the first instance wherein Vultr is involved in such customer service practice. They are reported to terminate accounts without any more negotiation to the owner. I guess they strictly adhere to their TOS without any consideration for the client’s side. Also, there is also a host that have numerous servers down on the time of Network Panda’s disastrous event. I do not know if they are just simply connected with each other with the same provider or it just coincidentally happened.

          I am only speaking as a regular hosting consumer and thus, these are only my personal insights. I hope the clients, Network panda and Vultr learned through this.

          I am wondering if you are also a client of Network Panda? Or Vultr perhaps?

          • Hands

            Hi Raz

            No, I’m not a client of Network Panda or Vultr. I saw Network Panda spamming a forum with lots of cheap web hosting offers. If you apply their discount codes the prices are next to nothing so I decided to look for reviews on their service as I can’t imagine the service is good as the price wouldn’t cover the cost of labour in dealing with say 10 support tickets in a year unless their technical support is untrained and are being paid next to nothing.

            I came across one forum where Network Panda didn’t like one of their clients discussing his experience of downtime so Network Panda responded by terminating his account for lying on a public forum!

            http://www.webhostingtalk.com/showthread.php?t=1272532

            I also come across this website where Network Panda have blatantly created fake reviews about themselves. Look at the suspicious glowing reviews on 05/01/2015, 06/01/2015, 07/01/2015, 10/01/2015 & 31/03/2015. For starters, they were too lazy to wait a longer period of time than one day before adding additional fake reviews – with the exception of 31/03/2015 where they waited 2 months. You can also see the deliberate clicking of thumbs up for positive fake reviews and thumbs down for any negative reviews. This tells you what a childish mentality they have.

            https://www.sitegeek.com/networkpanda

            Then I came across your post and I couldn’t believe my eyes when you mentioned they don’t have remote backups.

            Let’s just say I won’t be using Network Panda but I feel sorry for their poor client base who have been let down by people who don’t know how to run a hosting company, but then again what do you expect for a hosting service which only costs a few US dollars a year?

          • I myself had read those you mentioned before signing up with Network Panda. Yes. The prices are insanely cheap that really makes me wonder how they do it considering that they use Paypal and they even accept 1 month billing. A good chunk of the money will just go to Paypal fees! It is like so good to be true. But, I still decided to risk a few dollars and signed up to the service because that time I am only using a free host and this very cheap service will be a nice testing field for me. It was a testing purpose only then. But, Until now this site is still hosted by them (I guess, the test is not done yet. :-P).

            So, why did I decided to try their service? The domain was registered on 2011 (but they claim that they offer hosting service before, but only locally) and has nameserver history recorded in mid 2012. Many cheap hosters do not last for a year (I am not saying this is an enough gauge).

            Regarding the uptime issue stated by a WHT member, maybe its true that time but I do not yet encounter the same thing. Like what I say, the issue with Choopa/Vultr is the only downtime (major) I encountered so far, with some 1 min downtime recorded somewhere in July. I do not have proof to show you since I reset the monitor when I’ve been transferred to another server.

            For me, I do not currently run something that serious that I still decide to give them a chance. But when this blog grows, pretty sure I will look for another service as I deem to be needed. And, just to add, recently I found more hosts that offers almost the same price with Network Panda. And they were also doing the business for even a longer time.

          • Hands

            Your outage message tells us why they are so cheap. They don’t host and manage their own servers. They buy reseller packages on shared servers from other web hosts e.g. Choopa/Vultr. If they hosted and managed their own servers then there wouldn’t be an issue with gambling websites as it would up to them to decide which accounts to terminate/suspend. If they hosted their own servers they would have mirrored hard drives on remote servers as back up so technically if one datacenter goes down, backup servers from a remote location can copy the data to new servers at a third location instantly then it’s a matter of waiting for DNS propagation for the domains to resolve to the new server which could be anything from minutes to a couple of hours.

            If you buy a WHM reseller package for $20 per month and stuff it with 500 cpanel accounts charging $0.50 for each cpanel account per month, the monthly revenue would be $250 at a cost of $20 – a gross profit of $230 per month. Rinse and repeat on 20 x WHM reseller accounts and you gross $4600 profit per month. Obviously there will be other costs involved like employing a third party company from India to deal with support tickets and spam forums / add fake positive reviews to hosting review sites but at a guess, this is their business model.

            The problem with this business model is that any Cowboy operation with no experience in IT can do this and that’s precisely what you have here. Just because there are other cowboy operations using the same business model doesn’t mean it’s good. Once you take away other costs there isn’t much profit left to spend on security / backups / redundancy etc hence these businesses place a low priority on security / backups / redundancy.

            From what I have read on the forums, I wouldn’t be surprised if they terminated your account due to this blog post. They seem very defensive and anti-freedom of speech.

          • Thanks for the words you just shared Hans.

            I believe they just use a VPS package, make it work then use it for shared hosting. Since Vultr is mainly a VPS provider. Am I right? I have very limited knowledge about this but it is how I see it. That’s where they get the “We manage our servers” thing.

            Well then, if I got terminated, then they should be ready for a new article to go online few hours after. That article may not be somehow “still positive” unlike this one. I just blog what I really feel and regarding with their service, I am not being biased and I am weighing things equally.