Blog

Database connection errors on May 16th 2012 – Explanation

Posted by & filed under business, wordpress, wordpress hosting, zippykid.

Around 10:00AM PST, we started noticing a connectivity problem from our web servers to our database servers. We started investigating the issue, and got on the phone with Rackspace, to get a better idea of what was going on. After about an hour of being on the phone with them, we figured out the issue and the problem is resolved.

So what happened? The short version.

A configuration change was made to our environment, which started transferring traffic over an untested load balancer instead of the dedicated firewall. Rackspace is still figuring out how this happened, and why.

Long version

We’ve been growing like crazy, and with that growth has come architectural changes that need to be made, some of the changes we’ve made in the past have involved us switching to a hybrid cloud environment. Away from clusters built on cloud servers, into real iron. The first phase of this happened 5 months ago, we had a minor hiccup during that period, but it wasn’t as bad as this one. A few weeks ago, we noticed another need to upgrade, and we started talking with Rackspace about upgrading our infrastructure to what they called “Intensive”. This meant adding a dedicated hardware load balancer instead of cloud load balancers (Zeus).

This Sunday May 13th 2012, the load balancer was added into our configuration, it took less than 10 seconds, and everything seemed to work fine. The next step to this load balancer switch, was to coordinate a switch from the Cisco Firewall to this load balancer, for something called “RackConnect”. This allows our cloud servers to talk to our dedicated hardware. Somehow, the prep work that was being done to make this connection, got ahead of itself, and the environment changed. As of right now, no one knows why. Rackspace is investigating this.

We had plans on ditching cloud servers completely, in order to use a custom cloud built on Openstack, I’m actually in California right now at ChefConf 2012, and I was talking to someone from Dell about their openstack implementation on bare metal.. we may speed that up now, so we have more control.

If anything, all these experiences have given me some great insight into public and private clouds, and how and when to use them.

So, I apologize for the downtime, this is what happened, and we’re working on a way to prevent this from happening.

How to update your WordPress site automatically using Github and DeployHQ

Posted by & filed under themes, tips-and-tricks, version control, wordpress.

One of the things we’ve been working on over the past few months, is something called continuous deployment. Where, changes made locally on our computer, automatically get pushed to our servers for our viewers to see. What I’m going to describe below, is possible with any plan we have. We believe in eating our own dog food, so this workflow is possible without any add-ons, or change in plans.

This workflow, should work on any hosting provider too. This is not unique to ZippyKid, or WordPress

In order for this article to make sense, and to be useful to you, you will need to understand Github Flow and employ it within your organization and workflow. We’ve modified it slightly for our purposes, and how we see a lot of clients build WordPress sites for their customers.

For the lazy, make sure you have two remote branches, I prefer master, and staging, and an account at DeployHQ .

We keep a separate repository for our must-use (MU) plugins, and another for our theme. As changes are made to our theme, we push them first to the staging branch. We’ve configured Github to send a call to DeployHQ on every push to staging, which uses our SFTP credentials and deploys to our staging server. Once you and your team are happy with the output on our system. You can open a pull request to master. Once the request is approved by someone within your team. GitHub will ping DeployHQ again, but this time for the master branch, and things will be deployed to the live site.

Here’s some screenshots to make it easier to understand for some..

deploy hq deployment history

List of Servers to Deploy To.

Pull request to master

In the screenshot above, I’ve purposely shown how this can work with FTP and SFTP accounts. I don’t recommend you use FTP though.

MyZippyKid Plug Once you get approved into the MyZippyKid program, you can create individual users just for deployment, that will be jailed right into your theme folder. So, you don’t need to remember arcane paths to do this properly.

Our Net Promoter Score (NPS) results

Posted by & filed under wordpress hosting, zippykid.

Last week we sent out a survey to our customers, asking them if they would recommend us to other people. We got an astonishing amount of feedback on that. Apart from people taking the survey, I got a lot of emails back to me personally with details about what we’re doing right and wrong.

In the business community, this is called the Net Promoter Score (NPS). It’s based on the book The Ultimate Question.

Our average score was 9.91 out of a possible 10. I had a suspicion this would be the case, but it’s great to be validated by the survey, and the outages we had last week due to the attack on our systems. We didn’t lose a single customer. Our customers stuck with us, when they could have bailed.

At ZippyKid, we don’t treat customers as numbers on a spreadsheet, it’s not a customer acquisition game for us, our #1 focus is on making the existing customer happy, and doing the right thing for them. I believe the results of this survey show that we’re doing a good job of that. This is why we’re growing so rapidly, with a marketing budget that’s a fraction of our competitors. Our customers do the marketing for us.

I want to thank Kai, Rob, and Heidi for this. Without their dedication and commitment to our customers, this would not be possible. They’re the best sales people anyone could work with.

If you run a business of your own, do you track your Net Promoter Score? Why not?

Explanation of recent outages over the past week for around 400 customers.

Posted by & filed under wordpress hosting, zippykid.

For the past week, around 400 customers have been affected by outages, or degraded service on one of our clusters. This cluster is under what’s known as a Slowloris attack, where good connections are being mixed up with bad connections. It’s a slower form of a DDoS (Distributed Denial of Service) attack.

Symptoms of this are: 1. either site becomes completely in-accessible, some parts of the website become inaccessible (/wp-admin/)..

We’re working very hard to mitigate this attack with our upstream network provider, Rackspace. I’m really really sorry that we don’t have a very good solution in place yet. But, we should have it in place by the end of the day today. After going through tons of logs, tons of packet sniffing, and understanding exactly what’s going on, we think we have a solution that will prevent this from making your site inaccessible.

To give you an idea, we’re seeing all sorts of spikes.. like this



These aren’t bringing the servers down, just choking the outbound traffic. Which makes it look like the site site is down.

Co-incidentally, our friends up in Austin (WPEngine) have been having an opposite DDoS attack, where they’re getting flooded as well, we’ve been working behind the scenes exchanging ip addresses and symptoms to get a better handle on things.

Once we all get a better handle on things and service is reliable again, we’ll be visiting with Rackspace and their Network engineers to figure out ways to prevent and diagnose these attacks faster.

One thing we’ve already done, is ordered a load balancer from F5, we’ll be switching off the Cloud Load Balancers over to this device, as it’s much more capable than the Zeus load balancers that we were using.

What’s MyZippyKid all about?

Posted by & filed under business, wordpress, wordpress hosting, zippykid.

We’ve been getting a lot of requests from people who are eager to get into the beta for MyZippyKid. Since it’s beta, we’re still tweaking a lot of things, and entrance is currently limited to those who we’ve worked with in the past. Mainly, because they were the ones who helped drive the feature set, and understand that it’s beta. We don’t want all the new folks to use the beta, find a bug and have a worse experience. It’s getting very very close though. To satiate the people who want to see things, and possibly touch things.. we’ve put together a video.  This should give you an idea of what’s in store for 2012 at ZippyKid.

 




The game my dear friends, is afoot.

Announcing MyZippyKid – The best way to host and manage your WordPress sites.

Posted by & filed under blog, business, wordpress, wordpress hosting, zippykid.

Almost one year ago today, I signed the first ever term sheet in my life. Basically, I took money from investors, in return for equity in an idea I had, which seemed to have some legs, and the investors believed I was crazy, yet capable enough of making it grow. Thank you Gabriel Weinberg , and friends, for trusting me with your money. Since then, we’ve built a team of 6 awesome individuals, with one goal.

Our goal then, and today is the same. To build the best environment for businesses to build and host their websites.

We introduced some new concepts to the hosting industry, we were the first to stop using bandwidth and disk space as a means to pick a hosting plan. I always found those metrics confusing, more so, I used to get a lot of calls from people who had no clue how much disk space their website took up. It’s an internal metric for hosting companies, not website owners.

So, we came up with a plan that everyone would understand. I looked at the wedding industry, and borrowed from it. The more people that attend your wedding reception, the more you’ll end up paying for it. Hence, visitor based pricing was born. It seems obvious now, but go look at hosting plans prior to 2010, no one else did it. This idea took off, we were able to
deliver on our promises, and make people very happy. As we went from less than 100 sites, to over 4000, we noticed how people worked with their sites, and our system. Overall people are happy, but there’s always ways to make people happier. Today we’re launching something that will make existing, and new customers happier.

Announcing “My ZippyKid”, it’s a new way to work with WordPress sites, it’s the result of analyzing over 7000 help desk tickets, emails, phone calls (pre sales and support).

Here are some highlights of what’s coming.

  • Single sign on- If you want to, you can now sign up with your Facebook or Google account (Twitter and LinkedIn coming soon). 
  • Collaborators – We borrowed this from Github, and other code repository sites, if you need to invite a programmer to work on your site, just add him as a collaborator. No need to hunt for FTP credentials and other technical jargon he’s asking for. More importantly, you never need to send it over email any more
  • Zippy Clone – We see a lot of you building sites for specific verticals. We see the amount of time you put in to get each site up and running, with the same base template. Now, you’ll be able to take any site on our system, tweak it, and then when you’re happy with it, launch as many copies of it as you want. (Master templates coming in Q3 2012).

 

 

If you would like to joint the beta, go to zippykid.com/beta/ to get in line for the beta, we’ll be inviting people starting Tuesday, April 10th.

 

Managed WordPress Hosting Support – What you should expect from us and any other good WordPress host.

Posted by & filed under blog, business, Managed Wordpress, speed, tips-and-tricks, wordpress, wordpress hosting, zippykid.

This post explains some of the support options and level of support you can get from us here at ZippyKid. Whether you’re hosting a high traffic WordPress site, or a low traffic, but high value (City government) site, you need a way to figure out where the bottlenecks in your code are.

We can help you identify issues with your HTML, your database queries, and even calls to external services like Infusion Soft, or Facebook. While our competitors will give you a one page analysis of what’s wrong with the resulting HTML of your site, and tell you to use gzip, change expiration headers, and other low hanging fruit. We can actually tell you if your call to a Twitter feed, Facebook, or our database is the bottleneck.

Knowing this, enables you to make better informed decisions on how to optimize your website, rather than changing options in some poorly documented caching plugin, or viewing the results of some “page analyzer” that only looks at what you threw out 14 seconds after the request.

For example, the graph below, you can see that the server is done doing it’s thing in less 500 milliseconds, the network is done in less than 2.5 seconds. The rest is left up to the browser to process the javascript/html/css and render the page. Webpagetest.org gave this page a score of 95, because our system had already done the setting of headers, CDN, and image compression. But a browser that still takes 7.5 seconds to render? That’s unacceptable.

browser analysis
Uploaded with Skitch!

Now look at this analysis of the actual request.. less than 600 milliseconds, and the server is done, the rest is up to the network and the computer.

server-time-analysis
Uploaded with Skitch!

Check the external services analysis, now we can tell you how much time your site is spending talking to external services..

external service analysis
Uploaded with Skitch!

And lastly, we can analyze your custom queries, plugins and tell you where things are slowing down if at all.

Database activity - zippykid - New Relic
Uploaded with Skitch!

This analysis is available to our support staff 24×7, and available to enterprise customers ($250+/month), via the control panel at all times.

Nobody else in the managed hosting business, let alone in the Managed WordPress Hosting business is doing this, except for us. I hope the others step up their game and start offering this.. it’s not hard, and it’s good for the WordPress sites and the internet as a whole.

Worst Week Ever – Growing pains at ZippyKid

Posted by & filed under business, hosting, Managed Wordpress, php, rackspace, wordpress, wordpress hosting, zippykid.

I’m sorry for the worst week ever, in the history of ZippyKid, and it’s happy customers. We’ve been having slow downs and downtime across the board. We’ve isolated the problem down to a strained and maxed out Firewall, which is operating at 80% CPU capacity, and near full limits of it’s bandwidth capability.

We’re working very closely with Rackspace to get this managed device replaced, but due to the size of our network, doing so requires downtime, and coordination of the complex rules across the board. I’m really sorry for all the delays and not being able to give a good estimate of when things will be back to stable, we’re guessing by Friday night everything should be 100% stable, and zippy again.

Once again, I apologize profusely for this downtime, and my lack of communication throughout this, I’ve been busy trying to figure out the problem, and then working with Rackspace on a solution and procedures to implement it thoroughly.

I messed up, I didn’t see this coming sooner, I got caught up in the growth and dealing with new customers, but this is just a reminder to me, we’re a startup, we’re a small company and we need to slow down every now and then, and take inventory of where we are.

Once we fix these major issues with the existing network (we’re adding 3X capacity to the firewall), we’ll be able to introduce some of the major things we’re trying to introduce to the world of hosting.

The following two charts, are near real time graphs of the traffic being pushed through just two of our systems.. now imagine similar traffic across 20+ systems, and one firewall trying to manage it all.

Custom error messages for Rackspace Cloud Load Balancers

Posted by & filed under blog, rackspace, tips-and-tricks, wordpress hosting, zippykid.

If you use Rackspace Cloud Load balancers like we do, sometimes seeing the “Service Unavailable” message is annoying, and somewhat ambiguous to the visitor. In November Rackspace allowed you to upload your own error pages via the API, we’ve started implementing it for some of our Enterprise clients.

It’s a relatively simple process, once you go through the API and figure it out. Or you could use the python library and this Gist, by our latest team member Matt.

ZippyKid 2012 – Changes coming to ZippyKid

Posted by & filed under blog, business, Managed Wordpress, wordpress, wordpress hosting, zippykid.

In April of 2011, I decided to make ZippyKid a full time job and focus all my efforts into growing the company, my goal was to make a new kind of hosting company, that focused on WordPress and small businesses. I wanted to make the best WordPress hosting company, not just another company that manages updates to the WordPress core and plugins for you.

I realized back then, that saying “We upgrade your WordPress” as your only selling point, is like a restaurant saying “we wash your dishes when they get dirty”. We needed to do more than that, and we have.

Fast forward from April of 2011, when I only had 65 customers, to today, where we have several thousand, the company went from being just myself and Kai part time, to 6 full time employees. We are trusted from small mom and pop shops in small towns all over the world, to larger names like Reuters, and even a forward thinking government agency. Starting in February, we’ll be a team of 7. We’ve learnt a lot, and we know what we need to do to continue to be the best, and better than anyone else.

Originally, this post was going to be a list of 7 things we’re about to release in 2012, then I realized I didn’t want to be a vapor-ware company, so here are somethings we have either already launched, or are rolling out in the next 30 days.

The biggest one is that our base price will increase to $35/month starting February 29th 2012. If you sign up before February 29th 2012, your prices won’t go up.

  1. SSL for all. Because security is not an add-on. I had written a blog post in September talking about the need for this, I wanted to get this done, we know what to do and how to do it, but large scale implementation wasn’t/isn’t easy. You have to order SSL certs, you have to get certs validated, and you need to get a unique ip address. Currently we charge extra for SSL, but it’s something that should be offered standard.
  2. Email (already active) You’ll get 10 email addresses and unlimited aliases included with the base plan. It doesn’t make sense for you to have us host your WordPress site, but you go elsewhere for e-mail. We’ll either setup Google Apps for you, or set you up with Rackspace Email.
  3. Malware Scanning (already active) An independent third party (Sucuri) is scanning your site 3 times a day for malware and vulnerabilities, in the small chance you do get hacked, or are running an insecure version of WordPress or a plugin, we’ll be notified and we’ll fix it.
  4. SAS 70 Type II Compliance (already active) We’re the only WordPress hosting company that offers SAS 70 Type II compliance, basically, it means we take your data seriously, even if you’re just hosting pictures of cute kittens. They’re your cute kittens and no one but you has access to them.
  5. Quarterly Security Audits and Results (first one scheduled for February 2012) Everyone claims to have a secure network, who wouldn’t? We’re going to get audited by a third party (Trustwave), and we’ll post the results on the web quarterly, for all to see and judge for themselves.
  6. 30 Day Free Trial We’re so confident in our product, that we’ll give you 30 days to try us out, if you’re not happy, you can cancel without paying a dime. You’ll even be able to take the SSL certificate with you.

There are some other changes that I haven’t mentioned yet, as we’re still perfecting them. Our goal at ZippyKid is not to be the first with features, but the best. Keep an eye out on the blog and our newsletters, and you’ll find out about them shortly.