Downtime on Sunday

By April 17, 2007 8 Comments

We’re growing, and I’m sure that’s pretty obvious to all of you. From a technical perspective this has presented us with quite some challenges. We’ve needed to optimize the application almost weekly to meet demand, without pouring vast amounts of cash into new servers. We’ve needed to be on the ball monitoring load and performance as well, in order to respond when bad things happen. Testing, in a development and pre-production environment, has had to be ramped up considerably after we’ve repeatedly found our tests don’t address the sorts of issues tens of thousands of users throw at in the live environment.

It’s been a challenge, to say the least.

Along the way we’ve had some issues and failures, but also some successes. They tend to go unnoticed by and large. While there was something of a furore surrounding our segmenting out filled opportunities last week, it’s worth pointing out that that release also introduced zero new errors. We even made a release on Friday to address some on-going bugs and issues, and that went very smoothly indeed, to the point that we didn’t even take the site down to the deployment. Things are definitely improving.

We have reached a point now though where we need to physically scale the application once again. We last did this back in October 2006, adding new servers to meet the demand of a growing user-base, building in fail-over resilience and generally making the application more robust. We’re about to do it again.

We’ve outgrown the facilities of our current hosting provider and on Sunday night will be moving the application servers to a shiny new rack at a brand new data center here in Orlando. Unfortunately this means the site will be unavailable for a time. To minimize the impact on posties and advertisers alike, we’re making the move at 9pm Eastern (GMT – 5), and are blocking out a window of 3 hours to complete the move. This means the site will be back online at midnight Eastern time, just as Europe starts to wake up and do business.

By picking a Sunday night we hope not too many of you will be inconvenienced by this. I know a lot of posties blog at the weekends, but the hope is that most of you get your blogging done before Sunday evening. We also know that very few Advertisers ever log on and create opportunities on a Sunday night. The reviewing team here tend to work very strange, jaw-dropping hours so this give them a bit of a break which isn’t a bad thing at all.

This move sets us up for the next phase of, a phase in which we massively build out the offering to posties and advertisers and shore up our position as the first and most prolific consumer generated advertising network on the Internet. Thanks for taking the journey with us.


Join the discussion 8 Comments

  • Anonymous says:

    I think you guys should consider scrapping RoR and rewriting in PHP. RoR scales horribly from everything I’ve read and your site is proof. PHP isn’t “hip”, but at least it’s fast.

  • Pete Wright says:

    That’s an interesting viewpoint. PayPerPost was initially a PHP application but the technology was switched to Rails when we first went live as the PHP code was horribly unmaintainable, inelegant, and had that usual PHP mix of logic with presentation that’s so conflicted with good OO design principles. In terms of scaling, Rails scales just fine – the problem we’ve had is the rate of growth of the userbase. In 7 months or so we’ve gone from zero to millions of site hits. Growth has outpaced us to this point, but we’re addressing that. In terms of ease of scaling, it’s actually very easy to scale a Rails . The entire framework is built around the concept of horizontal scalability, so scaling the app (aside from the code optimization tasks) is simply a matter of adding more machines to a rack. The problem though, as I’m sure you know if you’ve ever worked on a massive online app the scale of PayPerPost, is that it takes time to order rack space, build machines to go in that rack etc etc. We’re working on a solution to that ongoing problem as well, with a little help from a partner of ours.

  • Anonymous says:

    Sunday works for me- I hardly do any blogging on the weekends anymore.

  • Tricia says:

    Good luck with the move- I hope it goes smoothly. I think the weekend is a good time to do this. You guys don’t put out new ops very often on the weekends lately – perhaps that’s because advertisers don’t create them, I don’t know, but with few new ops going out on weekends that probably is the best time for something like this and even for big system updates.The only reason there was a “furor” regarding the segmented red already filled ops a week or so ago was because we weren’t told about it before hand. So when we started to notice it many of us thought it was an error, or that our disclosure badges weren’t working (because there were changes to that at the same time too) or maybe even that our categories got wiped out. If you guys would make a post on the blog and forums when changes that we are sure to notice are about to happen – before they happen – I’m certain you’ll hear fewer complaints about it. Yeah, there still might be complaints, but likely not needless ones for something that is working as you planned. I’m sure we’d praise the behind the scenes stuff that we don’t really even seem to notice if you told us about it too. 🙂

  • Slithy Toves says:

    And as far as segmenting out the filled opps, I think the furor was primarily about the lack of awareness of what was going on. We saw it all as a bug, and that was concerning.I would STRONGLY prefer that the filled opps be shown as such so I don’t click only to get those dreaded words, which then boots me back to 20 opps per page/All Opportunities, when I really only want to see 50 opps per page/Qualified Opportunities. Loading the opps page is slightly quicker than before, but still takes quite awhile, so it is definitely an annoyance to click an opp only to find it was a waste of my time…Maybe they get highlighted in orange or yellow and slotted just under the truly open opportunities but before the non-qualified opportunities in the next release? Just an idea…

  • DewKnight says:

    Would you happen to be moving to DIMENOC? They’re like right across the street from you (111 n orange)

  • Owen Cutajar says:

    Good luck with the move guys. Ohh .. shiny new servers .. you’ll have to post a photo or the rack up one of these days

  • kat says:

    Best of luck with the move. I’m sure it will only help make PPP better and better.

Leave a Reply