As some of you undoubtedly noticed, we haven’t been processing stories correctly for a while. There was some bad mojo happening in our backend that wasn’t pulling down feeds properly. And without feeds, there weren’t any new stories showing up. The good news is that this should be fixed now and the server is currently playing catchup. The better news is that we’re putting in some protections for this and better notifications to us when things go wrong. The great news is that we’re also putting in place some optimizations to help speed things up in the process. So, despite the recent badness, there’s plenty of awesomeness coming up.

We’re also in the process of doing a rewrite of our core code. We’re slimming things down and taking what we’ve learned so far to build a better mousetrap, so to speak. The end result will be a better platform for us to add new features and run a stable service. So, stay tuned!

Update

Our optimizations appear to be doing the trick and your feeds should be populating once again. We tweaked the algorithm used in our job queue to pick which jobs to run, so there is a much more even distribution. The prior algorithm would bunch up the reading of all RSS feeds before it got to distributing and filtering them into your account. Now it will work on distributing each feed shortly after downloading it, rather than after all the other feeds have been downloaded. So, you should see faster turnaround times on your scrubbed feeds as a result.

In case you didn’t know, Jason’s really smart.

Summary (for the TLDR crowd)

We launched Feedscrub yesterday, it’s like a spam filter for your RSS feeds, you teach it what you like and it filters out everything else. We sent out press releases with invite codes to relevant blogs, networked locally (esp. via Twitter), and followed up on press inquiries. It netted us a ton of traffic, especially from our Mashable coverage, but we handled it by tweaking our MySQL queries during launch. See performance graphs below. And sign up for Feedscrub!

Prior to launch

We are 100% bootstrapped, and did all our own PR. It’s amazing how well you can do by being friendly and following up with folks. We sent out press releases to many of the popular tech blogs, several blogs that have previously covered our competitors, and local press/friends. Searching for your competitors’ brand on Google is one of the best things you can do to identify potential news sources. Make sure to network with your local start-up community–I started doing this 3-6 months in advance. As a CEO, it is your job to be well-networked with the start-up community. We were fortunate; Atlanta has recently developed a fabulous start-up community especially on Twitter.

Follow each blog’s contact/submission instructions to the letter. If they link to a blog post about how they like to be contacted, read it, all of it. You only get one shot to make a splash so do everything you can to increase your chances of getting coverage.

After I sent out all our PR I began to feel very nervous–it felt like 6 months of hard work was culminating in this one afternoon. I was especially nervous about scalability–if each of the blogs we pitched had covered us the traffic would have been crushing (although not an entirely bad problem to have). I had to tell myself to relax, and realize that we made a lot of preparations for scalability… caching, multithreading, multiple server support, etc.

The Launch

I woke up at 4:30am to fix a couple bugs and make a few improvements; then I started testing furiously. I checked everything top to bottom in every scenario I could think of, especially the sign-up process. We have settings in our config file that allow us to easily throttle our new users. Think carefully about this before you launch.

One thing we didn’t have prior to launch was a button to allow me to invite a user from our invite queue with one click. It quickly became apparent that that was something we badly needed, so I coded it up quickly and pushed to production.

Then we did our last push to production, launched, and waited. Several blogs picked us up right at 1pm, followed by an article in the Atlanta Business Chronicle that had some great praise and advice for us from a well-known Atlanta VC! We’re fully bootstrapped and that sort of VC intro was especially exciting. Soon after I received a tweet from Mashable saying they were going to do a story on us. Meanwhile we’re monitoring CPU load and slow SQL queries.

Everybody and their brother re-tweeted the Mashable post. Traffic picked up quickly and we noticed two particular queries–one running very frequently, the other was locking the table.

First we tackled the SQL query that was being called extremely frequently.

Tim and I feverishly re-factored our code; we deployed one fix almost immediately which helped significantly. We decided at that moment that PR for the launch was more critical than deploying the second optimization (which was also riskier than the first).

Around that time I started to feel great, perhaps one of the biggest highs I’ve ever had in my life. I had hoped that we’d get some good coverage but never thought it would feel so great to see my company on the front page of one of the biggest blogs on the net.

Well, a few hours later we got a second wave of traffic, and the optimization that we previously put off became immediately necessary. Slow queries compounded with high traffic, which caused MySQL to run up dozens of threads, meanwhile the hard disk was swapping like mad. I shut off all new users from the system (sign-ups using invite codes were re-directed temporarily to our invite queue), pushed the fix to staging and tested it quickly. When I pushed the fix to production we noticed an immediate noticeable decrease in load times and our job queue quickly caught up to the backlog. Crisis averted.

That second spike there? Yeah that was because of the locking query.

I finished off by following up on every blog that mentioned us, posting comments and responses to others’ comments. I feel like we’ve succeeded today.

Next comes the hard part: iteration. We’ve got a queue of features that we’d like to add but we’re especially listening to the users as they get to know Feedscrub. I’m sure they’ll have some great ideas for us.