Error Postmortem: Database write issues
Exciting things happened over the weekend. Namely, the beta server ran out of disk space. Oops. This caused any request that wrote to the database at all to just plain cut off.
Fortunately, the biggest offender wasn’t all those SWFs we downloaded (that’s at about 1.6GB right now) or even the MySQL database (nowhere near that). It was our server logs, that have been tracking each HTTP request we received since the server went live. It had hit about 5.5GB. Which is way too big.
So, we did something we should have done a while back: we now only keep access logs from the past 7 days. Protip: if you end up administering a web server, use logrotate from the beginning.
Anyway. Glad that was an easy fix. Last night we started pushing people to oldimpress by default, and are now pushing users back to newimpress. Yay!
Thanks for stickin’ with us through technical troubles. Y’all continue to be the best :)
—Matchu
