OpenNeo

[FOOLS] Server has flipped out

Hey Impressers,

As you may have noticed, we’ve been having some technical issues lately. Again, we’re really sorry for yesterday’s extended downtime. We’re almost back on our feet, so, as promised, here’s the technical breakdown of the issue.

We’ve been in contact with our hosting provider almost constantly over the past 24 hours, and it would seem that, due to a large electrical storm in the area, our server and many others in the building suffered a power surge, causing all of the bits on the hard drive to flip. Not good.

Thankfully, the team over at the hardware end has been working day in and day out to restore the original state of our server by reflipping all the bits back to their original positions. However, we’ve been informed that some of the data has yet to be unflipped.

So, if you notice any flipping-related issues on the site, thank you for your patience and please know that we are working hard to resolve the issue within the next 24 hours.

And Happy April Fool’s Day 2012! 

Dress to Impress gets a turbo boost!

Today was an exciting day of getting to know Dress to Impress a little bit better—specifically, which jobs take the database longest. And, um, some of the results were pretty frightening.

I’m rather ashamed to say that I can be pretty darn lazy when it comes to managing databases. Most of the site’s core data structure is years-old, from back when this project was little more than a neat idea. And back then I didn’t consider the fact that one day we’d have millions of rows of data (yep, another thing I learned today: millions!), which can be hard for a database to navigate without well-defined indexes on its data.

I won’t go into too much detail on a database index, except to say that, if you were a teacher filing all your students’ essays for future reference, it would be wise to index them by grouping them by student, so that looking up a particular student’s paper would be a matter of just checking the right folder instead of digging through the whole pile. In the case of databases, we have even more freedom: we can simultaneously index the same records by different attributes, so that the teacher can quickly look up all a students’ papers, the whole class’s papers on a given topic, all the A-grade papers, etc. without having to check each one. Clearly it’s a trade-off—you have to build these indexes as you go along if you want to use them later—but in many cases it’s a very, very good idea.

Dress to Impress, however, was tragically short on indexes, mainly because, when the project was small, it didn’t seem like a big deal. These days, however, some of the tasks the database performs involve checking each and every record to look up just one or two. Which is just plain awful, can sometimes take even a full second per query, and can easily be resolved with an index.

So, I took some time today to add some reasonable indexes—like an index that relates the outfit a user shares to the items on that outfit, which, without an index, is literally checking millions of rows. In fact, if you noticed 15 minutes or so of downtime this evening, that was the database chugging along, indexing all of our existing data. (If I’d known it would take that long, I would have waited until later tonight to run it—sorry about that!) Thankfully, everything has now been successfully indexed, and you should notice the site running much more quickly as a result.

So, thanks for bearing with some downtime, and I’m eager to see how much this small change boosts performance. (And I’m still a bit embarrassed that the database wasn’t indexed a loooong time ago, but at least everything’s all better now!)

Thanks for being awesome, and happy Wednesday!
—Matchu 

Adventures in Maintenance

Here’s the short version: more than a year ago I made a huge mistake in how Dress to Impress handles information. Today I fixed it.

Here’s the longer version:

Thursday night, we brought the main Dress to Impress server down for about an hour and a half, temporarily sending visitors the old server. The “enter a pet name” feature on the homepage was also spotty for the following 24 hours. Sorry for any trouble this may have caused, and thanks for understanding.

Usually when we push an update to the site, the server takes a few seconds to restart and most users are none the wiser. This time, however, we needed to make some very fundamental changes to our database structure that took quite some time to process, which would have made some sections of the site extremely unresponsive. Thankfully, the old server seemed to handle the job admirably. And everything seems to be running smoothly now, hooray!

And here’s the very long, very technical version:

First off, Dress to Impress tracks two different types of SWF assets: “biology” assets (like a Blue Shoyru’s head) and “object” assets (like how an Altador Cup Wig looks on that head). These SWFs have a unique ID number among assets of the same type—that is, though there can be both a biology asset #123 and object asset #123, there can’t be two biology assets with ID #123. Since both types of SWFs have a similar structure and need to do similar jobs (like produce PNGs of themselves for Image Mode), it seemed reasonable to store them in the same database table. Then, whenever we wanted to access information an SWF, we’d specify that we wanted, say, object asset #456 or biology asset #789. That was sufficient information for lookup, and everything seemed solid.

However, I’ll just be blunt about this bit: due to a painful inattention to detail, even though accessing SWF data works perfectly, updating that data has been critically broken for more than a year now. Specifically, after I would fetch biology asset #1337 and update it (for example, if TNT changed its zone ID), sometimes it would instead update object asset #1337. Ouch.

I can only assume that this sort of data corruption has been going on for quite some time now. For example, the Shadow Shoyru’s torso, biology asset #598, kept changing zone over the past few weeks (thanks again to everyone who reported the error!), because every time we saw the Mystery Island Lutari Anklets (which, umm, are bracelets, TNT), the site would attempt to update the zone for object asset #598. And update the biology asset instead. Sigh. This is also why so many Image Mode PNGs are broken: the wrong SWF was marked as converted, even though it had never actually gone through the conversion process. And it stands to reason that other data has been corrupted over the years, though I suspect that most errors are quickly patched automatically as time goes by, since every time a pet’s name is entered on the homepage, we update our data to match that pet’s appearance. Yay, crowd-sourcing!

(By the way, the whole 24-hour thing where loading pets by name was misbehaving: that was an unforeseen side-effect of the bug fix, and it has since been patched. Phew.)

So, we restructured our database, and everything now seems to be in order. We should never see that type of data corruption ever again, yay! However, I would like to point out that, though all seemed well in my basic testing—and I’m no longer receiving automatic error reports every single minute of the day, woot!—it’s very possible that I missed something. Please let me know at webmaster@openneo.net if you notice anything suspicious going on, and bear in mind that we may have to roll back the database if we discover a particularly large error.

So, that’s that. Thanks for being willing to handle a little downtime and for all the super-helpful bug reports that helped me track down this nasty little issue. It always makes me so happy to see that, even when I’m off at college and don’t really have the time to fully manage a big site like Dress to Impress, I don’t really need to: you guys take care of everything by yourselves. All the data you see on the site is added and updated by the community simply by entering pets’ names on the homepage, and it’s beautiful. Thank you for making my life so easy. You’re the best.

Happy new year!
—Matchu 

Image converters back online - and everything’s smooth!

UPDATE: Wow! The converters cleared the backlog much faster than expected. Everything should be smooth sailing now :)

So, as it happens, the Image Mode converters have been down for a while now, and finals week prevented me from looking into it too closely. However, I’ve since managed to track down the issue, and the converters are now back online and gleefully plowing through a backlog of SWF-to-PNG conversions. Hooray!

Unfortunately, this will likely result in a heavy server load over the next 24 hours. The converters are always on, waiting for conversion jobs, but those jobs only rarely appear. Today, however, the converters will be working non-stop to process their backlog, which may cause delays to the main site. Sorry for the trouble, and thanks for understanding.

For the technologically minded, here’s a brief run-through of the issue at play:

Conversion jobs failed to run due to a permissions error on the server: the converter would try to save some temporary files, but would be told that it’s not allowed to save files to that folder, because that folder for some reason belonged to my personal account instead of the converter’s account. I removed the folder, allowing the converter to replace it with a folder that belonged to the correct user, and all was well. Crazy how one small thing like that can bring down an entire feature, eh?

Botox round two

Sigh. Just like last week, Dreamhost started pointing impress.openneo.net to another customer’s website.

We emailed them asking them to both resolve the issue immediately, and find a more permanent solution. The Dreamhost staff implemented a short-term solution within minutes, though we haven’t heard back yet about why this seems to keep happening. In the meantime, newimpress.openneo.net still works whenever impress.openneo.net goes down, in case anyone asks.

Thanks for understanding. Happy Monday!

Sorry for the Botox downtime!

Well, that was odd. I woke up this morning and found that at some point while I was sleeping http://impress.openneo.net/ started forwarding to some cosmetic consultants website. Umm, weird.

To be clear: this was not any sort of attack against DTI, and the Botox site was totally unrelated to Neopets. Dreamhost had some internal trouble, and started pointing that domain to another customer’s website. The Dreamhost support team quickly resolved the issue after we reported it.

Sorry for the trouble, and happy Sunday!