OpenNeo

Adventures in Maintenance

Fri, 13 Jan 2012 15:03:00 -0800

Here’s the short version: more than a year ago I made a huge mistake in how Dress to Impress handles information. Today I fixed it.

Here’s the longer version:

Thursday night, we brought the main Dress to Impress server down for about an hour and a half, temporarily sending visitors the old server. The “enter a pet name” feature on the homepage was also spotty for the following 24 hours. Sorry for any trouble this may have caused, and thanks for understanding.

Usually when we push an update to the site, the server takes a few seconds to restart and most users are none the wiser. This time, however, we needed to make some very fundamental changes to our database structure that took quite some time to process, which would have made some sections of the site extremely unresponsive. Thankfully, the old server seemed to handle the job admirably. And everything seems to be running smoothly now, hooray!

And here’s the very long, very technical version:

First off, Dress to Impress tracks two different types of SWF assets: “biology” assets (like a Blue Shoyru’s head) and “object” assets (like how an Altador Cup Wig looks on that head). These SWFs have a unique ID number among assets of the same type—that is, though there can be both a biology asset #123 and object asset #123, there can’t be two biology assets with ID #123. Since both types of SWFs have a similar structure and need to do similar jobs (like produce PNGs of themselves for Image Mode), it seemed reasonable to store them in the same database table. Then, whenever we wanted to access information an SWF, we’d specify that we wanted, say, object asset #456 or biology asset #789. That was sufficient information for lookup, and everything seemed solid.

However, I’ll just be blunt about this bit: due to a painful inattention to detail, even though accessing SWF data works perfectly, updating that data has been critically broken for more than a year now. Specifically, after I would fetch biology asset #1337 and update it (for example, if TNT changed its zone ID), sometimes it would instead update object asset #1337. Ouch.

I can only assume that this sort of data corruption has been going on for quite some time now. For example, the Shadow Shoyru’s torso, biology asset #598, kept changing zone over the past few weeks (thanks again to everyone who reported the error!), because every time we saw the Mystery Island Lutari Anklets (which, umm, are bracelets, TNT), the site would attempt to update the zone for object asset #598. And update the biology asset instead. Sigh. This is also why so many Image Mode PNGs are broken: the wrong SWF was marked as converted, even though it had never actually gone through the conversion process. And it stands to reason that other data has been corrupted over the years, though I suspect that most errors are quickly patched automatically as time goes by, since every time a pet’s name is entered on the homepage, we update our data to match that pet’s appearance. Yay, crowd-sourcing!

(By the way, the whole 24-hour thing where loading pets by name was misbehaving: that was an unforeseen side-effect of the bug fix, and it has since been patched. Phew.)

So, we restructured our database, and everything now seems to be in order. We should never see that type of data corruption ever again, yay! However, I would like to point out that, though all seemed well in my basic testing—and I’m no longer receiving automatic error reports every single minute of the day, woot!—it’s very possible that I missed something. Please let me know at webmaster@openneo.net if you notice anything suspicious going on, and bear in mind that we may have to roll back the database if we discover a particularly large error.

So, that’s that. Thanks for being willing to handle a little downtime and for all the super-helpful bug reports that helped me track down this nasty little issue. It always makes me so happy to see that, even when I’m off at college and don’t really have the time to fully manage a big site like Dress to Impress, I don’t really need to: you guys take care of everything by yourselves. All the data you see on the site is added and updated by the community simply by entering pets’ names on the homepage, and it’s beautiful. Thank you for making my life so easy. You’re the best.

Happy new year!
—Matchu

New color: Eventide! A nice, pretty skyscape, to be sure....

Wed, 04 Jan 2012 19:48:00 -0800

New color: Eventide!

A nice, pretty skyscape, to be sure. I’m not convinced that we really need more new colors, but hey! At least they look nice.

Image converters back online - and everything's smooth!

Sun, 11 Dec 2011 13:26:00 -0800

UPDATE: Wow! The converters cleared the backlog much faster than expected. Everything should be smooth sailing now :)

So, as it happens, the Image Mode converters have been down for a while now, and finals week prevented me from looking into it too closely. However, I’ve since managed to track down the issue, and the converters are now back online and gleefully plowing through a backlog of SWF-to-PNG conversions. Hooray!

Unfortunately, this will likely result in a heavy server load over the next 24 hours. The converters are always on, waiting for conversion jobs, but those jobs only rarely appear. Today, however, the converters will be working non-stop to process their backlog, which may cause delays to the main site. Sorry for the trouble, and thanks for understanding.

For the technologically minded, here’s a brief run-through of the issue at play:

Conversion jobs failed to run due to a permissions error on the server: the converter would try to save some temporary files, but would be told that it’s not allowed to save files to that folder, because that folder for some reason belonged to my personal account instead of the converter’s account. I removed the folder, allowing the converter to replace it with a folder that belonged to the correct user, and all was well. Crazy how one small thing like that can bring down an entire feature, eh?

Stop American Censorship

Tue, 15 Nov 2011 22:27:21 -0800

Stop American Censorship:

The internet as we know it is under attack. Today, November 16, Congress is holding hearings on the first American Internet censorship bill, which would give the American government and the entertainment industry unprecedented power. This includes the power to block websites for infringing content posted by its users, even if the site didn’t know about it. The bill also makes it a felony with a five-year jail sentence to stream copyrighted content, even in an entirely noncommercial context, such as singing a pop song on Facebook.

We already have the Digital Millennium Copyright Act (DMCA), which gives the entertainment industry the power to demand that websites remove infringing content. The DMCA is what makes it possible for us to have free expression on sites like YouTube, Facebook, Twitter, and other social media platforms that allow us to both organize protests and live our daily lives. This bill would make it legally unfeasible to run websites with user-generated content, and could quite possibly spell the end of the Internet as we know it.

This bill is real and could pass this month. Don’t let the government throw the Internet into chaos with a Great Firewall of America. Stand up for a free society.

See AmericanCensorship.org for more information. And, please, don’t let this happen to the Internet we know and love. It’s the most powerful tool we have to express ourselves in the modern world, and such threats to our freedom are unacceptable.

Thanks for reading.
—Matchu

Botox round two

Mon, 22 Aug 2011 08:09:00 -0700

Sigh. Just like last week, Dreamhost started pointing impress.openneo.net to another customer’s website.

We emailed them asking them to both resolve the issue immediately, and find a more permanent solution. The Dreamhost staff implemented a short-term solution within minutes, though we haven’t heard back yet about why this seems to keep happening. In the meantime, newimpress.openneo.net still works whenever impress.openneo.net goes down, in case anyone asks.

Thanks for understanding. Happy Monday!

Sorry for the Botox downtime!

Sun, 14 Aug 2011 16:56:08 -0700

Well, that was odd. I woke up this morning and found that at some point while I was sleeping http://impress.openneo.net/ started forwarding to some cosmetic consultants website. Umm, weird.

To be clear: this was not any sort of attack against DTI, and the Botox site was totally unrelated to Neopets. Dreamhost had some internal trouble, and started pointing that domain to another customer’s website. The Dreamhost support team quickly resolved the issue after we reported it.

Sorry for the trouble, and happy Sunday!

Your Items is released!

Sun, 31 Jul 2011 21:18:00 -0700

Your Items is released!:

While waiting for the server upgrade for the most requested DTI feature ever, we’ve been working hard on the second most requested feature: the ability to track the items you own and want. We did it, and did it pretty darn well. Here’s the scoop:

Track your items: If you’re logged in, you should see on any item’s page the “I own this” and “I want this” buttons. If you say you own or want it, it’ll be added to Your Items, and be marked with a spiffy little icon across the whole site. You can also add items straight from the Your Items page with the neato autocompleting search form.
Sort into lists: You make lists of items you own or want, like an Up For Trade list, for example, or a list of items you still need for that next big outfit. Simple but powerful. Also, you can drag-and-drop to and from lists on Your Items, which, during testing, was hours of fun on its own.
Fine-tune privacy: Each of your lists has its own privacy switch, as do the items that aren’t in any list. You can make items private so no one else can see them, public so that anyone who goes to your items page can see them, or trading so that your name is listed on an item’s Infinite Closet page.
Trading 2.0: An item’s Infinite Closet page now includes a list of users who own or want that item and want to trade. So, if you need an item, you can see who has it and check to see if you have any items they want in exchange. It’s all very spiffy. Also, you can add your Neopets username to your lists page to make it easy for potential traders to Neomail you.
Import Neopets closet: I’m going to let the Neopets closet importer speak for itself. In short, you can get up and running with the items you own within seconds, and it’s pretty darn easy. Give it a shot :)
Export to Neopets petpage: The Neopets petpage exporter is similarly straightforward: your public items compiled into a simple but elegant petpage format. No need to send users offsite to share your trade lists with the Neoboards, am I right?

And that’s just the first iteration. As always, I’m looking forward to hearing from y’all on what works well, what doesn’t, and what we should improve. Please post specific ideas in our feedback forum, and send general praise or bug reports to webmaster@openneo.net.

Also, though I’m usually really, really good at responding to emails, I’m going to be spending all day tomorrow, August 1, with my little sister. Bonding and whatnot. Still, please drop me a line, and I’ll be sure to get to you as soon as I can :)

Thanks as always for your continued support! At time of writing, we’re $56 away from our fundraising goal. If you’re excited about this new feature and Dress to Impress in general, please donate to help us reach our goal. Thanks again!

Happy trading!

Happy teaser day! While Image Mode is pending a server upgrade,...

Sat, 30 Jul 2011 21:16:05 -0700

Happy teaser day! While Image Mode is pending a server upgrade, we’ve still been hard at work…

Upgrade successful!

Sat, 04 Jun 2011 20:16:11 -0700

Today we had some downtime here and there, but it was for a worthy cause :) In short, we swapped out some of the technology that runs Dress to Impress in order to reduce memory usage (that’s how much RAM the site is using) so that we can do more things at once.

Note that these changes involved moving to a different version of Ruby. I think I caught all of the major compatibility issues, but please keep an eye out for errors and report them when you see them.

Thanks for supporting Dress to Impress, both by spreading the word and donating to keep our super cool server running. You guys make it all possible :D

For the more technically minded…

Here’s the lowdown on our stack before the change. We were using the nginx web server (that’s kinda like Apache) as a proxy to share HTTP requests among three separate instances of Thin, a web server that runs Ruby on Rails applications like Dress to Impress. I originally chose this setup because Thin supports asynchronous connections: most servers can only handle one request at a time, which can be a problem if there’s one type of request that’s a serious bottleneck, like how typing a pet’s name into the box on the homepage means waiting on a request to Neopets.com, processing all that data, inserting it into the database, etc. However, asynchronous requests turned out to be too messy to implement, causing errors on various pages at random intervals. It would seem that the technology isn’t quite as polished as it needs to be. So, I instead left it at just having three Thin instances, since if one were busy loading a pet, the other two were probably free. (And loading a pet doesn’t really take that long, anyway.)

However, those Thin processes were running out of memory. Each Thin instance loaded up its own copy of Dress to Impress and all the libraries the site uses (and there are quite a few big ones!), which require a lot of RAM. This usually wasn’t a big problem, though once or twice a day Monit, our program that keeps an eye on the server, would restart a Thin process for taking up too much memory. Since it was only one at a time, y’all usually didn’t notice, since the other two stayed up; however, falling back on other servers is just our backup plan, and it’s not acceptable for that to happen regularly. Also, we’re working on a big feature that will take some serious processing power over the next few days—guess what it is!—so it’s important that we have significantly more RAM available without sacrificing the concurrency we get from multiple Thins.

So, we moved to Ruby Enterprise Edition via Phusion Passenger. That’s two important things there. The first is another version of the Ruby interpreter. Ruby is the programming language we use to make Dress to Impress, and there are a few different programs out there that run Ruby code. Ruby Enterprise Edition (REE) is a modification of the standard Ruby 1.8 interpreter, designed specifically to reduce memory usage. Phusion Passenger is another open-source project by the same company that replaces our Thin servers. Its main selling point is that it’s easy to plug into servers like Apache or nginx, but its best feature for us is that, when used with REE, Passenger also significantly reduces memory usage. It’s a clever trick: since every instance of Dress to Impress loads exactly the same fundamental application code, Passenger uses REE’s memory-sharing feature to only load one copy of the application code and share it among the application instances. So, instead of loading three separate copies of the app, we load three mini-servers that use the same application code. Much better for our memory usage by far :D

Hooray! We’re stable! Now the server is running more smoothly, and is able to also run a super cool background process or two…and those who stalk the Dress to Impress source code repository should know what they are for ;) It’ll be a few days until all that heavy-duty processing is done, but it will be so totally worth it. I’m excited.

Thanks for your continued support! It’s tons of fun to be able to make all this cool stuff for y’all. Here’s hoping you’ve enjoyed it :)

Weekend updates :)

Sat, 21 May 2011 20:11:00 -0700

We’ve had some fun this weekend fixing some minor issues around the site. Yay!

The first gender/emotion state for the Tyrannian Uni no longer has a big, fat White Koi sticking out of it. Please report any other glitched combinations you see, and note that, usually, a good temporary fix is to click one of the other numbered gender/emotion buttons.
Entries in the Infinite Closet items database now have prettier URLs and are significantly more search-engine-friendly.
We noticed that the NC Mall spider that automatically grabs previews of items being sold in the Mall (which, by the way, is 100% legit by Viacom’s terms) hasn’t been running for quite a few weeks. Turns out out deploy script wasn’t updating the crontab to refer to the app release’s specific path (), so it was trying to run the mall spider on an older version of Dress to Impress. This has since been fixed, and the spider is now working overtime to catch up on some things it missed.

In short, bug fixes and minor upgrades. Since major features, exciting as they are, can get exhausting to work on at great length. (But, man, are they exciting ;D)

Thanks for using Dress to Impress!
—Matchu

XSS vulnerability closed

Sat, 14 May 2011 07:02:00 -0700

Just a quick notice that Dress to Impress had an XSS vulnerability in the Infinite Closet section, which has since been resolved. We have no evidence of anyone abusing this vulnerability, nor would they would have been able to access sensitive account data, regardless.

If you’re interested, here’s the source code change that fixed the security issue. Short and sweet one-liner, escaping query output. Funny, because this is usually the first place I check when someone asks me to check their site for security issues xD I think I forgot that the Markdown interpolation doesn’t do escaping by default, like interpolation in most other spots in Rails 3.

TL;DR: Minor security hole fixed. It doesn’t look like anyone used it, nor would it have done them much good.

Thanks for using Dress to Impress!
—Matchu

FEATURE: Infinite Closet supports Baby, Maraquan, etc.

Mon, 02 May 2011 15:30:36 -0700

FEATURE: Infinite Closet supports Baby, Maraquan, etc.:

First off, sorry for the downtime. This quick feature is to make up for it :)

The Infinite Closet is our lovely wearables database, and it has a neat little feature for previewing items by species. However, that feature was flawed: some items only fit Baby, Maraquan, Mutant, 8-bit, etc., pets, some because they were just plain special (Baby Pajamas, Maraquan Dress, Elegant Mutant Cape) and some because they were paint brush items. The Infinite Closet would just try to stick these color-specific items on standard pets, and they wouldn’t fit, so it would claim that we’d never seen that combination before. It wasn’t very pretty.

So, fixed :) Now, all color-specific items show the right kind of pets at the bottom, so that the feature actually, ya know, works for those items. It’s the small stuff, right?

Thanks again for sticking with Dress to Impress. Y’all rock :D
—Matchu

Error Postmortem: Database write issues

Mon, 02 May 2011 06:03:21 -0700

Exciting things happened over the weekend. Namely, the beta server ran out of disk space. Oops. This caused any request that wrote to the database at all to just plain cut off.

Fortunately, the biggest offender wasn’t all those SWFs we downloaded (that’s at about 1.6GB right now) or even the MySQL database (nowhere near that). It was our server logs, that have been tracking each HTTP request we received since the server went live. It had hit about 5.5GB. Which is way too big.

So, we did something we should have done a while back: we now only keep access logs from the past 7 days. Protip: if you end up administering a web server, use logrotate from the beginning.

Anyway. Glad that was an easy fix. Last night we started pushing people to oldimpress by default, and are now pushing users back to newimpress. Yay!

Thanks for stickin’ with us through technical troubles. Y’all continue to be the best :)
—Matchu

ERROR: Database write issues

Sun, 01 May 2011 18:40:42 -0700

The server is having trouble writing to the database. Since I just got home from being out of town for, heh, days, I can’t investigate right now, but I bet we’re out of disk space. Looks like we’ll have to reevaluate our current strategy of downloading all the SWFs in order to handle cross-domain policy issues. We’ll be working on a fix very, very soon. Thanks!

Outfit saving all fixed :)

Fri, 22 Apr 2011 06:46:54 -0700

There was an issue with saving outfits in non-Chrome browsers. It has since been resolved.

It was kinda silly, really. I left in a line of code for debugging that tried to log a message to Chrome’s debug console every time an outfit saved. In browsers that didn’t support the console, the code would throw an error and never save the outfit.

This took me longer to resolve than most bugs, mostly because I was out of town last week, and spent much of this week recovering, catching up on work, etc. Thanks for keeping me up to date on the site’s issues even when I’m not around, and for understanding that DTI is a labor of love, and, therefore, working on it is subject to openings in my schedule. I’m happy to say that things seem to be looking up on that front, though :)

Thanks for using Dress to Impress! Show off them outfits!
—Matchu

Happy April Fool's!

Sat, 02 Apr 2011 00:03:06 -0700

Happy April Fool's!:

Hope you enjoyed my superb acting skills. *takes a bow as Lilia, Avast, Suzie, Jerry, etc.*

I know absolutely nothing about how restocking actually works; I just threw together an algorithm, said it was the official one, and watched the show :) Here’s hoping you didn’t waste too many hours on getting those hundreds of Draik Eggs that “restocked” yesterday!

Anyway. Shop Warlock lives on for your personal amusement and showing your friends how hilarious it was. Feel free to pass that link around for as long as you like.

Now, back to fancy business with Dress to Impress. Since the beta app seems to have been serving us well over the past few days, it looks like full migration day is fast approaching. The old and new servers are on slightly different databases: everything before November is the same, and everything after is totally separate. (Some users noted that their points went down significantly after we started automatically pushing users to the new server. Their points earned on the old server still exist; the new server just doesn’t know about them.)

Merging the two databases looks like it’ll be a fun challenge. The ideal result would be a true merge: contributions on the old server and contributions on the new server would be all grouped together, then we would recalculate everyone’s worth-their-weight-in-some-kind-of-pride-I-guess points to take into account contributions on both servers. If that proves not to be feasible, scores from the old server will take precedence.

Oh, hey, a quick aside: I know that many of the illnesses and cures from Neopets’s April Fool’s gag are glitched, only showing their appearance on one particular pet species. Since absolutely none of those items have anything to do with real outfits anyone might ever make in the future, I’m okay with that. Just thought y’all should know :)

Anyway. Hope you enjoyed the Warlock, and I’m looking forward to working on legit stuff for y’all in Dress to Impress ;) Thanks!
—Matchu

P.S. Mika C says hi from his lovely igloo home, and Lucie sends you all her formal invitation to continue to visit her shop regularly.

[NEW PROJECT] Shop Warlock

Thu, 31 Mar 2011 21:03:06 -0700

[NEW PROJECT] Shop Warlock:

OpenNeo now reveals its latest major project: Shop Warlock, an app that, by fancy ~~mathematics—~~whoops, magic—can tell you when an item will restock before it even happens! Tired of wasting time restocking? Now you only need to press F5 like crazy juuuust before the Draik Egg is scheduled to wander in—though, as always, you’ll still need to be pretty darn fast to snatch it up, or, if you’re up against the best, even notice it at all ;)

Curious? Give it a try! Shop Warlock is ready to stand alongside Dress to Impress as OpenNeo’s latest open-source Neopets app, meaning that it will be actively updated and maintained for a loooong time. And will also be just as awesome.

Here’s hoping that you find Warlock helpful :) Tell your friends! Spread the word! This is going to be one of the coolest things y’all have ever seen.

Happy Friday, and thanks for using Dress to Impress…and Shop Warlock!
—Matchu

[All clear?] SunnyNeo was hacked

Tue, 29 Mar 2011 15:14:00 -0700

This is a safety announcement regarding a website that is totally unaffiliated with Dress to Impress. We just thought you should know.

Update: Sunnyneo seems to be back up, but the Twitter account is permanently lost to the hacker. Unfollow. Also, the question mark after “all clear” is due to the fact that, after SN’s first announcement that the problems were solved, the site was re-hacked, and though currently not redirecting to the malicious site, doesn’t seem totally stable at time of writing, indicating that the hacker may still have access. Even though nothing really seems dangerous yet, it would be wise to wait a little bit before returning.

SunnyNeo, one of the big wigs in the Neopets fan site business, has been successfully hacked. Their website now redirects to a Neopoint-selling website, and their Twitter account has been similarly hacked, too, tweeting links to the malicious site.

Though there is no clear evidence of a cookie grabber on the site in question, it would be wise not to visit and to change passwords and the like if you have. Safety first.

Here’s hoping that SunnyNeo can get back on their feet and protected as soon as possible :) Good luck, team! And please, SN, let me know if there’s anything I can do to help.

Thanks for Dressing to Impress and staying safe.
—Matchu

Quick feature: "Share outfit" for logged-in folk, too

Mon, 28 Mar 2011 15:00:31 -0700

Got a quick e-mail today asking about if logged-in users could have that quick-and-nifty Share Outfit button, too. Put it in. Awesome :)

I’m a bit worried about how it will interact with the existing Save Outfit button. Everything looks good and working as expected, but lemme know if something seems off.

Nifty, less-quick features in the works :) Thanks for using Dress to Impress!
—Matchu

Quick feature: Outfits page

Wed, 23 Mar 2011 16:07:28 -0700

Quick feature: Outfits page:

So today Ataliah reminded me of a feature request from a while ago: a way to get straight to an outfit you’ve made rather than having to make a dummy outfit from the home page first. I was feeling in a particularly codey mood, so wrote it right up :) When logged in, you’ll now see a neat link up in the top right corner that says “Outfits,” pointing to a page that lists all the outfits you’ve created. Both easy and peasy.

It’s the little things that count :) Though there are some nice big things in the works, too ;)

Thanks, and enjoy!
—Matchu