OpenNeo Blog

This week in DTI: Performance upgrades!

Matchu — Thu, 03 Oct 2024 14:47:00 -0700

Hi, everyone! You might’ve noticed earlier this week that DTI had started to stutter a bit. We experienced a bit of a traffic spike, and pageloads would sometimes start taking a few seconds. (My phone was ringing off the hook to let me know!)

When we dug into it, there wasn’t any specific cause—just a bit more pressure causing our database to max out our server’s CPU a bit too often.

Rather than just throw more CPU at the problem (always an option, but never a cheap one!), we popped open the slow query log. We found the worst offenders, and started getting to work fixing database queries that have always been a bit slow—in some cases, for many years!

Here’s the three big changes we made:

Item searches: We sped up some of our tougher filters (occupies and fits) from taking a couple seconds at worst, to always nearly instant! (We did this by “caching” more data about the item’s appearance onto the item record itself.)
Trade listings: Earlier this year, we added the Offering/Seeking pages to Classic DTI, and built them faster than the Impress 2020 versions. Now, we’ve deleted Impress 2020’s slower version of the page, redirecting to Classic’s faster one instead. (This brings us closer to merging the apps, too!)
Modeling progress: When we’re missing data for new items, we ask for help at the bottom of the homepage—and we used the same upgrades from item searches to make this faster, too. (We also fixed the long-standing bug of Maraquan items asking for more help than we need!)

And these changes don’t just affect these features: when our CPU starts to peak, everything gets slower. And so when we make the site more efficient and stop hitting those limits, everything gets faster!

Since then, the site has been back to being steady, and I’ve been glad to see item searches—one of the most common things that happens on the site—run much faster for everyone!! And it’s always a joy to clean up decade-old code and bring it closer to what it always deserved to be 🥰

Hope everyone’s doing well, and that you’ve got fun Halloween plans! I’m excited to see your pets all dressed up again this year!!

By the way, I’m job-hunting right now!

After a few years consulting part-time, I’m looking for a full-time role again! (The tech layoffs came for me and my family, whuh oh—time to step it up!)

[Lore dump!] In addition to running Dress to Impress for 15 years, I’ve also been a senior engineer and tech lead at an edtech nonprofit (2015–2020), focused in B2B partnerships, team mentorship, and building integrated platforms to support multiple products across teams.

I’m primarily looking for remote roles hiring in the United States, leveled at senior or staff—but some of that could be flexible for the right situation!

Right now, I’m especially interested in:

Consulting firms, especially with a focus in nonprofit
Stewarding critical integrations long-term, like identity or payments
Shifting to information security, or technical writing
Engineering management, for the right human-focused team

If you happen to have an opportunity, please reach out and let me know! It’d help me out a lot.

Thanks for taking a look!
—Matchu

State of DTI: 2024

Matchu — Fri, 20 Sep 2024 13:15:00 -0700

Hi, everyone! Hough, this post is a long time coming but I have not had the power of Words—but today I do, and so we write!

First off, thank you for being so kind and helpful as you always are! Idk if y’all know that like, the community does the majority of the work for DTI? We’re just doing our best to keep the lil guy chuggin’ along underneath you! And thank you always for your kind words, too 🥺 It helps a lot.

After 15 years of DTI (what the heck??!), and 15 years of changing circumstances, our primary focus is keeping DTI around for the long-haul.

And, I’ll be real with ya, the choices I made in earlier times weren’t really designed for decades of operation 😅 (DTI I think is now officially older than I was when I started it? 🤐)

Back at the start, I wasn’t really uhh, good at working with others? So I never put together a formal team, and I built a codebase that was difficult to share.
And when we rebooted as Impress 2020, I borrowed the technology and design intuition I’d gained in professional software—not realizing that’s a level of complexity designed for teams of fifty to keep up with.

But now, here we are, with two apps instead of one, and just me as the operational bottleneck—and my own life focus has shifted more toward my personal household, with not even enough volunteer time to even do the work to hand over the keys 😖

So, today, DTI development is mostly a time management game. That’s why I’ve been focusing on long-term choices I hope will change this in the years to come! The themes are stability, simplicity, and sustainability.

I also want to call out and thank TNT in particular for their help with sustainability!! The NeoPass partnership has been a genuine game changer—I’ll talk about it more below!

Stability: Changes to keep the site running

At the start of the year, DTI was… uhhh, often broken 😖 We’d have multiple days of downtime, and I’ll be real with ya: it was mostly an awareness and time management problem.

Issues generally got resolved within hours or minutes of my sitting down to fix them, but… my world was large, and there were a lot of messes to clean beyond just DTI 😭 So, I didn’t discover issues quickly; and when I did, it would sometimes take a few hours, or even days, to become available.

When I was younger, it was easier to be on-call all the time… now, we have to be smarter about it. So, hey—we’re being smarter!

🔔 Alerts: We’ve set up automated notifications to tell us when the site goes down, or when a new kind of crash appears. (If you’re curious, we’re self-hosting GlitchTip!)
⚙️ Auto-monitoring: For crashes that kept happening, we configured our machine to intervene. (Like, if the site’s RAM usage gets high, we now auto-restart it.)
🧑‍💻 Simplifying: Reducing the complexity in our whole dang architecture has been huge for stability, too. (We’ll talk about it more below!)

With these changes, it’s been months since our last like, major downtime event. We’re still not as consistent as I’d like to be, but hey—more to come!

Simplicity: Cleaning house, clearing hallways

The high-level goal here is: I want to remove Impress 2020, but only once we no longer need it.

Right now, Dress to Impress is a weird combination of apps… I copy-pasted some of Impress 2020’s best UI changes over, but it all still used the Impress 2020 server code, even when you’re on impress.openneo.net.

This both doubled the danger of downtime (if either server goes down, that’s a problem for both!), and added networking complexities that are hard to track down (hello, “error loading species/color”!)

Like, it’s just not a great situation for stability, or for removing me as a bottleneck. There’s too much happening; and too much to know that, right now, only I know! 😬

So, here’s some of the changes we’ve been making:

➡️ Porting features: We finally brought Impress 2020’s improved Offering/Seeking trade listings page into the main app!
🌸 Simpler rewrites: We reimplemented the item page preview in Rails and plain HTML/CSS/JS, instead of being a fragile, frankenstein-y React component using Impress 2020 behind the scenes.
⚡️ Rewiring the wardrobe: The main outfit editor still has a couple API calls to Impress 2020, but very few! And we’re exploring the possibility of other ways to simplify that page, too.
🧹 Ancient code cleanups: As I wander the halls of the very old codebase, I occasionally stumble into something unused, or confusing, or way more complicated than it needs to be. And so I clean up, so my future roommates can be comfy here!

A lot of it is invisible work, toward the goal of destroying things rather than building them. And that’s important for us, I think: long-term projects with small teams survive by having less stuff going on.

I won’t get too much into why I chose to stick with upgrading the old app (built with Rails), instead of keeping the new Impress 2020 app (built with Next.js). Mostly just, I think the Rails app is simpler tech overall, and leans more into standards that I expect to be longer-lived—and so I expect it to be easier to maintain and welcome new collaborators into long-term!

We’ll check in with y’all before retiring Impress 2020 altogether, I think there’s still plenty of steps along the way! We’re just doing our best, bit by bit.

Sustainability: Finding more time, with TNT’s help

At the start of the year, the reality of the situation was just… I did not have volunteer hours available to give to DTI. What time I did have for coding was all going toward my technical consulting work, because our house needed the paycheck, y’know?

But you might’ve noticed, our site’s uptime got a lot better a few months ago—as it happens, around the same time we launched NeoPass! This was because the NeoPass partnership gave us the time to invest in other changes like alerts and monitoring, which just weren’t happening on their own.

The new “Shopping List” feature was also part of this partnership effort, too—though we didn’t end up building all of the integrated elements we were hoping for. (We were planning an “Add all to cart” NC Mall link, as a time saver for folks, and a way to help TNT know that DTI helped you out.) Ah, well—that’s product work! I hope it’s been helpful in this form, too 💝

Right now, we’re on a bit of a pause, as TNT’s dev team focuses on the Void Within plot and the other things on their plate. But our next plan is to look into (spoiler) ~~saving DTI outfits to your Neopets directly~~ 😳❗️ This is not a promise, but it is an intention! It’s the main integration y’all told us you’d be excited for in that survey we sent out, thank you for that btw!!

But personally, that’s why I’m excited about this partnership: I think we’ve done a good job threading the needle on building things that are genuinely useful for Neopians, not Worrisome, and ultimately building toward DTI’s own independent goals, too. I’m glad we’re not in the same stagnant place we were earlier this year, and I hope we can keep moving forward!

(By the way, TNT and I decided not to talk too much about the details of, like, what we signed and stuff… I would normally say more, y’know? But I hope this gives you enough to understand the needs and values we’re bringing to it! I promise I’m giving the community the best I know how to give, always 💜)

As always, thank you!

I hope this plan makes sense, as just one guy doing her best to keep the site online! And I hope this work can help set the stage for more people to join our team of maintainers in the coming years, too.

NeoPass for DTI

Matchu — Wed, 13 Mar 2024 12:20:00 -0700

Hi, everyone! We’ve got big news coming up: we’re partnering with Neopets to add “Login with NeoPass” and some other integrations to bring our sites a bit closer together! Here’s what to expect and why.

Over time, Neopets is planning to send more users our way, and we want them to have a smooth experience when they get here!

So, new users will be able to click “Login with NeoPass” and use their existing Neopets account, instead of creating a new DTI username and password. Existing DTI users can also link accounts if they want, too!

All of this functionality is optional, and removable at any time! Usernames and passwords will still work as before—and unlike official Neopets accounts that need long-term permanent linkage, we intend to offer both linking and unlinking, so you can always have options.

We also know that a lot of the pain points in Neopets and DTI right now come from transferring info between our sites by hand. It’s possible this could set us up for other smoother experiences in the future, too! (Nothing like that in the first release though—we’ve just been chatting with TNT about what might come next!)

Links to NC Mall

We’re also planning to add a few links from DTI to the NC Mall, which we’ll do our best to make thoughtful and unobtrusive. There’s two main reasons for this!

First off, when Neopets sends users our way, we don’t want them to get confused and stuck here. Existing DTI users know their way around NC, but new users probably won’t, so we’ll add a couple hints for how to get their designs onto Neopets.com.

The second reason is: we believe Dress to Impress is a critical part of the Neopets economy, and we want TNT to be able to see that, too. We’ll include a lil referral code in the link so TNT can know which shoppers came from DTI, and can evaluate accordingly. (We expect this to be important for us long-term!)

Why now?

Dress to Impress has always been a very small-staff volunteer project, and it’s been clear to everyone over the past few years that we’re struggling to balance DTI with the rest of our lives 😖 Work and life and family have their own needs, and they’ve been increasing!

And so… there are reasons we’re being careful talking about details right now, but the gist is: we’re hoping that partnering with TNT will not only help us fill gaps in the customization user experience, but can also be part of a more sustainable future for Dress to Impress long-term. I hope we can tell you more about it soon!

I know full well, and I’m sure you do too, that partnerships between companies and fan projects can be complicated. I promise I’m doing my best to represent you all, focusing on securing what’s right for the community, and keeping in mind the importance of autonomy! We’ll keep DTI independent, only do things we believe genuinely serve everyone, and keep a critical eye as we go.

So, yeah! It’s NeoPass time! We’ll be working on this in the coming months, and I’ll let you know more along the way. If you have questions or thoughts, please email me at matchu@openneo.net, and I’ll do my best to listen and help!

Thanks as always, everyone. We’ll talk more soon! 💖 —Matchu