Todays Lesson in Humility: WPEngine

Intro to WPEngine

WPEngine is a a commercial WordPress hosting service that supposedly simplifies many of the usual administrative tasks, such as backup and recovery, staging, and security management for its users.

Like most WordPress hosting services, WPEngine lets one use a “custom domain” for a web page. So, for example, strategar.com – which in the way WPEngine works, is really just a CNAME record pointing at https://strategarlive.wpengine.com. This works reasonably well, and is at least no worse than the similar function on WordPress.com. It’s all a bit unclean when compared to just running WordPress on an apache or nginx server on hosted Linux, but is probably easier for normal humans to grapple.

A nice integration that WPEngine provides is integration with SSL to produce a certificate for the custom domain, using Certbot, aka Let’s Encrypt. The Common Name (CN) of the resulting cert is configured to be the custom domain name. This is pretty much righteous as a value-add, although really it’s just two commands on that Linux alternative. Still, some hosts used to charge big $ for this.

The WPEngine Staging Model

Built in a Snap

WPEngine appears to build its platform on top of a Copy-on-Write (COW) file system, perhaps Btrfs or zfs. These file systems have a nifty ability to rapidly “snapshot” a volume, preserving the state of all files in the volume. This lets one make a “backup” (really, a snapshot) of a web site (itself stored in a volume) quite readily. It doesn’t have all the benefits of a real detached backup. If the disk array that the data is on catches fire, the data is still toast. But it’s still pretty handy for recovering from user error or userspace hackers.

WPEngine Environments

WPEngine’s real claim to fame is what it calls environments. In addition to the production server image (with its potentially custom domain name), each site can have a parallel “staging” image, and a “development” image. Or, in WPEngine’s terms, a production environment, staging environment, and a test environment.

Recovery With Added WordPress Logic Means Migration

WPEngine builds on the fast-snapshot and volume-copy capabilities of its COW filesystem to let one “back up” one environment (say, staging), then “restore” that image onto another environment (say, production). With a little bit of added WordPress logic to automatically fix the root path in the wp-config file (and hopefully to fix non-relative URLs in the content), this lets one easily develop site updates on the dev environment, copy them to a staging environment for review and approval while going back to work on the dev environment, and then quickly roll them out to production when approved. This sounds AWESOME.

In the Bad Old Days (like yesterday and tomorrow), we might have done the same thing using the Updraft Plus plugin to back up one server and then push its data to another server. If one didn’t actually need backup (one does), this WPEngine feature could save $50 a year on plugin costs, plus some cloud service costs, and be easier to use. But there are flies in the ointment, and we’ll come back to that.

WordPress Plugin License Management and Staging

WordPress runs on plugins, after-market addon programs. A typical commercial site will use ten to twenty, most of which have to be commercially licensed (payed for). Licenses most commonly associate some sort of license ID (a serial number that says you paid for it) with the server’s host name. Copy a web site with a licensed plugin to another server (or another WPEngine environment), and it either won’t run, or more commonly, won’t update. And plugins get updated ALL THE TIME. They need updating like a college kid needs Saturday night. And Spring Break. And cheap beer.

Many plugin developers, however, understand the “dev and staging” model, and don’t make users buy additional licenses for the plugins running on these non-production systems. Most commonly, they use a special subdomain of the main site’s domain name to indicate that this is a non-production server. So for www.strategar.com we might have staging.strategar.com and dev.strategar.com. Nifty, huh?

This way one can install the license key on the dev server; it gets ignored, but gets copied to the staging server when development is finished. It gets ignored on the staging server too, but then gets copied to the production server, and voila, it works!

Where it All Goes Wrong

Remember we said that WPEngine doesn’t allow the site’s domain name to be applied to the dev and staging environments? Yep, That totally breaks the plugin licensing waivers for the dev and staging systems. They each need licenses. And what’s worse, after each migration, (say, staging to production) all the licenses have to be reinstalled for use at the new URL. This can turn a 2 minute job into an error-prone 2-hour job, with 2 hours of downtime. It Just Doesn’t Work.

Now, what one can usually do is just install a “production” license on the dev server, which lets it get updates, then propagate this (now invalid) license up to staging and productiuon. They won’t get updates, but most plugins will continue to work, so this mostly works. But not for sites that have user-entered content on the production system and therefore must do back-staging …

The Hell of Back-Staging and WHY WPEngine Staging Utterly Fails

Some web sites have end-users inputing data right onto the production server. One example is UncorkTexasWines. This site lets end-users (in this case, winery event planners) post events into the server, updating the calendar in real time. Development and staging in this sort of environment work VERY DIFFERENTLY.

Here, migration goes the other way. We might back up the production server, copy that data to the dev server, and then develop new code – that would then require either hot-patching of the production system to deploy, or a complicated ritual of locking the production system, copying its database down to the development server, integrating the changes, and then forward-propagating the whole mess back up. It’s hard but necessary. But that’s not the worst part.

Remember that plugins get updated ALL THE TIME? These updates occasionally break things. So one never wants to load them on a production server without trying them on a COPY of the production server. So the typical process is to back-stage a copy of the server (production -> staging or test), install the updates on the test server, and if they work, then install them on the production server. The problem is that this means that BOTH servers have to be eligible for updates – meaning they have to be in the same domain (and named appropriately). WPEngine chokes, dies, and fails utterly in this regard. So much potential, so much failure.

So, What Should They Have Done?

They should have figued out how to apply the “custom domain” to each of the staging and dev environments. It’s not that hard; it’sa few extra lines in the nginx config file for each environment. And if they added the subdomain names as DNS names to the certificate, they don’t even need to generate new certificates.

But no, they’d rather tell me I don’t know how the Internet works. Go figure. Christmas time is coming, and both Santa and I are members of the Greybeards. I hope they like coal.