Best method to update Craft on a live site to avoid losing data

Question

What is the best way to update Craft on a live website? Preferably without losing any data?

There are a few ways to do this, and, can often be situational. Essentially the main problem is potentially losing out on data that’s POSTed during the site being updated.

If a site is small: I often crawl the site before hand, making sure everything is cached with varnish, do a DB backup, then do the update locally (sync’d with the live DB), and then push & deploy the files. (Varnish holds the site up despite it being unusable behind the scenes). Even if the DB is a local, there is still downtime involved in updating it.

If a site is bigger, or the Craft versions differ dramatically: Though more time consuming, I would setup another directory on the same server (with an independent updated database), make sure it’s working on a test link, then switch the nginx config to point to the new directory for that site.

In any situation I think that all situations could be detrimental to a user. If a user has written out a long form which could potentially be submitted during an update, or, a client could be logged in to Craft, working on a big edit to a page and hit “save" (I know you can disable their access), but the same issue exists, when you use that feature too.

Answer 1

I’ve written a plugin that allows to announce scheduled maintenance in the CP and on the front-end by making use of the template variables it provides.

https://github.com/carlcs/craft-maintenance

Here’s some example code for how you can announce the maintenance

{# Announce maintenance 1 hour in advance #}
{% set announcement = craft.maintenance.getAnnouncement('1 hour') %}

{% if announcement and announcement.blockCp %}
    <span>{{ announcement.message }}</span>
{% endif %}

And to make sure your site visitors don’t add or update records to the database you can add conditionals like so

{# Disable shop link when the site is undergoing scheduled maintenance #}
{{ isCpMaintenance ? 'Shop closed' : '<a href="/shop">Shop</a>' }}

Answer 2

I think you're asking the IT equivalent of "If I go outside when it's storming, how can I not be hit by lightning?" There's always going to be that chance. When you run a public server, that's pretty similar.

I'm not a high availability expert but as someone's who ran servers on behalf for clients, what you're really asking is 2 questions:

1. How much downtime can we afford?
2. How do we engineer a solution to minimize that downtime?

As you alluded to, how you answer question 1) is going to inform 2). Properly answering question 2) is probably beyond the scope of this Q&A format, but since we're going the down the rabbit hole...

If your answer to question 1 is next to zero, then you need to evaluate how much time/effort/money will required to do this. Does it matter if your mom and pop's brochure based site is down for half a minute while you update? Probably not.

If you're running a multiple server setup for a largish organization, then this is also a question for them and what they're willing to throw at IT to make that happen. No server is 100% available, 24/7 and responsive to users. The companies who do this well are also compensated accordingly. When you add redundancy to the mix, you also double your potential for failure so it's a double edged sword.

You can minimize that chance of messing up a user's work by doing a few things to proactively protect against downtime and just as importantly, frame the user's expectations:

Communication

When do your users usually work on the site? If the site's users and authors are typically 9-5 corporate in nature, update the site off-hours when less people are going to be working or visiting on the site. And if you think downtime is going to more than a few minutes, inform them of the maintenance window.

At least in terms of Craft, updating takes seconds usually. I liken updates to a frog jumping from lily pad to lily pad. The lily pads are point updates. The smaller jumps are going to be a lot easier to make then jumping from one side of the pond to the other.

You're also spreading out your uptime with multiple smaller jumps instead of huge leaps. If you put off updating, you're increasing the risk of something blowing up later on (schema updates, etc).

Updating software is not like updating hardware where you know the hardware is going to fail at some point, so you plan for it. Writing bug-free software is harder. Being agile here is important. You also run less of a risk of having unplanned downtime because you didn't jump and now there's a huge security update.

Engineering

The other important piece of uptime is if something does go wrong during the update, how do you recover? When Craft updates, unless you've turned off automatic database backup, it'll save a copy in case things go hairy and then revert back so you're not stuck in a weird state.

I would also question the need for a long form to begin with. Can we engineer the form in such a way that it saves the user's work periodically (either to their browser or the server) as they fill it out? Or make each form part shorter?

If someone is working on a long form, it's counter intuitively also less likely they're going to be hit by server downtime, because they're spending more time on it before it gets submitted.

Lindsey D points out that how you deploy can make a difference as well. Deployment services (eg: DeployBot) in coordination with some VPS providers offer atomic deployment. This guarantees that your site remains "untouched" until the new files are 100% uploaded. Using something like Capistrano can do this as well. AWS offers their own set of deploy and server management tools to help manage your uptime too. (Again, you pay the piper for all of this, which is why AWS gives you a year for free in their free tier.)

There's probably a few things you can do but those are just off the top of my head.