Contents

Staged rollout and automatic rollback

About 7 minApplies to v2.3.1Updated 2026-11-02

Configuration and node-program changes are never pushed to every node at once. They are delivered in stages, so a problem stops where its blast radius is smallest.

Configuration changes: three stages

Configuration changes such as sites and rules move 1 node → 10% of nodes → all nodes. The next stage starts only when every node in the current stage has installed the version with no new alert. If a node fails to load it, the rollout stops at the current stage and raises an alert.

  • Several edits made within 10 seconds are merged into one version, so a run of saves does not create a string of tiny versions.
  • You can pause automatic publishing for 30 minutes to fit a change window.

Under System Admin › System Upgrade, on the Node tab, the "Config publishing" section shows the version being delivered and its stage, and lets you intervene:

ActionEffect
Advance nowSkip the wait and move to the next stage
Pause / ResumePause or resume automatic advancing
Roll back to previousReturn the configuration to the previous version
Push to all nodes nowSkip the 1-node and 10% stages and deliver to all nodes; requires ticking a confirmation box and is written to the operation log
Caution "Push to all nodes now" gives up staged protection. Use it only for urgent fixes.

Node-local automatic rollback

For 5 minutes after a new configuration takes effect, each node watches its own health. If the 5xx ratio, worker crashes or the rule budget clearly worsen, the node returns to the previous configuration by itself and refuses that version. The master raises a "node automatically rolled back config" alert and the rollout stays at the current stage instead of widening.

The node list and node edit page show "Config rolled back" with the version and the reason. After the cause is fixed, click "Re-push" to deliver the same configuration again.

Node program upgrades: staged and reversible

Upgrading the node program in System Admin › System Upgrade is also staged: first one node, then 10%, then all; every stage must fully succeed before the next. Each upgrade creates a rollout record, and an in-progress record can be paused ("Pause", "Resume") or reversed with "Roll Back": this stops the rollout, and nodes already upgraded return to their pre-upgrade version.

The upgrade replaces the whole package and switches atomically. If the new config fails its check or nginx is not ready, the node switches back to the old version and keeps serving. To go back, use "Roll Back" on the rollout record for the node program, or "Roll back to previous" in the configuration release panel for the configuration.

Upgrading the master

Upgrade Master and Roll back master are also under System Admin › System Upgrade. Tick "Auto-upgrade nodes when the Master is upgraded" to have nodes follow, going through the same stages. While a master upgrade is still queued or installing, "Stop Upgrade" cancels it; a switch that has already started will finish. See Online upgrades for the steps.

Tip When a node rolls back automatically, read the version and reason in the alert first. Don't re-push immediately; do it once you've confirmed the problem is fixed in the configuration.