Skip to content

Your App Works. Can You Restart It Safely?

4 min read3 views★ featured[Backend]
Cover image for "Your App Works. Can You Restart It Safely?"

Starting a server is usually one command. Stopping it safely is a protocol.

A deployment can arrive while one request is writing to the database, another is uploading a file, and a worker is halfway through an external API call. If the process exits immediately, the user-facing result may be an ambiguous timeout, a partial artifact, or work that never resumes.

The goal of graceful shutdown is to stop accepting responsibility faster than you can finish or safely hand it off.

Begin draining before disappearing

For a hypothetical web service, I would define the shutdown sequence explicitly:

Mark instance as draining
Stop claiming new background jobs
Stop admitting new application work
Allow accepted work a bounded completion period
Persist or release recoverable ownership where appropriate
Close dependent clients after their users finish
Exit

The ordering matters. Closing the database pool before handlers finish turns an orderly deployment into a collection of avoidable errors.

The exact behavior of listeners, proxies, and health checks depends on the runtime and deployment platform. Test the sequence through the real routing layer, not only by calling the shutdown function in isolation.

Know which signal reaches the process

Docker's stop operation sends the configured stop signal, normally SIGTERM, to the container's main process and escalates to SIGKILL after the grace period. Your application's cleanup budget must fit within that outer deadline. Docker container stop

A shell wrapper that does not forward signals can prevent your handler from running as intended. Inspect how the application is launched rather than assuming the runtime receives every signal.

In Node.js, installing a signal listener changes the default exit behavior for supported signals. Your handler must arrange completion and termination; merely logging “shutting down” is not enough. Node.js signal events

Stop admission without abandoning accepted work

Node's HTTP server close behavior is documented by runtime version; modern versions stop new connections and handle idle connections as described in the API. Long-lived connections and upgraded protocols need separate consideration. Node.js server.close

Rather than presenting a tiny snippet as a complete shutdown implementation, I would write the application's invariants first:

No new jobs are claimed after draining begins.
Every accepted HTTP write either completes or remains safely retryable.
No dependent resource is closed while accepted work still needs it.
The process exits before the supervisor's force-kill deadline.

Then connect those invariants to the framework's listener and lifecycle APIs. A WebSocket server, streaming endpoint, and short JSON API will need different draining behavior.

Readiness and liveness answer different questions

Readiness asks whether this instance should receive new traffic. Liveness asks whether the process needs recovery or restart. A draining process can be alive while intentionally not ready.

For the hypothetical deployment, I would first remove the instance from traffic, allow for routing propagation according to the platform, and reject new work that still reaches it with a documented transient response. Accepted requests get a bounded time to finish.

Do not use an arbitrary sleep as proof that every upstream router has noticed. Measure the actual behavior during a rolling deployment.

Also check that rejecting a request doesn't trigger unsafe retries in a client or proxy. Shutdown correctness and request idempotency are connected.

Workers need recoverable ownership

Suppose a worker claims an export job and holds ownership only in process memory. If the machine disappears, another worker may never learn that the job should run again.

A durable lease gives ownership an expiry. A replacement worker can reclaim abandoned work, while conditional updates prevent a stale worker from marking someone else's claim complete. The external effect still needs its own duplicate protection.

During graceful shutdown, stop claiming first. Let short jobs finish if they fit the budget. For longer jobs, preserve a safe checkpoint or allow the lease/retry protocol to recover. Releasing a lease while the old worker continues executing can cause concurrent processing, so ownership changes need to agree with actual execution.

A timeout is a budget, not a guarantee

If an upstream API can hang for sixty seconds but the deployment allows twenty seconds for shutdown, the system has incompatible deadlines.

Bound downstream calls. Reserve time for recording outcomes and closing resources. When the deadline expires, termination may still be necessary; the recovery design must account for that possibility.

Graceful shutdown improves ordinary deployments. It cannot replace crash recovery because machines can fail without sending a cooperative signal.

Test the restart under meaningful work

In a disposable environment, run requests and jobs while repeatedly replacing the process. Check completed operations by their durable identifiers, not just by HTTP success counts.

Include a job whose side effect succeeds just before the worker exits, an upload in progress, and a request that outlasts the graceful window. Confirm the next instance recovers expected work without duplicating the business effect.

A service that starts reliably is deployable. A service that drains, stops, and recovers predictably is much easier to operate.

# Comments (0)

No comments yet. Be the first.

0/1000