Laravel on k3s, Part 3: Day-Two Operations

7 min readUpdated on

Laravel on k3s — Part 1 · Part 2 · Part 3 · Part 4

Start with an operating signal, not a dashboard

An operator needs to answer three questions quickly: is a release healthy, is capacity being exhausted, and can data be recovered? A dashboard can help, but it is not the source of truth. Keep a small command runbook in the repository and make the output part of an incident note. The following checks distinguish a bad application release from a node or scheduler problem before anyone starts deleting Pods.

sh
kubectl -n storefront get deployment web
kubectl -n storefront rollout status deployment/web --timeout=60s
kubectl -n storefront get pods -l app=storefront-web -o wide
kubectl -n storefront get events --sort-by=.lastTimestamp | tail -n 30
kubectl -n storefront top pods
kubectl top nodes

top requires Metrics Server; if it is unavailable, say that observability is missing rather than guessing from CPU limits. A Pod in CrashLoopBackOff needs its previous container logs and its Events. A Pod in Pending is commonly an image-pull, missing-Secret, taint, or resource-request problem. Restarting it may remove the only useful evidence.

sh
pod="$(kubectl -n storefront get pod -l app=storefront-web -o jsonpath='{.items[0].metadata.name}')"
kubectl -n storefront describe pod "$pod"
kubectl -n storefront logs "$pod" --all-containers=true --tail=200
kubectl -n storefront logs "$pod" --previous --all-containers=true --tail=200

Do not put database passwords or whole request bodies into those logs. Laravel should emit a request or correlation ID, the route or operation name, an error class, and the release SHA. Pass a release identifier as a non-secret environment variable and add it to structured logs. That turns “customers see 500s” into a question that can be joined with the exact workload revision.

yaml
env:
  - name: RELEASE_SHA
    value: "3f18d7c"
  - name: LOG_CHANNEL
    value: stderr

For queue workers, inspect the queue's own depth and failure count rather than using web-pod readiness as a proxy. Horizon, Redis, or a managed queue each have different signals. A web Deployment may be healthy while orders wait for a stopped worker. Conversely, raising worker replicas without considering a downstream API can turn a slow dependency into a rate-limit incident.

Treat credentials as a rotation workflow

Kubernetes Secret objects are an API transport mechanism, not a complete secret management policy. Limit who can create, patch, and read them; enable k3s encryption at rest only after understanding its key handling; and prefer a controller that reads a managed secret store when that fits the organisation. The important property is that the application has a rehearsed way to change a credential without copying it into Git, a chat, or terminal history.

For a value injected through envFrom, updating the Secret does not update an already running process. Rotate in this order: create the new credential at the provider, update the protected secret source, trigger a controlled web and worker rollout, verify new connections, then revoke the old credential. The following command deliberately restarts the template; it does not reveal any Secret content.

sh
kubectl -n storefront rollout restart deployment/web
kubectl -n storefront rollout restart deployment/worker
kubectl -n storefront rollout status deployment/web --timeout=180s
kubectl -n storefront rollout status deployment/worker --timeout=180s

Do not rotate APP_KEY with this procedure. Laravel encrypted cookies and stored encrypted values may require a staged application migration and a specific key-rotation design. Treat a generic “restart after every Secret change” rule as a useful default, not a replacement for knowing each value's meaning.

Back up data, then prove restoration

For a Laravel application, the database and object uploads are usually more important than Kubernetes manifests. Manifests can be rebuilt from Git; a customer's order or uploaded document cannot. Choose a backup method supported by the database provider, encrypt it, store it outside the VPS failure domain, and retain enough history for operator error or delayed discovery of corruption.

This illustrative MySQL command creates a logical backup from a dedicated backup account. Run it from a protected maintenance environment, stream to encrypted off-node storage, and never place its password in a committed script:

sh
mysqldump --single-transaction --routines --events --hex-blob \
  --host="$DB_HOST" --user="$BACKUP_USER" --password \
  "$DB_DATABASE" | gzip > "storefront-$(date +%F-%H%M).sql.gz"

The backup is not verified because the command returned zero. On a schedule, restore one backup into an isolated database with a new name, run an integrity query, and record the time. Do not restore over production just to test it.

sh
gunzip -c storefront-2026-09-05-0200.sql.gz | \
  mysql --host="$RESTORE_DB_HOST" --user="$RESTORE_USER" --password storefront_restore
mysql --host="$RESTORE_DB_HOST" --user="$RESTORE_USER" --password \
  --database=storefront_restore --execute='SELECT COUNT(*) AS orders FROM orders;'

Record RPO (the maximum acceptable data loss, for example 24 hours) and RTO (the time allowed to restore service, for example four hours). A daily backup does not promise zero data loss, and a snapshot is not automatically portable to a new disk or provider. Test uploads separately: an object-storage lifecycle policy, bucket versioning, and application database references all affect the actual recovery result.

Upgrade the platform as a change, not a chore

A single-node k3s cluster has no invisible maintenance window. Check the target k3s release notes, backup critical data, confirm free disk and rollback options, then schedule a maintenance period. First confirm the node and system Pods are healthy; after the change repeat the public application check and inspect the server service logs. The exact upgrade command follows the current k3s release documentation, so do not paste an old internet one-liner into production.

Keep the node itself patched, restrict SSH, test firewall rules, and alert on disk pressure before the kubelet starts evicting Pods. On a single VPS, a database, image cache, container logs, and backups compete for the same disk. Resource limits prevent one container from consuming everything, but they do not create capacity or high availability.

Incidents, rollback, and the boundary of this design

For a broken web release, first preserve evidence, identify the image, and use the rollback proven in Part 1 only if the schema remains compatible:

sh
kubectl -n storefront rollout history deployment/web
kubectl -n storefront rollout undo deployment/web
kubectl -n storefront rollout status deployment/web --timeout=180s
curl --fail --show-error https://shop.example.com/up

If the problem is a node, disk, compromised credential, corrupt data, or a non-backward-compatible migration, a Deployment rollback is insufficient. Use the incident procedure, involve the data owner, and decide whether to restore, fail over to a managed service, or keep the application read-only. Do not make production recovery decisions from a generic blog command.

This operating model is a poor fit when no one owns alerts, patching, restores, and access review. A managed platform with managed database backups may be a better choice for a small team. k3s is useful when its explicit controls serve real workloads and the team has time to practice them—not when Kubernetes is used to conceal an unstaffed operations function.

Production starts after kubectl apply. Operators need logs correlated to request IDs, deployment status, pod restarts, queue depth and database backup restoration evidence. Use kubectl rollout status after deploy and investigate describe events before repeatedly restarting pods.

Kubernetes Secrets are base64-encoded, not encrypted by default. Restrict RBAC, use an external secret manager or encryption-at-rest where appropriate, and rotate credentials with an application-aware plan. Persistent volume snapshots are not enough until restoration is rehearsed. Document restore time and data-loss objectives rather than calling any backup “safe.”

Related articles

Existing system support

Need help with a live application?

I help companies improve live systems, clean up delivery workflows, and ship new features without adding avoidable complexity.

Comments (0)
Sign in to leave a comment

You need to be signed in to add a comment.

Login

Need someone to take responsibility for the next step?

Let’s talk about your project and define a scope that actually makes sense for your goals.