# Database operations Operate, scale, resize, maintain, and delete Operator-managed QuestDB clusters. # Database operations Examples use `` and ``. Status is trustworthy only after the controller has observed the current spec. ## Check database health Check generation freshness first, then conditions. `status.phase` is only a human-readable summary. ```sh kubectl get questdbcluster -n \ -o jsonpath='generation={.metadata.generation}{" observedGeneration="}{.status.observedGeneration}{" phase="}{.status.phase}{"\n"}' kubectl get questdbcluster -n \ -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" reason="}{.reason}{" observed="}{.observedGeneration}{" message="}{.message}{"\n"}{end}' ``` Do not act on old conditions while `.metadata.generation` differs from `.status.observedGeneration`. For an ordinary writable cluster, writer readiness requires all four signals: current generation, `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. `Available=True` alone can mean only that a replica or a read-serving primary is available. `WriteHealthy=False/PrimarySuspended` names the impaired WAL table or tables in its message; reads and writes to other tables can remain available. `WriteHealthy=True` reports the latest `wal_tables()` observation. It does not perform a synthetic write and does not guarantee free disk capacity, so retain application write probes and storage monitoring where those guarantees are required. An intentional replica-only follower is the exception: no primary is correct, so `WriteHealthy` is omitted rather than reported healthy. Require current generation, `phase=Following`, `status.replication.following=true`, the expected `readyInstances`, and an appropriate `ReplicationHealthy` follower result. `True/FollowingExternalSource` reports observed progress; a quiet source may report `Unknown/StreamNotDetermined`, which is acceptable only after confirming the source identity and roots. Treat `ReplicationHealthy=False` as unhealthy. ```sh kubectl get questdbcluster -n -o wide kubectl get pods -n -l questdb.io/cluster= \ -L questdb.io/instance,questdb.io/role -o wide kubectl get pvc -n -l questdb.io/cluster= \ -L questdb.io/role,questdb.io/bootstrap -o wide ``` ## Services and ports | Service | Purpose | | --- | --- | | `-rw` | Current primary. Use for writes and administration. | | `-ro` | Created for store-backed replication. Routes to qualified replicas, but falls back to the primary when no replica qualifies. It is **not** strict replica isolation. | | `` | Headless, internal identity/DNS service. It publishes unready addresses and must not be used as a client availability endpoint. | | Port | Protocol | Published | | --- | --- | --- | | 9000 | HTTP and Web Console, including [QWP](/docs/configuration/qwp/) over WebSocket | Always | | 8812 | PostgreSQL wire protocol | Always | | 9009 | [InfluxDB Line Protocol](/docs/connect/compatibility/ilp/overview/) over TCP | Always | | 9003 | Minimal HTTP server for health and database metrics | Always | | 9007/UDP | [QWP UDP receiver](/docs/configuration/qwp/#qwpudpbindto) | Only when enabled | QWP over WebSocket has no port of its own. Ingestion (`/write/v4`) and streaming query results (`/read/v1`) are served by the HTTP server on 9000 and share its network settings, so they are available wherever 9000 is. The QWP UDP receiver is off by default. Enable it with [`spec.protocols.qwp.udp.enabled`](/docs/enterprise-kubernetes-operator/configuration/#wire-protocols); until then, port 9007 is neither opened on the pod nor published on any Service. It is the only port here that is not published on `-ro`. The operator owns these Services and reconciles them as `ClusterIP` (the identity Service is headless). For temporary access, use port-forwarding. For durable external access, create a separate customer-managed Ingress, Gateway, or LoadBalancer that targets the operator Service. Do not mutate the operator-owned Service type. ## Connect Find the admin Secret without printing its value: ```sh ADMIN_SECRET="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.adminSecretName}')" kubectl get secret "$ADMIN_SECRET" -n \ -o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp' ``` When an administrator explicitly needs the credential, load it into the environment without echoing it and clear it after use: ```sh ADMIN_PASSWORD="$(kubectl get secret "$ADMIN_SECRET" -n \ -o jsonpath='{.data.password}' | base64 -d)" kubectl port-forward -n service/-rw 8812:8812 \ >/tmp/questdb-pgwire-port-forward.log 2>&1 & PF_PID=$! trap 'kill "$PF_PID" 2>/dev/null; unset ADMIN_PASSWORD' EXIT sleep 2 PGPASSWORD="$ADMIN_PASSWORD" psql -h 127.0.0.1 -p 8812 -U admin -d qdb kill "$PF_PID" unset ADMIN_PASSWORD trap - EXIT ``` Use `service/-ro` only for read-only traffic that can tolerate primary fallback. To access the Web Console temporarily, forward `service/-rw` from local port 9000 and stop the process after the session. ## Make mutable changes safely Changes to `spec.image`, `spec.imagePullSecrets`, `spec.resources`, `spec.config`, replication tuning, or scheduling recreate affected pods. A rolling change remains single-writer-safe, but recreating the primary can briefly interrupt writes. ### Before you start 1. For an ordinary writable cluster, confirm current generation, `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. `Available=True` alone is not writer readiness. 2. For an intentional follower, instead confirm current generation, `phase=Following`, `status.replication.following=true`, the expected ready count, and an appropriate non-failing follower `ReplicationHealthy` result as described above. 3. Save the current spec. 4. Make one logical change at a time. ```sh kubectl get questdbcluster -n -o yaml \ > /secure/path/-before.yaml ``` ### Change and watch Apply a reviewed manifest or a narrow patch. This example changes an engine setting: ```sh kubectl patch questdbcluster -n --type merge \ -p '{"spec":{"config":{"cairo.max.uncommitted.rows":"500000"}}}' GEN="$(kubectl get questdbcluster -n \ -o jsonpath='{.metadata.generation}')" for _ in $(seq 1 60); do OBS="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.observedGeneration}')" [ "$OBS" = "$GEN" ] && break sleep 5 done [ "$OBS" = "$GEN" ] kubectl wait questdbcluster/ -n \ --for=condition=Available --timeout=5m ``` The `kubectl wait` above is only an availability gate. Before declaring the writer ready, re-read the current-generation conditions and require `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. Then verify pod UIDs/restarts, connectivity, and the intended setting. If the change fails, inspect `ConfigRejected`, events, and pod logs. Recover by reverting the mutable spec to the saved value and repeat the bounded generation and full writer-health checks. Do not try to revert immutable fields; create a new cluster when required by the [API Reference](/docs/enterprise-kubernetes-operator/reference/api/). ## Scale out A replica is born from a completed backup and then consumes object-store WAL. Before increasing `spec.instances`, the cluster needs: - `spec.objectStoreRef`; - enabled scheduled backup; and - a completed seed in `.status.replication.seed`. An explicit `spec.replication` block is optional; add it only for tuning. ```sh BACKUP_STATUS="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.backup.lastBackup.status}')" SEED="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.replication.seed.backupInstanceName}')" printf 'backup=%s seed=%s\n' "$BACKUP_STATUS" "$SEED" [ "$BACKUP_STATUS" = "completed" ] && [ -n "$SEED" ] kubectl patch questdbcluster -n --type merge \ -p '{"spec":{"instances":3}}' ``` Watch with bounded checks: ```sh for _ in $(seq 1 90); do READY="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.readyInstances}')" [ "$READY" = "3" ] && break sleep 10 done [ "$READY" = "3" ] kubectl get questdbcluster -n \ -o jsonpath='{range .status.conditions[?(@.type=="ReplicationHealthy")]}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}{range .status.replication.replicas[*]}{.instance}{" caughtUp="}{.caughtUp}{" caughtUpNow="}{.caughtUpNow}{" lagTxns="}{.lagTxns}{"\n"}{end}' ``` Verify `readyInstances` equals the requested count, `ReplicationHealthy=True`, and each current replica is healthy. `caughtUp` means it has caught up at least once; it is a latch. `caughtUpNow` is the live freshness reading. An absent `caughtUpNow` means freshness was not determined and must fail closed. ## Scale in Lowering `spec.instances` removes replicas. Instance serials are monotonic and are never reused, so names may not remain contiguous after scale or promotion. | `spec.pvcRetentionPolicy` | Removed replica PVC | | --- | --- | | `Retain` (default) | Kept for deliberate inspection or cleanup. | | `Delete` | Deleted during replica scale-in. A later replica gets a new serial and seeds again. | The policy applies only to replicas removed by scale-in. It never deletes the current primary PVC and does not govern whole-cluster deletion. On every managed PVC, `questdb.io/role` is the instance's **current** role and changes across promotion; `questdb.io/bootstrap` records how the volume was born and remains ancestry/state. Do not infer the current primary from `bootstrap`. Before scaling in, identify the primary and replicas, choose the retention policy, and inventory PVCs: ```sh kubectl get questdbcluster -n -o wide kubectl get pvc -n -l questdb.io/cluster= kubectl patch questdbcluster -n --type merge \ -p '{"spec":{"instances":1,"pvcRetentionPolicy":"Retain"}}' GENERATION="$(kubectl get questdbcluster -n \ -o jsonpath='{.metadata.generation}')" OBSERVED="" AVAILABLE_OBSERVED="" for _ in $(seq 1 120); do OBSERVED="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.observedGeneration}')" AVAILABLE_OBSERVED="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.conditions[?(@.type=="Available")].observedGeneration}')" [ "$OBSERVED" = "$GENERATION" ] && \ [ "$AVAILABLE_OBSERVED" = "$GENERATION" ] && break sleep 5 done if [ "$OBSERVED" = "$GENERATION" ] && \ [ "$AVAILABLE_OBSERVED" = "$GENERATION" ]; then kubectl wait questdbcluster/ -n \ --for=condition=Available=True --timeout=10m && kubectl get questdbcluster -n -o wide && kubectl get pods,pvc -n -l questdb.io/cluster= -o wide else printf 'Timed out waiting for generation %s (status %s, Available %s)\n' \ "$GENERATION" "$OBSERVED" "$AVAILABLE_OBSERVED" >&2 false fi ``` Verify the managed and ready counts, current primary, the full ordinary writer-health or separate follower contract, and the expected retained or deleted replica PVCs. Confirm each current primary/replica PVC's `questdb.io/role` label matches that role. ## Grow storage Storage is expand-only, and `spec.storage.storageClassName` is immutable. ### Before you start ```sh STORAGE_CLASS="$(kubectl get questdbcluster -n \ -o jsonpath='{.spec.storage.storageClassName}')" kubectl get storageclass "$STORAGE_CLASS" \ -o jsonpath='allowVolumeExpansion={.allowVolumeExpansion}{"\n"}' kubectl get pvc -n -l questdb.io/cluster= ``` Proceed only when `allowVolumeExpansion=true` and the CSI driver supports the requested expansion. ### Change and verify ```sh kubectl patch questdbcluster -n --type merge \ -p '{"spec":{"storage":{"size":"200Gi"}}}' GENERATION="$(kubectl get questdbcluster -n \ -o jsonpath='{.metadata.generation}')" OBSERVED="" AVAILABLE_OBSERVED="" for _ in $(seq 1 120); do OBSERVED="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.observedGeneration}')" AVAILABLE_OBSERVED="$(kubectl get questdbcluster -n \ -o jsonpath='{.status.conditions[?(@.type=="Available")].observedGeneration}')" [ "$OBSERVED" = "$GENERATION" ] && \ [ "$AVAILABLE_OBSERVED" = "$GENERATION" ] && break sleep 5 done if [ "$OBSERVED" = "$GENERATION" ] && \ [ "$AVAILABLE_OBSERVED" = "$GENERATION" ]; then kubectl wait questdbcluster/ -n \ --for=condition=Available=True --timeout=10m && kubectl get questdbcluster -n -o wide && kubectl get pvc -n -l questdb.io/cluster= \ -o custom-columns='NAME:.metadata.name,REQUESTED:.spec.resources.requests.storage,CAPACITY:.status.capacity.storage,PHASE:.status.phase' else printf 'Timed out waiting for generation %s (status %s, Available %s)\n' \ "$GENERATION" "$OBSERVED" "$AVAILABLE_OBSERVED" >&2 false fi ``` The availability wait is not proof of writer health. Require the current-generation `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy` conditions before closing the change. If `StorageResizeBlocked=True`, read its reason/message and fix the StorageClass or CSI limitation. A size reduction is rejected. To change StorageClass, restore into a new cluster. ## Node maintenance and disruption budgets The default PodDisruptionBudget (PDB) uses `minAvailable: 1` for a singleton and `instances-1` for a replicated cluster. A singleton therefore blocks voluntary eviction and can make `kubectl drain` wait indefinitely. Before planned maintenance, choose one safe option: 1. scale out and wait for a healthy, current replica; 2. lower `spec.scheduling.podDisruptionBudget.minAvailable`; or 3. set `spec.scheduling.podDisruptionBudget.enabled: false` only after accepting database downtime. ```yaml spec: scheduling: podDisruptionBudget: enabled: true minAvailable: 0 ``` Restore the normal PDB after maintenance and verify the current-generation writer contract: `Available=True/PrimaryReady`, `Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. A PDB protects only voluntary disruption. On node loss, the operator reports `InstanceUnreachable` and does not automatically promote a replica. It avoids recreating a pod while the old pod may still run on an unreachable node. Restore the node/volume or follow the explicit [promotion and failover](/docs/enterprise-kubernetes-operator/high-availability/#promotion-and-failover) procedure. ## Delete a database cluster Deleting a `QuestDBCluster` removes operator-owned pods, Services, ConfigMaps, and related resources. Its data PVCs are deliberately unowned and remain, and object-store backup/WAL data also remains. `pvcRetentionPolicy` does not change this whole-cluster behavior. ### Before you start Inventory and export the CR, PVCs, PV reclaim policies, store reference, and effective object prefixes: ```sh kubectl get questdbcluster -n -o yaml \ > /secure/path/-cluster.yaml kubectl get pvc -n -l questdb.io/cluster= -o wide kubectl get questdbobjectstore -n ``` :::danger Deleting the custom resource stops the database and removes its managed runtime objects. Confirm applications are stopped and that the CR export and data-retention inventory are complete. ::: ```sh kubectl delete questdbcluster -n --timeout=5m kubectl get pvc -n -l questdb.io/cluster= ``` Decide separately whether to retain or delete each PVC/PV according to its StorageClass reclaim policy. Decide separately whether to retain or delete cloud objects using customer-owned cloud tooling and policy. The operator never cleans the object store.