Interactive engineering question repository focusing on internal mechanics, crisis reflexes, and diagnostic methodology.
When 1 node drops, quorum is maintained (2/3 majority), so the cluster continues operating normally. When 2 nodes drop (1/3), quorum is lost; pmxcfs switches to read-only mode, and all VM starts, configuration changes, and cluster operations freeze.
It provides an external 1-vote tie-breaker for 2-node or even-numbered clusters to prevent split-brain ties. It provides only a quorum vote—it does NOT provide storage or compute resources.
Corosync heartbeat packets are extremely sensitive to millisecond latency spikes and jitter. If shared with VM, Storage, or Migration traffic, buffer bloat will cause corosync timeouts and freeze the cluster.
Standard ICMP ping is insufficient; you must inspect UDP ring latency, jitter, and Corosync retransmission logs.
Even if SSH and ICMP work, the `pve-cluster` or `corosync` service may be crashed, the isolated cluster ring network severed, or pmxcfs locked.
It is a FUSE-based cluster filesystem mounted at `/etc/pve`, backed by SQLite and replicated in real-time across all nodes via Corosync.
Quorum majority was lost due to network partition or node failures.
Power off and physically disconnect the node to be removed first; then execute `pvecm delnode <node>` from one of the remaining active nodes.
It is a catastrophic condition where network isolation leads both partitioned sides to believe they are the authoritative cluster, writing to the same shared VM disk simultaneously. PVE prevents this via Quorum Majority voting and Watchdog Fencing.
Live Migration is zero-downtime memory transfer during planned maintenance. HA is automated crash recovery (reboot on another node) after unexpected hardware failure.
Because when physical hardware crashes or loses power, volatile RAM is instantly erased; there is no live memory state left to transfer.
To ensure a non-responsive or failed host is guaranteed dead and can no longer write to shared storage before another node starts its VMs.
Rules that constrain VMs to run only on designated nodes with specific failover priority orders.
1) Did the surviving cluster have quorum? 2) Did fencing complete? 3) Does the VM reference local storage (local ISO/disk) or PCI hardware missing on the target? 4) Inspect CRM logs.
Because virtual disks already reside on shared storage, only volatile RAM and CPU register state are streamed over the network (downtime < 500ms).
In addition to streaming RAM state, all virtual disk blocks are mirrored in real-time to the target host using QEMU NBD (Network Block Device) mirror protocol.
If the VM modifies RAM faster than network throughput can transfer dirty pages, live migration cannot converge, leading to an infinite migration loop.
The target downtime threshold required to pause the VM and transfer the final batch of dirty RAM pages cannot be satisfied due to heavy memory write load.
If the target host processor is older or lacks specific instruction flags present on the source, live migration will instantly trigger a guest kernel panic.
Select the highest common instruction set supported by the oldest CPU in the cluster (e.g., `x86-64-v2-AES` or `x86-64-v3`).
Physical hardware registers and DMA memory mappings are tied to a specific physical card and cannot be serialized and streamed across the network.
Due to missing VLAN/bridge interfaces, insufficient free host RAM, missing storage pools, or incompatible CPU instruction sets on the target host.
Allocating more total vCPUs than available physical CPU threads. It becomes dangerous when multiple VMs simultaneously spike to 100% load, causing extreme scheduling latency and unresponsive guests.
The percentage of time a virtual machine wants to execute instructions on CPU but is forced to wait by the hypervisor scheduler because physical cores are busy serving other VMs.
On multi-socket servers hosting large-memory database VMs, NUMA is critical to avoid cross-socket UPI memory latency penalties.
Generally no. Oversizing vCPUs forces the Linux hypervisor scheduler to pay severe co-scheduling penalties, reducing overall VM performance.
It maximizes L3 CPU cache hits and eliminates context switching for real-time/database workloads. However, on overcommitted hosts, it eliminates scheduler flexibility and creates severe queuing delays.
The hypervisor inflates a balloon driver inside the guest to reclaim unused RAM for the host. If the guest lacks the VirtIO balloon driver, the setting is completely inoperative.
On large-memory VMs (32GB+) to reduce CPU Translation Lookaside Buffer (TLB) cache misses and accelerate memory access speeds.
The application is not CPU-bound; it is blocked waiting on disk storage I/O or memory locks (I/O Wait / `%wa` or CPU steal `%st`).
By distinguishing between Linux filesystem buffer/cache (`buff/cache`) and actual process anonymous memory (RSS / `Active(anon)`).
Ceph: 5+ node clusters requiring shared resilient storage for seamless HA. Local ZFS: Maximum single-node NVMe IOPS with PBS backup replication. LVM-Thin: Lightweight local block storage with thin-provisioning.
With 3x replication, if 1 node drops, the remaining 2 nodes cannot find a 3rd distinct host to satisfy CRUSH failure domain rules, leaving the cluster permanently degraded.
If MON quorum is lost, clients cannot retrieve the active cluster map, causing all storage I/O operations across the entire datacenter to freeze immediately.
Down: The OSD daemon is stopped/unreachable, but its data is still expected to return. Out: Ceph declares the disk dead and triggers active data rebalancing to other OSDs.
Execute `ceph health detail` to determine the exact root cause (e.g. degraded PGs, down OSDs, near-full capacity thresholds, or clock skews).
Investigate physical drive SMART read timeouts, OSD network interface packet drops, controller queue hangs, or OSD memory exhaustion.
A logical subdivision that groups millions of objects into manageable collections for mapping to OSDs, preventing individual object tracking overhead.
Degraded: Object replica count is below target but readable. Undersized: Fewer OSDs than pool size policy. Inactive: Minimum replica count cannot be satisfied, causing complete I/O freeze.
Because background data reconstruction consumes massive disk I/O and network bandwidth, forcing client VM requests into deep queues.
Because a write transaction is not acknowledged to the VM until it commits across 3 OSDs over the network; network latency directly dictates disk write latency.
The Adaptive Replacement Cache (read cache) which defaults to consuming up to 50% of total host RAM. If unconstrained, spawning new VMs can trigger kernel OOM kills.
Mirrors provide aggregate random IOPS multiplied by disk pairs. RAIDZ provides only the random IOPS of a single disk regardless of drive count due to parity calculation lock-step.
A ZFS snapshot is an internal block pointer on the same physical pool (not an external backup; if the disk dies, snapshots die). A PBS backup is an encrypted, deduplicated copy residing on an independent server.
All virtual machine write I/O immediately suspends; guest filesystems switch to read-only mode and data corruption can occur.
VM I/O processes hang in uninterruptible sleep (`D state`), causing applications to freeze until NFS connectivity is restored.
If Multipath I/O is not configured, SCSI commands timeout, the block device is dropped, and guest filesystems switch to read-only.
It functions as a software Layer-2 Ethernet switch in kernel space, connecting physical NICs with virtual machine `tap` interfaces.
It eliminates creating separate bridges for every VLAN, dynamically filtering and trunking up to 4096 802.1Q VLAN tags over a single `vmbr0`.
1) Guest IP/Gateway, 2) VM tap port VLAN tag, 3) Host `bridge-vlan-aware` configuration, 4) Physical switch port trunk VLAN allowed list.
Verify whether `bridge-vlan-aware` is enabled on the host bridge and check if `pve-firewall` or ebtables rules are dropping DHCP/ARP broadcast packets.
An Access port belongs to a single VLAN carrying untagged frames. A Trunk port multiplexes multiple VLANs carrying 802.1Q tagged frames.
Active-Backup requires no switch configuration and keeps one link idle for failover. LACP negotiates with the switch to provide both link redundancy and aggregated multi-flow bandwidth.
To prevent out-of-order packet delivery which destroys TCP throughput, a single TCP flow is pinned to a single physical link via packet header hashing.
An intermediary switch port, bridge, or interface in the path remains at MTU 1500, causing silent packet drops and massive TCP retransmission storms.
Send an 8972-byte ICMP payload with the Don't Fragment (DF) bit enabled (8972 payload + 28 bytes IP/ICMP header = 9000 bytes).
Inspect VM default gateway configuration, Linux bridge binding, and verify whether NAT masquerading or firewall forwarding rules are missing on the host.
1) Missing or invalid DNS configuration in `/etc/resolv.conf`, or 2) The upstream firewall/gateway lacks NAT forwarding rules for the VM's IP subnet.
A switch port configuration mismatch (missing VLAN trunk allowed list), physical cabling/transceiver failure on that node, or a local `interfaces` misconfiguration.
Classical vzdump writes full compressed disk images every time. PBS splits data into 4MB SHA-256 chunks, achieving global deduplication, client-side encryption, and instant incremental backups.
Data streams are sliced into 4MB chunks, hashed with SHA-256, and indexed. If a chunk hash already exists on disk, it is not rewritten; only its reference is appended to the backup manifest.
The atomic storage block in PBS; an encrypted and compressed slice of data averaging 4MB in size, addressed uniquely by its SHA-256 cryptographic digest.
Prune = Deleting the catalog index card of a book from the library. Garbage Collection (GC) = Walking the bookshelves and physically shredding orphaned pages that no catalog card references.
Because Prune only deletes metadata manifests; physical chunks are not freed until Garbage Collection (GC) runs and passes the 24-hour safety grace window.
Because datastores contain millions of small 4MB chunk files, forcing random disk I/O bottlenecks during metadata status checks (especially on mechanical HDDs).
Verification tests cryptographic chunk integrity on disk (SHA-256 match). A Restore Test proves that the operating system and database actually boot, mount, and function correctly in production.
During backup, the QEMU Guest Agent filesystem freeze (`fs-freeze`) failed, capturing the database in an inconsistent state, or the guest OS bootloader was corrupted.
For high-speed metadata lookups across millions of chunks and built-in protection against silent data corruption (bit-rot) via ZFS checksumming.
Deploy ZFS RAID10 (Striped Mirrors) for maximum random IOPS and fast rebuilds; for large capacity RAIDZ2 pools, attach an NVMe Special Metadata vdev.
For enterprise production and high VM densities, SSDs are mandatory. For multi-petabyte cold archives, enterprise HDDs backed by NVMe metadata caching can be utilized.
To prevent heavy backup traffic (especially initial baseline transfers) from saturating corosync heartbeats, storage networks, and live VM interfaces.
Backup: PVE clients push VM data to PBS. Sync Job: A secondary PBS server pulls missing deduplicated chunks directly from a primary PBS server over WAN.
Because site-wide physical disasters (fire, flood, electrical catastrophe, physical theft) or ransomware outbreaks destroy local secondary stores simultaneously.
The backups are mathematically irrecoverable; all data is permanently and irrevocably lost.
1) Active PVE host `/etc/pve/priv/storage/<id>.enc`, 2) Corporate Secrets Manager (HashiCorp Vault), 3) Physical printed offline Paperkey.
Execute a comprehensive `Verify Job` across the entire datastore to scan and report corrupted chunk hashes and affected snapshot manifests.
A specific chunk unique to that VM suffered bit-rot corruption, or a previous backup write was interrupted mid-transfer.
Connect PVE directly to the Remote PBS server over WAN using the master encryption key, and restore VMs with `--live-restore 1` for immediate boot.
1) Install a clean Proxmox VE host, 2) Configure base network bridges/VLANs, 3) Add PBS storage target and provide encryption keys, 4) Restore all VMs sequentially via `qmrestore --live-restore 1`.
VM backup captures virtual disks and VM `.conf` files. Host config backup captures `/etc/pve`, `/etc/network/interfaces`, and Corosync cluster definitions.
Add the PBS storage via web GUI or CLI (`pvesm add pbs ...`), navigate to Datacenter -> Storage -> Backups, select the VM snapshot, and click Restore.
Because encrypted PBS chunks cannot be decrypted without the secret key; even with 100% of the storage chunks present, data recovery is mathematically impossible.
RPO: Maximum acceptable data loss duration (e.g., last 15 minutes of transactions). RTO: Maximum acceptable downtime duration required to bring systems back online (e.g., 30 minutes).
Having backups means data copies exist. Being DR-ready means having an audited, rehearsed, documented, and timed playbook proven to restore operational business services.
Restore randomly sampled production VMs into an isolated sandbox VLAN/bridge on a dedicated staging host, and run automated health-check scripts without IP collision risks.
Because site-wide physical disasters (fire, structural collapse, flood, power surge, theft) destroy both backup systems simultaneously.
PVE Cluster is a tightly-coupled local cluster requiring shared Corosync quorum. PDM is a loosely-coupled multi-datacenter management layer that centrally monitors independent clusters without shared quorum dependencies.
Zero operational impact! All PVE clusters continue executing local VM workloads, HA failovers, and storage replication without interruption.
Via HTTPS/TLS using PVE REST APIs and scoped API Tokens with cryptographic authentication.
Passwords grant broad interactive access and are difficult to revoke cleanly. API Tokens have restricted privileges (least-privilege), expiration dates, and can be instantly revoked without changing user credentials.
Authentication: "Who are you?" (Identity verification via PAM, LDAP, OIDC, 2FA). Authorization: "What are you permitted to do?" (RBAC permissions to start, modify, or delete VMs).
Assign granular PDM RBAC roles (e.g. `PVE.Auditor` or `PVE.PowerUser`) scoped strictly to specific Cluster Path ACLs.
It aggregates global VM inventories, cross-datacenter storage utilization, and active alerts on a single unified pane of glass instead of requiring separate logins to dozens of disparate clusters.
To ensure that WAN outages, cloud partition events, or centralized PDM failures do not disrupt local cluster HA failover, storage I/O, or VM execution.
1) Verify Quorum via `pvecm status` (4/5 active). 2) Confirm Node 3 Watchdog Fencing completed via `ha-manager status` and check if failed VMs have local ISO/disk dependencies. 3) Inspect `ceph health detail` for inactive PGs. 4) Verify switch VLAN trunking on target nodes. 5) Safely boot remaining failed VMs.
1) Check VM RAM dirty rate against network bandwidth in QEMU task logs, 2) Verify MTU and interface packet drops on the 10 GbE link, 3) Enable QEMU `auto-converge` and temporarily increase downtime tolerance.
Temporarily pause or abort running backup jobs and immediately initiate Garbage Collection (GC). When datastore is 100% full, new chunk writes fail immediately, breaking all active backups until orphaned chunks are cleared.
You CANNOT trust it! Verification failure proves on-disk chunk corruption (bit-rot or storage controller error). Attempt test restore to an isolated sandbox and immediately trigger a Force Full Backup for the affected VM.
1) Install clean Proxmox VE on fresh hardware, 2) Configure identical switch VLAN bridges, 3) Add Remote PBS storage with encryption key, 4) Execute `qmrestore` with `--live-restore 1` to boot all mission-critical VMs within seconds.
Triage Sequence: 1) Quorum (2/3 majority valid?), 2) Node 2 Fencing (isolated by watchdog?), 3) Ceph I/O (any inactive PGs blocking disk writes?), 4) HA Failed VM (local ISO/storage blocking start?), 5) Network (VLAN trunk permissions on target switch port?), 6) PBS (trigger replacement full backup for corrupted snapshot).