{"id":66575,"date":"2025-08-05T09:00:35","date_gmt":"2025-08-05T07:00:35","guid":{"rendered":"https:\/\/nolabnoparty.com\/?p=66575"},"modified":"2025-08-04T10:53:37","modified_gmt":"2025-08-04T08:53:37","slug":"shutdown-proxmox-cluster-with-ceph-storage","status":"publish","type":"post","link":"https:\/\/nolabnoparty.com\/en\/shutdown-proxmox-cluster-with-ceph-storage\/","title":{"rendered":"Shutdown Proxmox cluster with Ceph storage"},"content":{"rendered":"<p><img decoding=\"async\" class=\"aligncenter wp-image-66679 size-full\" title=\"shutdown-proxmox-cluster-with-ceph-01\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-01.jpg\" alt=\"shutdown-proxmox-cluster-with-ceph-01\" width=\"602\" height=\"202\" \/><\/p>\n<p>To shutdown <a href=\"https:\/\/nolabnoparty.com\/en\/create-a-proxmox-cluster\/\">Proxmox cluster<\/a> and prevent data loss or corruption, especially when Ceph storage is in use, you must follow a specific procedure.<\/p>\n<p>When working with a high-availability infrastructure, performing maintenance tasks can be a delicate process. Shutting down a cluster, even for a planned event like a power outage or hardware upgrade, requires <strong>careful execution<\/strong> to prevent data corruption and unexpected downtime.<!--more--><\/p>\n<p>This guide will walk you through the correct procedure for gracefully shutting down a <a href=\"https:\/\/nolabnoparty.com\/en\/extend-root-partition-in-proxmox-with-lvm\/\">Proxmox<\/a> cluster with a <a href=\"https:\/\/nolabnoparty.com\/en\/proxmox-ve-configure-a-ceph-storage-cluster\/\">Ceph storage<\/a> backend, ensuring that <strong>all services and data remain intact<\/strong>.<\/p>\n<p>&nbsp;<\/p>\n<h2>Shutdown Proxmox cluster with Ceph storage<\/h2>\n<p>Before proceeding with the actual shutdown, ensure the Ceph cluster is in a <strong>healthy state<\/strong>.<\/p>\n<p><span style=\"color: #0000ff;\"># ceph -s<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-66681\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-02.jpg\" alt=\"\" width=\"600\" height=\"301\" title=\"\"><\/p>\n<p>To prevent Ceph storage from automatically marking an OSD (Object Storage Daemon) as out when it goes offline, the <em>noout<\/em> flag must be set to stop the cluster from <strong>triggering a data rebalancing process<\/strong>.<\/p>\n<p>Without the <em>noout<\/em> flag, Ceph's default behavior is to <strong>mark an unresponsive OSD as out<\/strong> after a short timeout (typically 5 minutes), and then begin to re-replicate the data from that OSD to others in the cluster. This can be a <strong>waste of resources<\/strong> and cause unnecessary <strong>performance degradation<\/strong> if the OSD is expected to come back online shortly.<\/p>\n<p>Run the command:<\/p>\n<p><span style=\"color: #0000ff;\"># ceph osd set noout<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-66683\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-03.jpg\" alt=\"\" width=\"600\" height=\"86\" title=\"\"><\/p>\n<p>Shutdown all VMs and Containers.<\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-large wp-image-66685\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-04-600x290.jpg\" alt=\"\" width=\"600\" height=\"290\" title=\"\"><\/p>\n<p>If your infrastructure uses the <a href=\"https:\/\/nolabnoparty.com\/en\/proxmox-configure-high-availability-ha\/\">HA capability<\/a>, you should disable the HA manager during a <b>planned shutdown or maintenance<\/b> of a Proxmox node. It prevents the HA manager from performing <strong>unnecessary and disruptive actions<\/strong>, such as migrating services, when you are intentionally taking a node offline.<\/p>\n<p>Check the status of the HA manager.<\/p>\n<p><span style=\"color: #0000ff;\"># ha-manager status<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-66687\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-05.jpg\" alt=\"\" width=\"600\" height=\"154\" title=\"\"><\/p>\n<p>On <strong>each node<\/strong> stop the <em>pve-ha-lrm<\/em> service to prevent each node from trying to take action on its local services. <span class=\"citation-12 citation-end-12\">This service manages the local VMs and containers and execute the commands it receives from the CRM.<\/span> If you only stop it on one node, the <strong>other nodes will still be running their LRM services<\/strong> and the CRM will be making decisions.<\/p>\n<p><span style=\"color: #0000ff;\"># systemctl stop pve-ha-lrm<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-66689\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-06.jpg\" alt=\"\" width=\"600\" height=\"50\" title=\"\"><\/p>\n<p>Once the LRMs are stopped on all nodes, you can <strong>stop the CRMs on each node<\/strong>. <span class=\"citation-11 citation-end-11\">This service also runs on every node, but only <strong>one node at a time is the master CRM<\/strong>.<\/span> <span class=\"citation-10 citation-end-10\">This master is the decision-maker for the entire cluster.<\/span><\/p>\n<p>If you stop the master CRM but the LRMs are still running, a <strong>new master will be elected<\/strong>\u00a0and it will continue to manage the highly available resources, potentially overriding any manual changes you've made.<\/p>\n<p><span style=\"color: #0000ff;\"># systemctl stop pve-ha-crm<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-66691\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-07.jpg\" alt=\"\" width=\"600\" height=\"54\" title=\"\"><\/p>\n<p>Stopping both services on all nodes effectively <strong>freezes the high-availability stack<\/strong> for the entire cluster.<\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-large wp-image-66693\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-08-600x310.jpg\" alt=\"\" width=\"600\" height=\"310\" title=\"\"><\/p>\n<p>Now all <strong>Ceph services<\/strong> must be stopped.\u00a0Run the following command on each node:<\/p>\n<p><span style=\"color: #0000ff;\"># systemctl stop ceph-osd.target ceph-mon.target ceph-mgr.target<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-large wp-image-66695\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-09-600x46.jpg\" alt=\"\" width=\"600\" height=\"46\" title=\"\"><\/p>\n<p>Run the <em>sync<\/em> and <em>shutdown<\/em> commands on <strong>all nodes<\/strong>. The <em>sync<\/em> command ensures <strong>pending disk writes are flushed<\/strong>. Wait for each node to <strong>fully shutdown<\/strong> before moving to the next.<\/p>\n<p><span style=\"color: #0000ff;\"># sync &amp;&amp; shutdown now<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-66697\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-10.jpg\" alt=\"\" width=\"600\" height=\"73\" title=\"\"><\/p>\n<p>The three nodes have been shutdown.<\/p>\n<p>&nbsp;<\/p>\n<h2>Power on the cluster<\/h2>\n<p>After shutdown Proxmox, there is no <strong>specific sequence to follow<\/strong> when powering on the physical nodes to restore cluster functionality.<\/p>\n<p>After the nodes have powered on, wait for them to <strong>form a cluster quorum<\/strong>.<\/p>\n<p><span style=\"color: #0000ff;\"># pvecm status<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-66699\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-11.jpg\" alt=\"\" width=\"600\" height=\"559\" title=\"\"><\/p>\n<p>Check the <strong>Ceph storage cluster health<\/strong>. Note the <em>noout<\/em> flag is <strong>reported as set<\/strong>.<\/p>\n<p><span style=\"color: #0000ff;\"># ceph -s<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-66701\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-12.jpg\" alt=\"\" width=\"600\" height=\"342\" title=\"\"><\/p>\n<p>Since the <em>noout<\/em> flag was set during the shutdown procedure, you must <strong>unset the noout flag<\/strong>.<\/p>\n<p><span style=\"color: #0000ff;\"># ceph osd unset noout<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-66703\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-13.jpg\" alt=\"\" width=\"600\" height=\"71\" title=\"\"><\/p>\n<p>The HA status check reports that it is <strong>not active<\/strong>.<\/p>\n<p><span style=\"color: #0000ff;\"># ha-manager status<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-66705\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-14.jpg\" alt=\"\" width=\"600\" height=\"170\" title=\"\"><\/p>\n<p>When the Proxmox cluster is powered on, the HA service must be reactivated in the <strong>reverse order<\/strong>.<\/p>\n<p>On <strong>each node<\/strong> run this command to enable the <strong>CRM service<\/strong>.<\/p>\n<p><span style=\"color: #0000ff;\"># systemctl start pve-ha-crm<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-66707\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-15.jpg\" alt=\"\" width=\"600\" height=\"50\" title=\"\"><\/p>\n<p>Then run this command to enable the <strong>LRM service<\/strong>.<\/p>\n<p><span style=\"color: #0000ff;\"># systemctl start pve-ha-lrm<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-66709\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-16.jpg\" alt=\"\" width=\"600\" height=\"50\" title=\"\"><\/p>\n<p>The Proxmox cluster is now up and running.<\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-large wp-image-66711\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/2025\/08\/shutdown-proxmox-cluster-with-ceph-17-600x343.jpg\" alt=\"\" width=\"600\" height=\"343\" title=\"\"><\/p>\n<p>Following this procedure, you can <strong>safely shutdown<\/strong> <a href=\"https:\/\/www.proxmox.com\/en\/\" target=\"_blank\" rel=\"noopener\">Proxmox<\/a> with Ceph without risk of data loss or corruption.<\/p>\n<p><img decoding=\"async\" title=\"signature\" src=\"https:\/\/nolabnoparty.com\/wp-content\/uploads\/images\/firma.jpg\" alt=\"signature\" \/><\/p>\n","protected":false},"excerpt":{"rendered":"<p>To shutdown Proxmox cluster and prevent data loss or corruption, especially when Ceph storage is in use, you must follow a specific procedure. When working with a high-availability infrastructure, performing maintenance tasks can be a delicate process. Shutting down a cluster, even for a planned event like a power outage or hardware upgrade, requires careful execution to prevent data corruption and unexpected downtime.<\/p>\n","protected":false},"author":3,"featured_media":66679,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rop_custom_images_group":[],"rop_custom_messages_group":[],"rop_publish_now":"no","rop_publish_now_accounts":{"linkedin_93tdZWzMZc_93tdZWzMZc":"","facebook_2879994398731222_17841400390232720":"","twitter_113568041_113568041":"","mastodon_115463926174894442_115463926174894442":""},"rop_publish_now_history":[{"account":"facebook_2879994398731222_234218296615646","service":"facebook","timestamp":1756882891,"status":"success"},{"account":"twitter_113568041_113568041","service":"twitter","timestamp":1756882948,"status":"success"},{"account":"linkedin_93tdZWzMZc_93tdZWzMZc","service":"linkedin","timestamp":1756882983,"status":"success"}],"rop_publish_now_status":"done","footnotes":""},"categories":[2798],"tags":[2890,2063,2795,2298],"class_list":["post-66575","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-proxmox-en","tag-ceph","tag-cluster-en","tag-proxmox","tag-shutdown-en","has_thumb"],"_links":{"self":[{"href":"https:\/\/nolabnoparty.com\/en\/wp-json\/wp\/v2\/posts\/66575","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nolabnoparty.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nolabnoparty.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nolabnoparty.com\/en\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/nolabnoparty.com\/en\/wp-json\/wp\/v2\/comments?post=66575"}],"version-history":[{"count":0,"href":"https:\/\/nolabnoparty.com\/en\/wp-json\/wp\/v2\/posts\/66575\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nolabnoparty.com\/en\/wp-json\/wp\/v2\/media\/66679"}],"wp:attachment":[{"href":"https:\/\/nolabnoparty.com\/en\/wp-json\/wp\/v2\/media?parent=66575"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nolabnoparty.com\/en\/wp-json\/wp\/v2\/categories?post=66575"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nolabnoparty.com\/en\/wp-json\/wp\/v2\/tags?post=66575"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}