Root partition increase & Emergency mode issue.

Latest response

I have root partition size of 50G.

/dev/sda1 /boot/efi
/dev/sda2 /boot
/dev/sda3 /part
- rhel-root 50G
-rhel-swap 8G
-rhel -home 200G

created new partition of /dev/sda4 partition size of 100G using fdisk command.
After creation of /dev/sda4
dev/sda1 /boot/efi
/dev/sda2 /boot
/dev/sda3 /part
- rhel-root 50G
-rhel-swap 8G
-rhel -home 200G
/dev/sda4 /part

Here, I added /dev/sda4 partition space to rhel-root to make it to 150G.
I followed the below steps:
1.Use vgextend command
# vgextend rhel /dev/sda4
2.Grow the LV using lvextend.
# lvextend /dev/rhel/root /dev/sda4
3.To Grow the file system
# xfs_growfs /dev/rhel/root.

Now I want to delete the /dev/sda4 of 100G space which is added to rhel-root.
Using parted command deleted the partition 4 (/dev/sda4).
rm 4
It asked me to reboot the machine. After machine reboots, it runs into Emergency mode. I can’t able to do anything.

MY QUESTION is that we already have /dev/sda3 with root, swap, & home, why can’t machine boot from the /dev/sda3 which has all files to boot up. Here I am just deleting the extra space what I added from /dev/sda4 (100G) to /dev/sda3 right.

Responses

Hello Shiva K

Welcome to the Red Hat discussion forums. In the middle of the post, you mentioned you used xfs_growfs to grow the file system. I believe you are using RHEL 7.

Apologies in advance here for the lengthy reply I am giving here and an answer you were not anticipating. In short, if you really did delete /dev/sda4 after adding it to your "rhel" Volume Group, your likely best option is a system reload. I'd highly recommend for any new systems that you create sub-partitions partitions under your "/" file system somewhat explained below and in a possible kickstart in my next reply.

If you actually did add then delete /dev/sda4 to your "rhel" Volume Group, that would cause serious issues to your "rhel" Volume Group and explain the system's difficulty in booting. If this is the case I anticipate a reload of the system is in your future.

I'd recommend getting an idea of what subdirectory under "/" was filling up and coming up with a partitioning method that will allow you to assign subdirectory sizes to "/", such as separating "/var" from "/", and "/opt" from "/" (namely, making separate partitions for these and other file systems.

Here's an example. Those that use Docker (for example, and you might not be using docker) end up filling up "/var" or "/var/lib" because Docker containers can consume a lot of space. In this specific case in this specific paragraph, having a separate "/var" from "/" would protect "/" from filling up. However if someone ran a docker-heavy system, their "/var" could fill up if their "/var/lib" becomes unruly with docker images.

Another example. I've seen people put software under "/opt" with disregard of the impact of "/" (mabye not you, but I've seen it), so in a case such as that, I'd recommend to those who need space under "/opt" to make a large eough file system to take the necessary software that would go under "/opt".

Yet another example. I've also seen the "/" file system taken over when no separate "/var" exists, and either databases, tomcat, docker images or even just log files from heavy logging fill up "/" because no separate "/var" or "/var/log" or "/var/lib" exists. See this thread for a discussion on that https://access.redhat.com/discussions/641923

What is a good partitioning method? It will depend on your own case use. In principle, it's good to assign partitions to things under "/" such as "/var" or "/var/lib" "/var/log" or "/opt" to protect the "/" file system.

From your description, it sounds like you expanded the volume group "rhel" with /dev/sda4 using vgextend and lvextend. It sounds like after expanding the Volume Group named "rhel" you deleted the underlying 100G device of /dev/sda4? Is this correct?

When building systems, I'd recommended examining what on your "/" file system is consuming the majority of the space and making a separate file system for whatever under the "/" partition was consuming the most.

Example: Someone says their "/" file system is at 90% and there is some time to work on it. I'd recommend: - Finding the point source for "/" and temporarily mount it under a new directory called "/slash" just for the purpose of examining the largest things under it.

  • Mount it as shown below - Doing this will reveal what is the largest things under "/" file system ignoring other mount points.

  • Run a du command as shown below to see what

  • The results of that du command for "/slash/*" will reveal what is using the most under the "/" file system, then you can make a new fresh partition for whatever is filling up "/" (from your examiniation of "/slash" and rsync (copy) it to the new directory, then reassign the partition after the rsync with /etc/fstab and reboot.

  • After doing this, you can examine/fix SELinux contexts, and validate the data has been copied, and remount the "/slash" file system and remove what was UNDER it.

1) making note of the partition where "/" is mounted such as:

[root@server1] # df -PhT /
Filesystem               Type    Size   Used    Avail   Use%  Mounted on
/dev/mapper/rhel-root    xfs      40G  31.5G      35G    90%  /

[root@server1] # mkdir -p /slash/notmounted
[root@server1] # mount /dev/mapper/rhel-root /slash
[root@server1] # du -sk /slash/* | sort -nr

2) The intent of the above block is to mount the "/" to the new directory named "/slash" for testing purposes.

-- Namely to isolate it to determine what under "/" is consuming the most.

3) Make note of whatever was at the top of the du -sk /slash/* | sort nr results

-- (the sort function above will make the largest things appear a top)

-- Commonly heavy-hitter sub-directories often include one or some of "/slash/tmp" or "/slash/usr" was the largest thing, or perhaps "/slash/opt" or "/slash/opt"

4) If (for example) it was '/slash/opt' filling it up,

-- Then I'd make a separate file system for /slash/opt and mount it temporarily as "/opt2(and it does not have to be added to the "rhel" volume group)

-- Then do an rsync from /slash/opt to /opt2

  -- Whatever file system is the largest **UNDER** "/" - I'd recommend making a separate partition for that.

5) Let's say it was "/slash/opt" that was filling up your "/" file system from your "du" command above.

6) Make a separate file system (other people reading this might have to add another disk) and mount it under /opt2 TEMPORARILY.

Then in "rescue.target" mode, (not multi-user.target) perform an rsync of /slash/opt to the new temporily mounted /opt2 as root.

  • If you are using SELinux, ensure the SELinux contexts are correct

Change the /etc/fstab to make "/opt" to be the new partition you made and very-temporarily mounted under "/opt2"

  • Boot the system and then validate the contents undre "/opt2" are what was under "/shash/opt" and are the proper selinux context.

After validating all is well, remove the UNDERLYING /slash/opt/* contents (but not the directory

In short the procedure above is for those who discover their "/" file system is getting dangerously close to 100% consumption

  • The idea is to mount the mountpoint that goes to "/" temporarily to "/slash" in order to see what is actually filling up "/" without the disctraction of possible sub-mounts

  • Mount the "/" file system also under "/slash" temporarily as described above

  • Examine the consumption under of "/slash" using the du command mentioned previously

  • Make note of the directory/directories under "/slash" that are the largest

  • Consider creating a new fresh file system to accommodate what is found under "/slash" and decide on a new file system.

    -- It does not necessarily have to be added to the "rhel" Volume group, or whatever the Volumne group name is.

  • Make the file system decided upon from the last step, mount it temporarily (not using /etc/fstab) under /xyz2

    -- (where "xyz" is the name of the sub directory that is taking the space you've discovered from above

  • Go to rescue.target and if needed remount (temporarily) the "/slash" partition and the new partition you're using

    -- In the example I provided, we picked "/opt2" for the new file system. your actual partition maybe dirferent based on your results above

  • rsync the /slash/opt (in this example, evaluate it for your own) to /opt2 as root in rescue.target mode

  • Validate the data moved properly, ensure proper SELinux contexts

  • I'd recommend executing touch /.autorelabel just to make sure before the reboot.

  • Make the necessary entry in /etc/fstab for the TEMPORARY mount of (in our example here) of "/opt2" to become "/opt"

  • reboot, evaluate

  • IMPORTANT: validate your new file system '/opt' (in our example) is actually mounted as shown in /etc/fstab, else fix as needed.

  • If it works, then temporarily re-mount the "/" again under "/slash" and **after validating the data moved and if the system works and the data is in the new mountpoint you can evaluate when to remove "/slash/opt/*" (do not remove "/slash/opt" directory!!)

What will help in advance is an ounce of prevention when initially buidling the server. Make necessary sub-partitions under "/" to avert "/" from filling up, such as "/var/tmp" "/var" "/tmp" "/opt" "/var/log" "/var/lib" and others as required.

Again, sorry for the lengthy post. As I mentioned at the start of this, if you really did remove /dev/sda4 after adding it to your volume group named "rhel" - then that explains why your system is having huge issues with booting.

The idea with the large post above is to give an idea if what to do if you face this issue again, or to have an ounce of prevention. In the next post, I'll add a partitioning idea (evaluate for yourself, adjust as required) for a kickstart

Regards,

RJ

This would be an excerpt of a kickstart. You'll find a link to the Red Hat kickstart generator in the text below.

I'd recommend adding additional partitions in your system build - when you build it. Consider using a kickstart method.

The partitions below are an example. Your server needs will probably require adjustment to the example below. Evaluate and adjust as needed.

The overwhelming idea is to protect "/" from being consumed due to some sub-directory being highly consumed.

  • Example, /opt could consume "/" if "/opt" is not it's own filesystem/partition when people load up 3rd party software in /opt partition

  • Another example "/var/lib/docker" often is a heavy consumer of disk space when adding docker images. If there is no "/var" or "/var/lib" or "/var/lib/docker - the consumption will fill up the parent file system the directory is using.

  • A way to determine if something is a mountpoint, or what the parent file system is:

  • If there is no `/var' file system and you run this command, you'll get this output:

[root@server1] # mountpoint /var
/var is not a mountpoint

Now if you created a filesystem to separate "/var" from "/" then when you ran the command above, you'd get /var is a mountpoint instead as output

Can you go overboard on this? Certainly. Does it require adjustment perhaps from one server role to another? Absolutely. So why? In short, to protect the "/" file system from being filled up from a sub-directory that ought to be a file system but is not.

Bottom line, create sane (for your environment) sub-partitions to the "/" file system at build time to avoid having to react after a build by a need to expand the "/" because of disk consumption that was not thought of in advance.

#### is this excessive? probably.
#### But why not?, it aids to  protect "/" or other things from filling
####################################
zerombr
### note, make sure not to have secondary raid attached during kickstart because the kickstart will wipe it.
clearpart --all --drives=sda --initlabel

part /boot --fstype xfs --size=3200
# UEFI /efi - note below
# UEFI /efi - note below
# if you require /boot/efi - enter it below
# part /boot/efi --fstype="vfat" --size=1024
# UEFI /efi - note above
# UEFI /efi - note above

bootloader --location=partition --append="rhgb quiet crashkernel=auto"
part swap --size=16384

volgroup disk0 --pesize=4096 pv.0
part pv.0 --fstype=lvmpv --ondisk=sda --size=100 --grow
part /boot --fstype=xfs --size=3200 --asprimary

logvol /        --vgname=drive0 --fstype=xfs --size=52048 --name=root
logvol /var     --vgname=drive0 --fstype=xfs --size=31024 --name=var
logvol /var/tmp --vgname=drive0 --fstype=xfs --size=31024 --name=vartmp
logvol /var/lib --vgname=drive0 --fstype=xfs --size=31024 --name=varlib
logvol /var/log --vgname=drive0 --fstype=xfs --size=31024 --name=varlog
logvol /opt     --vgname=drive0 --fstype=xfs --size=32000 --name=opt
logvol /tmp     --vgname=drive0 --fstype=xfs --size=20000 --name=tmp
logvol /usr     --vgname=drive0 --fstype=xfs --size=20000 --name=usr
### The next line takes the remainder of the disk and makes it available for later.  I can't tell you how often this has come
### in handy especially when someone asks for a server and they needed additional storage in some directory
### and they didn't mention it until after the build or someone else added things that were unexpected
### If you have a drive that is over a terabyte, know in advance and adjust your partitioning above as required so 
### you don't end up with a 2.3 TB partition below (unless you want it as such)
logvol /justincase --vgname="drive0" --fstype=xfs --size=100 --name=justincase --grow

# Other partitioning to consider only if required
# If you have heavy auditing, consider uncommenting this.  If you use rsyslog to push audit logs elsewhere, you might not need it
#logvol /var/log/audit --vgname=drive0 --fstype=xfs --size=31024 --name=varlogaudit
# NOTE: uncomment if you are going to use docker
#logvol /var/lib/docker --vgname=drive0 --fstype=xfs --size=31024 --name=varlibdock

### end partitioning
### end partitioning

%packages
## put in your list of packages here
# Red Hat Kickstart Generator see  https://access.redhat.com/labs/kickstartconfig/
# this example does not have enough packages, this example focuses on partitioning
@standard
%end

Hope this helps. The idea in this specific post is to give an example in a kickstart of some partitioning in advance. Again, you ought to evaluate it for your own environment. Some will look at this and call it excessive. They are probably right, maybe. However, for my environment, it has helped a lot. We make adjustments as required and have kickstarts based on the role of the server that were pre-considered even before the server was built with the benefit of some years of experience of other such servers and painful experience - and we do this to avoid previous painful experiences. The partitioning example I provided in this reply is an excerpt of a kickstart see the link in the excerpt for the Red Hat kickstart lab and corresponding installation guide.

Regards

RJ

Thank you RJ for your response. There are notable points even though reply is long.

Hi Shiva,

Better too long and well explained than too short and leaving open questions - right ? :)

Regards,
Christian

Indeed that was nicely explained... Thank you RJ for all your valuable inputs.

Hi Sadashiva,

I agree with what you say : definitely cool stuff from RJ (as always) - much appreciated ! :)

Regards,
Christian

Hi Shiva,

For later reference, please change the title: xfs file system decrease & emergency mode issue.

RJ gave you a great recipe how to fix the issues.

==================================================================================================== Requirement: Have a successful backup!, if not do not try this, for failure is not an option!

Warning: names in this recipe may be different on your system:

Now a hint on how to "reduce" the size of a xfs file system.

In fact it is a replacement of the xfs file system by a smaller one.

Be aware you need a DVD/USB drive to do this for the /, /usr or /var mounted file systems.

other filesystems that can be unmounted do not need a boot from a media like DVD/USB drive . Booted from DVD or flash drive, do not chroot to /sysroot

Let us "reduce" /dev/rhel/lvopt, so we can remove /dev/sd4

Let us assume you have enough disk space and create a new logical volume called /dev/rhel/lvopt2 with a xfs file system on it

mkdir /tempmount

mount /dev/rhel/lvopt2 /tempmount

stop all applications that use /opt (if you did not boot from DVD/USB drive)

cp -avr /opt/* /tempmount

ls -R /opt | wc -l # a time consuming calculation if you have a lot of files / packages installed.

umount /opt

check if all files from/opt can be found on /tempmount too.

ls -R /tempmount | wc -l # takes about the same time as ls -R /opt | wc -l (on my demo system over 11 min).

umount /tempmount

if and only if the count are the same:

lvremove /dev/rhel/lvopt

pvmove /dev/sd4

pvs (check if /dev/sd4 is really empty)

vgreduce /dev/rhel /dev/sd4

__needs to be done a running system___

vi /etc/fstab # on a DVD it has to be vi /sysroot/etc/fstab

replace the line /dev/rhel/lvopt /opt …

with /dev/rhel/lvopt2 /opt …

mount /opt

restart your applications #preferred over a start, so you know for sure that pid files are removed.

Regards,

Jan Gerrit Kootstra

Thanks Jan

I have also this issues. How to fix this issues after extend and reduce root partition. Kindly help me.

After extend and reduce root partition, the boot process falls in emergency mode and not working dracut command to rebuild ram disk(initramfs). Please help me how to fix that issues.