Help with LVM & increasing metadata size

Latest response

Hello,

We have a RHEL7 box with a large RAID (few hundred TB) allocated as a single volume group (data-vg) containing a thin pool (data-thin). Within that are 3-4 thinly provisioned logical volumes and 15 odd snapshots.

We are using 58% data but worryingly 73% metadata. The system was left to determine metadata LV size itself on creation of the vg, and we now notice it only allocated 128MB.

I am unable to increase the metadata size, because the volume group and thin pool use all the extents available and I read shrinking thin pools is not supported. (I do see someone has written a tool hosted on GitHub, https://github.com/nkshirsagar/thinpool_shrink but not sure I want to trust our data to it).

The host does have separate RAID1 SSD's for the OS. (It uses LVM for root/swap). I was hoping I could use some space on there to extend the data-vg, but it looks like I can't have multiple vg's on a single physical volume.

Is there any way out of this pickle?

Thanks!

NS
Log in to join the conversation

Responses

RJ Hinton's picture Guru 15795 points
10 January 2021 7:34 PM

Hi Nick Smith,

I’m not sure if I can get to the bottom of this, but I think some more detail will help whomever assists.

Please redact or change anything you do not want public

Can you give us some more detail:

  • What command(s) did you run to determine the metadata usage (perhaps post the command and output)
  • df -PhT /path/to/mountpoint
  • lspci | grep name_of_mountpoint
  • Just curious, what file system type is it formatted with? (XFS, or what?)
  • lvdisplay name_of_lv
  • vgdisplay name_of_vg
  • What is using the thin pool, docker images?
  • What kind of snapshots are these? File system snapshots?

To resolve it through the discussion forum will take a bit of a dive into the details. You might get a more prompt answer through Red Hat support, and include an sosreport

You could post more detail here and I believe others will post here in reply, and perhaps also submit a ticket too.

Regards,
RJ

NS Newbie 10 points
12 January 2021 11:42 AM

Thanks for the pointers RJ, here is some additional information, I hope it's useful:

I determine metadata usage for the thin pool (data-thin) using the lvs command:

lvs -a

LV                                     VG            Attr  LSize   Pool       Origin         Data%  Meta%
---
 data-thin                 data-vg  twi-aotz--  465.67t                                       59.52    76.14                           
 [data-thin_tdata]         data-vg  Twi-ao---- 465.67t                                                                      
 [data-thin_tmeta]         data-vg  ewi-ao---- 120.00m                                                                      
 [lvol0_pmspare]           data-vg  ewi------- 120.00m                                           
--- (other lv's / snaps omitted for brevity)

There are a few lv's in the thin pool - all are formatted as XFS - for example:

df -PhT /my-data/

Filesystem                    Type  Size    Used   Avail Use% Mounted on
/dev/mapper/data--vg-my--data xfs      320T  245T   76T  77% /my-data

lspci

lspci | grep my-data

returns nothing. There is a hardware RAID card on the PCI bus that runs the RAID array.

Thin pool

lvdisplay data-thin

 --- Logical volume ---
  LV Name                data-thin
  VG Name                data-vg
  LV UUID                AxdvF0-AGmG-eu26-4jf9-Fgtu-kKtt-x8THGZ
  LV Write Access        read/write
  LV Creation host, time myhost, 2020-12-07 11:31:32 +0000
  LV Pool metadata       data-thin_tmeta
  LV Pool data           data-thin_tdata
  LV Status              available
  # open                 4
  LV Size                465.67 TiB
  Allocated pool data    59.52%
  Allocated metadata     76.14%
  Current LE             122073027
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:4

Example LV display for my-data

lvdisplay /dev/data-vg/my-data
  --- Logical volume ---
  LV Path                /dev/data-vg/my-data
  LV Name                my-data
  VG Name                data-vg
  LV UUID                dY3r1E-QtYU-T30h-HSvB-9QPD-VQGd-ktI3a0
  LV Write Access        read/write
  LV Creation host, time myhost, 2020-12-07 11:34:40 +0000
  LV Pool name           data-thin
  LV Status              available
  # open                 1
  LV Size                320.00 TiB
  Mapped size            76.90%
  Current LE             83886080
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:6

Volume group:

vgdisplay data-vg
  --- Volume group ---
  VG Name               data-vg
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  97
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                16
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               465.67 TiB
  PE Size               4.00 MiB
  Total PE              122073087
  Alloc PE / Size       122073087 / 465.67 TiB
  Free  PE / Size       0 / 0   
  VG UUID               hsQlxZ-0V8z-6XD1-NceF-IqOn-tYXI-2iDdeD

The host pool is being used as a data storage area and a backup target. We use lvm to take snapshots of the nightly backup and retain them for a period of time.

Please let me know if anything else will help.