Resolve partial lvm logical volume

Submitted by sandip on Wed, 03/19/2014 - 10:51

Server rebooted during lvm snapshot and put volume group in inconsistent state.
```
# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  vg0    1   3   0 wz--n- 229.66G 9.66G
  vg1    2   3   0 wz-pn- 279.46G 9.46G
```
Note the "p" attribute in vg1 which put the group in partial mode.

To resolve, I first vgreduce then re-created the volume group from a good known archive:

# vgreduce --removemissing --force vg1
  vsrv31_var_lib_mysql is expected to have only one segment using it, while it has 0
  Failed to find mirror_seg for vsrv31_var_lib_mysql
  vsrv31_var_log_mysql is expected to have only one segment using it, while it has 0
  Failed to find mirror_seg for vsrv31_var_log_mysql
  vsrv01_var_log is expected to have only one segment using it, while it has 0
  Failed to find mirror_seg for vsrv01_var_log
  Wrote out consistent volume group vg1

Upon listing the volume groups, it automatically removed /dev/sdd1 from vg1.

# vgs
  WARNING: Inconsistent metadata found for VG vg1 - updating to use version 1675
  Removing PV /dev/sdd1 (2tfAMr-erXr-fm1G-dicV-W08Q-eqg6-CL8nbH) that no longer belongs to VG vg1
  VG   #PV #LV #SN Attr   VSize   VFree  
  vg0    1   3   0 wz--n- 229.66G   9.66G
  vg1    1   0   0 wz--n- 139.73G 139.73G

#pvs
  PV         VG   Fmt  Attr PSize   PFree  
  /dev/md2   vg0  lvm2 a-   229.66G   9.66G
  /dev/md3   vg1  lvm2 a-   139.73G 139.73G
  /dev/sdd1       lvm2 --   139.73G 139.73G

Extended vg1 to included sdd1:

# vgextend vg1 /dev/sdd1
  Volume group "vg1" successfully extended

# pvs
  PV         VG   Fmt  Attr PSize   PFree  
  /dev/md2   vg0  lvm2 a-   229.66G   9.66G
  /dev/md3   vg1  lvm2 a-   139.73G 139.73G
  /dev/sdd1  vg1  lvm2 a-   139.73G 139.73G

# vgs
  VG   #PV #LV #SN Attr   VSize   VFree  
  vg0    1   3   0 wz--n- 229.66G   9.66G
  vg1    2   0   0 wz--n- 279.46G 279.46G

# lvs
  LV                 VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  lv0_145186_var_www vg0  -wi-ao  16.00G                                      
  lv0_swap           vg0  -wi-ao   4.00G                                      
  lv0_vz             vg0  -wi-ao 200.00G

Then did a listing of vg1 configs from archives (files located in /etc/lvm/archive):

# vgcfgrestore -l vg1

  
  File:         /etc/lvm/archive/vg1_02156.vg
  VG name:      vg1
  Description:  Created *before* executing '/usr/sbin/lvcreate --size 5000m --snapshot --name snap_vsrv31 /dev/vg1/vsrv31_var_lib_mysql'
  Backup Time:  Thu Feb 27 02:39:26 2014

   
  File:         /etc/lvm/archive/vg1_02157.vg
  VG name:      vg1
  Description:  Created *before* executing '/usr/sbin/lvremove -f /dev/vg1/snap_vsrv31'
  Backup Time:  Thu Feb 27 07:35:34 2014

   
  File:         /etc/lvm/archive/vg1_02158.vg
  VG name:      vg1
  Description:  Created *before* executing '/usr/sbin/lvcreate --size 5000m --snapshot --name snap_vsrv31 /dev/vg1/vsrv31_var_lib_mysql'
  Backup Time:  Fri Feb 28 02:37:38 2014

Restored the last known good archive:

# vgcfgrestore -f /etc/lvm/archive/vg1_02158.vg vg1
  Restored volume group vg1

# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  vg0    1   3   0 wz--n- 229.66G 9.66G
  vg1    2   3   0 wz--n- 279.46G 9.46G

# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/md2   vg0  lvm2 a-   229.66G 9.66G
  /dev/md3   vg1  lvm2 a-   139.73G    0 
  /dev/sdd1  vg1  lvm2 a-   139.73G 9.46G

# lvs
  LV                   VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  lv0_145186_var_www   vg0  -wi-ao  16.00G                                      
  lv0_swap             vg0  -wi-ao   4.00G                                      
  lv0_vz               vg0  -wi-ao 200.00G                                      
  vsrv01_var_log       vg1  -wi---  10.00G                                      
  vsrv31_var_lib_mysql vg1  -wi--- 250.00G                                      
  vsrv31_var_log_mysql vg1  -wi---  10.00G

Activated the volume group:

# vgchange -ay vg1
  3 logical volume(s) in volume group "vg1" now active

# lvs
  LV                   VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  lv0_145186_var_www   vg0  -wi-ao  16.00G                                      
  lv0_swap             vg0  -wi-ao   4.00G                                      
  lv0_vz               vg0  -wi-ao 200.00G                                      
  vsrv01_var_log       vg1  -wi-a-  10.00G                                      
  vsrv31_var_lib_mysql vg1  -wi-a- 250.00G                                      
  vsrv31_var_log_mysql vg1  -wi-a-  10.00G