Problem: Duplicate hdisk5 entry in datavg, unable to mount any FS - I/O Error...
# errpt
...
26120107 0502230508 U S LIBLVM PHYSICAL VOLUME DEFINED AS MISSING
5BEAD71B 0502225308 I S LIBLVM Activation of a no quorum volume group w
...
# lspv
hdisk0 0004f53e0bb5ef0a rootvg active
hdisk1 0004f53e62ebfe8b rootvg active
hdisk5 0004f53e18414d17 datavg active <----- only one disk
hdisk2 0004f53e4e337f67 datavg active
hdisk3 0004f53e4e33821c hb_vg1
hdisk4 0004f53e4e338552 hb_vg2
# lsvg -p datavg
datavg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk2 active 789 0 00..00..00..00..00
hdisk5 missing 394 246 79..00..09..79..79 <------ duplicated entry
hdisk5 active 394 394 79..79..78..79..79 <------ duplicated entry
Diagnose:
In the VGDA of the disks is a dupicate entry reffering to hdisk5. That's why the datavg can be varied on, but the hdisk5 is appearing once as active but once as missing and the filesystems can't be mounted.
Working solution:
Rebuilding of datavg to create new VGDA on all hdisks:
Step 1. Create a PP map file for every LV on datavg to recreate it without loosing data.
1.a - if IT'S possible to varyonvg:
# varyonvg datavg
# lsvg -l datavg
datavg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
lvexphome jfs2 4 4 1 closed/syncd /export/home
lvmqhadata jfs2 40 40 1 closed/syncd /MQHA/XZ0T0021/data
lvmqhalog jfs2 8 8 1 closed/syncd /MQHA/XZ0T0021/log
lvoraexe jfs2 96 96 2 closed/syncd /oraexe
lvora jfs2 8 8 1 closed/syncd /oracle/ORACCVP
lvoraarch jfs2 8 8 1 closed/syncd /oracle/ORACCVP/arch
lvoraarchzip jfs2 24 24 1 closed/syncd /oracle/ORACCVP/archzip
lvorabkp jfs2 96 96 1 closed/syncd /oracle/ORACCVP/backup
lvoradat jfs2 440 440 2 closed/syncd /oracle/ORACCVP/dat1
lvwwwexe jfs2 40 40 1 closed/syncd /wwwexe
lvwwwapp jfs2 96 96 2 closed/syncd /wwwapp
lvmaestro jfs2 12 12 1 closed/syncd /opt/maestro
loglv00 jfs2log 1 1 1 closed/syncd N/A
lvoraexp jfs2 64 64 1 closed/syncd /oracle/ORACCVP/export
Example of creating a PP map for lvora LV:
# lsvg -M datavg | awk '/lvora:/ {print $1}' > lvora.map
# cat lvora.map
hdisk2:205
hdisk2:206
hdisk2:207
hdisk2:208
hdisk2:563
hdisk2:564
hdisk2:565
hdisk2:566
1.b - if it ISN'T possible varyonvg:
# lspv
hdisk0 0004f5680c560e39 rootvg active
hdisk1 0004f568615a79b1 rootvg active
hdisk2 0004f53e4e337f67 datavg
hdisk3 0004f53e4e33821c hb_vg1
hdisk4 0004f53e4e338552 hb_vg2
hdisk5 0004f53e18414d17 datavg
grep list of physical volumes belonging to datavg:
# readvgda /dev/hdisk2 | grep "pvh.pv_id:"
pvh.pv_id: 0004f53e4e337f67
pvh.pv_id: 0004f53e18414d17
1st pv_id is a pv1 - 0004f53e4e337f67 (hdisk2)
2nd pv_id is a pv2 - 0004f53e18414d17 (hdisk5) etc...
Now we can start grep a PP map for lvora LV:
# readvgda /dev/hdisk2 | awk '/lvora:/
* pv1:205:1 lvora:1:0:0:0:0:0
* pv1:206:1 lvora:2:0:0:0:0:0
* pv1:207:1 lvora:3:0:0:0:0:0
* pv1:208:1 lvora:4:0:0:0:0:0
* pv1:563:1 lvora:5:0:0:0:0:0
* pv1:564:1 lvora:6:0:0:0:0:0
* pv1:565:1 lvora:7:0:0:0:0:0
* pv1:566:1 lvora:8:0:0:0:0:0
we need to substitute pv1 for hdisk2, pv2 for hdisk5 and clear unnecessary strings...
# readvgda /dev/hdisk2 | awk -F: '/lvora:/ {sub(/\* /,"");sub(/pv1: */,"hdisk2:");sub(/pv2: */,"hdisk5:");print $1":"$2}'
hdisk2:205
hdisk2:206
hdisk2:207
hdisk2:208
hdisk2:563
hdisk2:564
hdisk2:565
hdisk2:566
Create PP map for every LV in VG:
# ls -la *.map
-rw-r--r-- 1 root system 11 May 04 01:09 loglv00.map
-rw-r--r-- 1 root system 44 May 04 01:08 lvexphome.map
-rw-r--r-- 1 root system 132 May 04 01:09 lvmaestro.map
-rw-r--r-- 1 root system 440 May 04 01:08 lvmqhadata.map
-rw-r--r-- 1 root system 88 May 04 01:08 lvmqhalog.map
-rw-r--r-- 1 root system 88 May 04 01:08 lvora.map
-rw-r--r-- 1 root system 88 May 04 01:08 lvoraarch.map
-rw-r--r-- 1 root system 264 May 04 01:08 lvoraarchzip.map
-rw-r--r-- 1 root system 1056 May 04 01:08 lvorabkp.map
-rw-r--r-- 1 root system 4712 May 04 01:09 lvoradat.map
-rw-r--r-- 1 root system 1056 May 04 01:08 lvoraexe.map
-rw-r--r-- 1 root system 704 May 04 01:09 lvoraexp.map
-rw-r--r-- 1 root system 1056 May 04 01:09 lvwwwapp.map
-rw-r--r-- 1 root system 440 May 04 01:09 lvwwwexe.map
Step 2. Varyoff(if is needed) and export datavg
# varyoffvg datavg
# exportvg datavg
Step 3. Create NEW datavg with correct VGDA
# mkvg -f -y datavg -s 128 -V 100 hdisk2 hdisk5
(-f is required to force creation of NEW VG on already defined defected VG, -V is optional, in my case it was a cluster, so I needed specific MajorNumber for this VG.)
To check MajorNumber of VG use command:
# ls -la /dev/datavg
crw-rw---- 1 root system 100, 0 May 04 05:10 /dev/datavg
Step 4. Recreate all LV's from map file
# mklv -y lvexphome -L /export/home -m lvexphome.map -t jfs2 datavg 4
# mklv -y lvmqhadata -L /MQHA/XZ0T0021/data -m lvmqhadata.map -t jfs2 datavg 40
# mklv -y lvmqhalog -L /MQHA/XZ0T0021/log -m lvmqhalog.map -t jfs2 datavg 8
# mklv -y lvoraexe -L /oraexe -m lvoraexe.map -t jfs2 datavg 96
# mklv -y lvora -L /oracle/ORACCVP -m lvora.map -t jfs2 datavg 8
# mklv -y lvoraarch -L /oracle/ORACCVP/arch -m lvoraarch.map -t jfs2 datavg 8
# mklv -y lvoraarchzip -L /oracle/ORACCVP/archzip -m lvoraarchzip.map -t jfs2 datavg 24
# mklv -y lvorabkp -L /oracle/ORACCVP/backup -m lvorabkp.map -t jfs2 datavg 96
# mklv -y lvoradat -L /oracle/ORACCVP/dat1 -m lvoradat.map -t jfs2 datavg 440
# mklv -y lvwwwexe -L /wwwexe -m lvwwwexe.map -t jfs2 datavg 40
# mklv -y lvwwwapp -L /wwwapp -m lvwwwapp.map -t jfs2 datavg 96
# mklv -y lvmaestro -L /opt/maestro -m lvmaestro.map -t jfs2 datavg 12
# mklv -y loglv00 -m loglv00.map -t jfs2log datavg 1
# mklv -y lvoraexp -L /oracle/ORACCVP/export -m lvoraexp.map -t jfs2 datavg 64
Step 5. Write JFS2 loging method as INLINE to LV control block of every LV (thanks IBM for this command)
# putlvcb -f "vfs=jfs2:log=INLINE:options=rw:account=false" lvmqhadata
# putlvcb -f "vfs=jfs2:log=INLINE:options=rw:account=false" lvmqhalog
# putlvcb -f "vfs=jfs2:log=INLINE:options=rw:account=false" lvoraexe
# putlvcb -f "vfs=jfs2:log=INLINE:options=rw:account=false" lvora
# putlvcb -f "vfs=jfs2:log=INLINE:options=rw:account=false" lvoraarch
# putlvcb -f "vfs=jfs2:log=INLINE:options=rw:account=false" lvoraarchzip
# putlvcb -f "vfs=jfs2:log=INLINE:options=rw:account=false" lvorabkp
# putlvcb -f "vfs=jfs2:log=INLINE:options=rw:account=false" lvoradat
# putlvcb -f "vfs=jfs2:log=INLINE:options=rw:account=false" lvwwwexe
# putlvcb -f "vfs=jfs2:log=INLINE:options=rw:account=false" lvwwwapp
# putlvcb -f "vfs=jfs2:log=INLINE:options=rw:account=false" lvoraexp
Step 6. (optional) Import FS's from datavg to /etc/filesystems
If you don't have a backup of /etc/filesystems use this command to add all FS from datavg.
(all entries from datavg are removed when you run "exportvg datavg" command)
# imfs datavg
(after this command I notice that FS are imported to /etc/filesystem, BUT logging method is not set. I have to fill log = INLINE for every FS from datavg by my hand.)
Step 7. Verify if datavg has been recreated successfuly.
# lsvg -l datavg
datavg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
lvexphome jfs2 4 4 1 closed/syncd /export/home
lvmqhadata jfs2 40 40 1 closed/syncd /MQHA/XZ0T0021/data
lvmqhalog jfs2 8 8 1 closed/syncd /MQHA/XZ0T0021/log
lvoraexe jfs2 96 96 2 closed/syncd /oraexe
lvora jfs2 8 8 1 closed/syncd /oracle/ORACCVP
lvoraarch jfs2 8 8 1 closed/syncd /oracle/ORACCVP/arch
lvoraarchzip jfs2 24 24 1 closed/syncd /oracle/ORACCVP/archzip
lvorabkp jfs2 96 96 1 closed/syncd /oracle/ORACCVP/backup
lvoradat jfs2 440 440 2 closed/syncd /oracle/ORACCVP/dat1
lvwwwexe jfs2 40 40 1 closed/syncd /wwwexe
lvwwwapp jfs2 96 96 2 closed/syncd /wwwapp
lvmaestro jfs2 12 12 1 closed/syncd /opt/maestro
loglv00 jfs2log 1 1 1 closed/syncd N/A
lvoraexp jfs2 64 64 1 closed/syncd /oracle/ORACCVP/export
# mount -a
node mounted mounted over vfs date options
-------- --------------- --------------- ------ ------------ ---------------
/dev/hd4 / jfs2 May 04 05:59 rw,log=/dev/hd8
/dev/hd2 /usr jfs2 May 04 05:59 rw,log=/dev/hd8
/dev/hd9var /var jfs2 May 04 05:59 rw,log=/dev/hd8
/dev/hd3 /tmp jfs2 May 04 05:59 rw,log=/dev/hd8
/dev/hd1 /home jfs2 May 04 06:00 rw,log=/dev/hd8
/proc /proc procfs May 04 06:00 rw
/dev/hd10opt /opt jfs2 May 04 06:00 rw,log=/dev/hd8
/dev/lvmonlogs /mon/logs jfs2 May 04 06:00 rw,log=INLINE
/dev/lvtivoli /usr/local/Tivoli jfs2 May 04 06:00 rw,log=INLINE
/dev/lvmqm /usr/mqm jfs2 May 04 06:00 rw,log=/dev/hd8
/dev/lvmqhadata /MQHA/XZ0T0021/data jfs2 May 04 06:04 rw,log=INLINE
/dev/lvmqhalog /MQHA/XZ0T0021/log jfs2 May 04 06:04 rw,log=INLINE
/dev/lvexphome /export/home jfs2 May 04 06:04 rw,log=/dev/loglv00
/dev/lvmaestro /opt/maestro jfs2 May 04 06:04 rw,log=/dev/loglv00
/dev/lvora /oracle/ORACCVP jfs2 May 04 06:04 rw,log=INLINE
/dev/lvoraarch /oracle/ORACCVP/arch jfs2 May 04 06:04 rw,log=INLINE
/dev/lvoraarchzip /oracle/ORACCVP/archzip jfs2 May 04 06:04 rw,log=INLINE
/dev/lvorabkp /oracle/ORACCVP/backup jfs2 May 04 06:04 rw,log=INLINE
/dev/lvoradat /oracle/ORACCVP/dat1 jfs2 May 04 06:04 rw,log=INLINE
/dev/lvoraexp /oracle/ORACCVP/export jfs2 May 04 06:04 rw,log=INLINE
/dev/lvoraexe /oraexe jfs2 May 04 06:04 rw,log=INLINE
/dev/lvwwwapp /wwwapp jfs2 May 04 06:04 rw,log=INLINE
/dev/lvwwwexe /wwwexe jfs2 May 04 06:04 rw,log=INLINE
Enjoy! :-)
