I have updated Ubuntu from version 13.04 to version 13.10, only to discover that the SATA disks IDs have changed, and my ZFS pool now fails.
On new Ubuntu 13.10 the disk IDs are now ata-* instead of scsi-SATA_*.
This is the pool status after the update:
pool: nestpool
state: UNAVAIL
status: One or more devices could not be used because the label is missing
or invalid. There are insufficient replicas for the pool to continue
functioning.
action: Destroy and re-create the pool from
a backup source.
see: http://zfsonlinux.org/msg/ZFS-8000-5E
scan: none requested
config:
NAME STATE READ WRITE CKSUM
nestpool UNAVAIL 0 0 0 insufficient replicas
raidz2-0 UNAVAIL 0 0 0 insufficient replicas
scsi-SATA_WDC_WD4000F9YZ-_WD-WCC1F0046946 UNAVAIL 0 0 0
scsi-SATA_WDC_WD4000F9YZ-_WD-WCC4A0026423 UNAVAIL 0 0 0
scsi-SATA_WDC_WD4000F9YZ-_WD-WMC1F0011145 UNAVAIL 0 0 0
scsi-SATA_WDC_WD4000F9YZ-_WD-WMC1F0049294 UNAVAIL 0 0 0
scsi-SATA_WDC_WD4000F9YZ-_WD-WMC1F0051143 UNAVAIL 0 0 0
scsi-SATA_WDC_WD4000F9YZ-_WD-WMC1F0051756 UNAVAIL 0 0 0
scsi-SATA_WDC_WD4000F9YZ-_WD-WMC1F0056625 UNAVAIL 0 0 0
scsi-SATA_WDC_WD4000F9YZ-_WD-WMC1F0200560 UNAVAIL 0 0 0
logs
mirror-1 UNAVAIL 0 0 0 insufficient replicas
scsi-SATA_Samsung_SSD_840S1ATNEAD707062H-part2 UNAVAIL 0 0 0
scsi-SATA_Samsung_SSD_840S1ATNEAD707066K-part3 UNAVAIL 0 0 0
After a very long research on the Internet I started following procedure:
First exported the pool using: zpool export nestpool
Then tried to import back the pool using: zpool import -m -f -d /dev/disk/by-id nestpool
(I have also tried with and without all combinations of -m, -f and -d, as well as usind the ID instead of the zpool name)
But the import fails with the message: cannot import 'nestpool': one or more devices is currently unavailable
This is the current output of zpool import
pool: nestpool
id: 3947768928242827823
state: DEGRADED
status: One or more devices contains corrupted data.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
config:
nestpool DEGRADED
raidz2-0 ONLINE
ata-WDC_WD4000F9YZ-09N20L0_WD-WCC1F0046946 ONLINE
ata-WDC_WD4000F9YZ-09N20L0_WD-WCC4A0026423 ONLINE
ata-WDC_WD4000F9YZ-09N20L0_WD-WMC1F0011145 ONLINE
ata-WDC_WD4000F9YZ-09N20L0_WD-WMC1F0049294 ONLINE
ata-WDC_WD4000F9YZ-09N20L0_WD-WMC1F0051143 ONLINE
ata-WDC_WD4000F9YZ-09N20L0_WD-WMC1F0051756 ONLINE
ata-WDC_WD4000F9YZ-09N20L0_WD-WMC1F0056625 ONLINE
ata-WDC_WD4000F9YZ-09N20L0_WD-WMC1F0200560 ONLINE
cache
ata-Samsung_SSD_840_PRO_Series_S1ATNEAD707062H-part1
logs
mirror-1 UNAVAIL insufficient replicas
ata-Samsung_SSD_840_PRO_Series_S1ATNEAD707062H UNAVAIL corrupted data
ata-Samsung_SSD_840_PRO_Series_S1ATNEAD707066K UNAVAIL
Log partitions were not correctly identified, they should be ata-Samsung_SSD_840_PRO_Series_S1ATNEAD707062H-part2 and ata-Samsung_SSD_840_PRO_Series_S1ATNEAD707066K-part3. And the argument -m during import is not helping.
I cannot find a way to tell zpool import to use a different path/id for the log devices. Any help and idea for fixing this problem will be much appreciated. What else can I do to recover this pool?
(Here is the current output of gdisk -l for all disks: http://pastebin.com/bBec21WN)
As everything was still intact and the only problem that prevented importing the pool was the wrong recognition of the partitions used for the log devices, the solution is to create temporary symbolic links under /dev/disk/by-id for the missing partitions.
In my case like this:
ln -sf ../../sda2 ata-Samsung_SSD_840_PRO_Series_S1ATNEAD707062H
ln -sf ../../sdb3 ata-Samsung_SSD_840_PRO_Series_S1ATNEAD707066K
After that, in my case zpool import changed the old wrong IDs to new ones that are present now under Ubuntu 13.10 (each disk and partition has 2 IDs now), and I had to create symbolic links for these as well, like this:
ln -sf ../../sda2 wwn-0x50025385503e8531
ln -sf ../../sdb3 wwn-0x50025385503e8535
After that zpool import automagically started to recognise the correct partitions.
For importing I ran the command:
zpool import -f nestpool
And the pool was imported. DO NOT FORGET to fix the temporary links! Also after a server restart the links will be recreated and fixed.
In my case zpool did recognise the correct partitions during import and no further action was necessary. Otherwise it is necessary to remove the ZIL devices and re-add them back, before anything else!
Do not forget to run scrub and all should be back in place.
A lesson learned for the future is that ZFS recognises the partition if it is the first partition on disk, this is for example the case when using full disks for ZFS. So try to use full disks or when possible only first partition on disks.
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments