CONCEPT
With Oracle 12.1.0.2, Oracle released ASM Filter Driver. As Oracle ASMLib, this feature is also available only on Linux but it gets installed along with the Oracle Grid Software, there is no need to install extra packages like for ASMLib.
Oracle ASM Filter Driver (Oracle ASMFD) is a kernel module that resides in the I/O path of the Oracle ASM disks. This is definitely NOT a replacement of ASMLib but DEFINITELY a better option. You can still go with any of the below configuration:
a) Use ASMLib
b) Use ASM Filter Driver (ASMFD)
c) Don't use any of these at all
In other words, like ASMLib, ASMFD is also an optional configuration but when used, it gives you the capability to scan & label ASM disks and also can be used for device persistence.
There is one major difference between ASMLib and ASMFD, which is, ASMFD rejects any I/O requests that are invalid or not coming through Oracle. This action eliminates accidental overwrites of Oracle ASM disks that would cause corruption in the disks and files within the disk group. I personally like this feature a lot because it eliminates the possibility of accidental damage of ASM disks and this alone is very good reason for me to use this feature in an enterprise.
CONFIGURATION
Now I am going to explain how we can start using ASMFD, if it is not configured on your already configured/running environment.
Here is how we can check if ASMFD is in use or not:
As Grid Infrastructure software owner, run the following command:
oracle@linux-ora-01:+ASM>env | grep ORACLE
ORACLE_SID=+ASM
ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/grid
oracle@linux-ora-01:+ASM>$ORACLE_HOME/bin/asmcmd afd_state
ASMCMD-9526: The AFD state is 'NOT INSTALLED' and filtering is 'DEFAULT' on host 'linux-ora-01'
The above output confirmed that ASMFD is not installed on this box. We can also check the same using sqlplus by login to ASM instance as sysasm:
oracle@linux-ora-01:+ASM>sqlplus / as sysasm
SQL*Plus: Release 12.1.0.2.0 Production on Mon Feb 23 16:53:40 2015
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Automatic Storage Management option
SQL> SELECT SYS_CONTEXT('SYS_ASMFD_PROPERTIES', 'AFD_STATE') FROM DUAL;
SYS_CONTEXT('SYS_ASMFD_PROPERTIES','AFD_STATE')
--------------------------------------------------------------------------------
NOT AVAILABLE
And we have the same results.
Here is all the ASM disk layout of my system:
SQL> select name, path, total_mb, free_mb from v$asm_disk;
NAME PATH TOTAL_MB FREE_MB
------------------------------ ------------------------------ ---------- ----------
DATA_0000 /dev/sdb 51492 43658
DATA_0001 /dev/sdc 51492 43658
FRADG_0000 /dev/sdd 51492 50906
6 rows selected.
And as of now, my ASM discovery string is set to /dev/sd* in ASM resource profile:
oracle@linux-ora-01:+ASM>srvctl config asm
ASM home: <CRS home>
Password file: +DATA/orapwasm
ASM listener: LISTENER
Spfile: +DATA/ASM/ASMPARAMETERFILE/registry.253.867758081
ASM diskgroup discovery string: /dev/sd*
As Grid software owner, set the ASM discovery string to use the labels as well:
oracle@linux-ora-01:+ASM>asmcmd dsset 'AFD:*','/dev/sd*'
oracle@linux-ora-01:+ASM>
oracle@linux-ora-01:+ASM>
oracle@linux-ora-01:+ASM>asmcmd dsget
parameter:AFD:*, /dev/sd*
profile:AFD:*,/dev/sd*
Note: Why we are using "AFD:"? Because once we label the disk, it will show up as AFD:and that name will be used for disk discovery.
We must include the existing ASM disks along with new "AFD:" diskstring, otherwise the command will fail with following error:
oracle@linux-ora-01:+ASM>asmcmd dsset 'AFD:*'
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-15014: path '/dev/sdb' is not in the discovery set (DBD ERROR: OCIStmtExecute)
Stop the HAS stack and configure ASMFD:
oracle@linux-ora-01:+ASM>crsctl stop has -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'linux-ora-01'
CRS-2673: Attempting to stop 'ora.test.db' on 'linux-ora-01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'linux-ora-01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'linux-ora-01' succeeded
CRS-2677: Stop of 'ora.test.db' on 'linux-ora-01' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'linux-ora-01'
CRS-2677: Stop of 'ora.DATA.dg' on 'linux-ora-01' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'linux-ora-01'
CRS-2673: Attempting to stop 'ora.FRADG.dg' on 'linux-ora-01'
CRS-2677: Stop of 'ora.FRADG.dg' on 'linux-ora-01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'linux-ora-01'
CRS-2677: Stop of 'ora.evmd' on 'linux-ora-01' succeeded
CRS-2677: Stop of 'ora.asm' on 'linux-ora-01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'linux-ora-01'
CRS-2677: Stop of 'ora.cssd' on 'linux-ora-01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'linux-ora-01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
oracle@linux-ora-01:+ASM>
Configure the ASMFD as root:
[root@linux-ora-01 ~]# /u01/app/oracle/product/12.1.0.2/grid/bin/asmcmd afd_configure
Connected to an idle instance.
AFD-620: AFD is not supported on this operating system version: 'unknown'
ASMCMD-9524: AFD configuration failed 'ERROR: afdroot install failed'
I am using Oracle Unbreakable Linux 7 and it's still not working for me:
[root@linux-ora-01 ~]# uname -r
3.10.0-123.13.2.el7.x86_64
[root@linux-ora-01 ~]# uname -a
Linux linux-ora-01 3.10.0-123.13.2.el7.x86_64 #1 SMP Thu Dec 18 10:51:45 PST 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@linux-ora-01 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.0 (Maipo)
The above error is due to Oracle bug: 18321597 (Metalink Doc: 1951850.1). Download patch 18321597 and install on Grid home.
PATCHING
Once you have the patch and latest OPatch (12.1.0.1.2 or later) downloaded, apply the patch:
Here is my opatch version:
oracle@linux-ora-01:+ASM>./opatch version
OPatch Version: 12.1.0.1.6
OPatch succeeded.
As you can see I have no interim patches applied so far:
oracle@linux-ora-01:+ASM>./opatch lsinventory
OPatch could not create/open history file for writing.
Oracle Interim Patch Installer version 12.1.0.1.6
Copyright (c) 2015, Oracle Corporation. All rights reserved.
This is a read-only home and users need to run OPatch with -customLogDir option.
Oracle Home : /u01/app/oracle/product/12.1.0.2/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/12.1.0.2/grid/oraInst.loc
OPatch version : 12.1.0.1.6
OUI version : 12.1.0.2.0
Log file location : /u01/app/oracle/product/12.1.0.2/grid/cfgtoollogs/opatch/opatch2015-02-23_19-01-22PM_1.log
Lsinventory Output file location : /u01/app/oracle/product/12.1.0.2/grid/cfgtoollogs/opatch/lsinv/lsinventory2015-02-23_19-01-22PM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Grid Infrastructure 12c 12.1.0.2.0
There are 1 products installed in this Oracle Home.
There are no Interim patches installed in this Oracle Home.
--------------------------------------------------------------------------------
Apply the patch as root user:
Note: If you don't already have Oracle Configuration Management response file (ocm.rsp) then you can create one by executing:
$ORACLE_HOME/OPatch/ocm/bin/emocmrsp
[root@linux-ora-01 OPatch]# export ORACLE_HOME=/u01/app/oracle/product/12.1.0.2/grid
[root@linux-ora-01 OPatch]# export PATH=$ORACLE_HOME/OPatch:$ORACLE_HOME/bin:$PATH
[root@linux-ora-01 OPatch]# which opatch
/u01/app/oracle/product/12.1.0.2/grid/OPatch/opatch
[root@linux-ora-01 OPatch]# which opatchauto
/u01/app/oracle/product/12.1.0.2/grid/OPatch/opatchauto
[root@linux-ora-01 OPatch]# opatchauto apply /u02/media/patches/18321597 -oh /u01/app/oracle/product/12.1.0.2/grid -ocmrf /u02/media/ocm.rsp
OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.
OPatchauto Version : 12.1.0.1.6
OUI Version : 12.1.0.2.0
Running from : /u01/app/oracle/product/12.1.0.2/grid
opatchauto log file: /u01/app/oracle/product/12.1.0.2/grid/cfgtoollogs/opatchauto/18321597/opatch_gi_2015-02-23_19-06-27_deploy.log
Parameter Validation: Successful
Configuration Validation: Successful
Patch Location: /u02/media/patches/18321597
Grid Infrastructure Patch(es): 18321597
It does not contain any DB patch
Patch Validation: Successful
User specified following Grid Infrastructure home:
/u01/app/oracle/product/12.1.0.2/grid
Performing prepatch operations on SIHA Home... Successful
Applying patch(es) to "/u01/app/oracle/product/12.1.0.2/grid" ...
Patch "/u02/media/patches/18321597/18321597" successfully applied to "/u01/app/oracle/product/12.1.0.2/grid".
Performing postpatch operations on SIHA Home... Successful
Apply Summary:
Following patch(es) are successfully installed:
GI Home: /u01/app/oracle/product/12.1.0.2/grid: 18321597
opatchauto succeeded.
Check the recently applied patch as Grid software owner user:
oracle@linux-ora-01:+ASM>./opatch lsinventory
OPatch could not create/open history file for writing.
Oracle Interim Patch Installer version 12.1.0.1.6
Copyright (c) 2015, Oracle Corporation. All rights reserved.
This is a read-only home and users need to run OPatch with -customLogDir option.
Oracle Home : /u01/app/oracle/product/12.1.0.2/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/12.1.0.2/grid/oraInst.loc
OPatch version : 12.1.0.1.6
OUI version : 12.1.0.2.0
Log file location : /u01/app/oracle/product/12.1.0.2/grid/cfgtoollogs/opatch/opatch2015-02-23_19-17-04PM_1.log
Lsinventory Output file location : /u01/app/oracle/product/12.1.0.2/grid/cfgtoollogs/opatch/lsinv/lsinventory2015-02-23_19-17-04PM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Grid Infrastructure 12c 12.1.0.2.0
There are 1 products installed in this Oracle Home.
Interim patches (1) :
Patch 18321597 : applied on Mon Feb 23 19:12:51 CST 2015
Unique Patch ID: 18187560.1
Patch description: "ACFS Patch Set Update : 12.1.0.2.1 (18321597)"
Created on 2 Dec 2014, 21:31:40 hrs PST8PDT
Bugs fixed:
19127216, 19475588, 18957085, 19353057, 19279106, 19201087, 19270227
19335268, 18951113, 19184398, 19149476, 19450090, 19013966, 19183802
19051391, 18321597, 19557156, 19195735, 18877486, 18510745, 19355146
19001684, 18955907, 19060056, 19134464
--------------------------------------------------------------------------------
OPatch failed with error code 15
oracle@linux-ora-01:+ASM>
This patch also starts the Oracle HAS stack, so we need to bring that down again as oracle user:
oracle@linux-ora-01:+ASM>crsctl stop has -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'linux-ora-01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'linux-ora-01'
CRS-2673: Attempting to stop 'ora.test.db' on 'linux-ora-01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'linux-ora-01' succeeded
CRS-2677: Stop of 'ora.test.db' on 'linux-ora-01' succeeded
CRS-2673: Attempting to stop 'ora.FRADG.dg' on 'linux-ora-01'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'linux-ora-01'
CRS-2677: Stop of 'ora.DATA.dg' on 'linux-ora-01' succeeded
CRS-2677: Stop of 'ora.FRADG.dg' on 'linux-ora-01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'linux-ora-01'
CRS-2677: Stop of 'ora.asm' on 'linux-ora-01' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'linux-ora-01'
CRS-2677: Stop of 'ora.evmd' on 'linux-ora-01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'linux-ora-01'
CRS-2677: Stop of 'ora.cssd' on 'linux-ora-01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'linux-ora-01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
oracle@linux-ora-01:+ASM>
Let's try Configuring the ASMFD again as root:
[root@linux-ora-01 ~]# /u01/app/oracle/product/12.1.0.2/grid/bin/asmcmd afd_configure
Connected to an idle instance.
AFD-627: AFD distribution files found.
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.
ASMCMD-9524: AFD configuration failed 'ERROR: OHASD start failed'
We can ignore the last error for now but we do have a working ASMFD now.
oracle@linux-ora-01:+ASM>asmcmd afd_state
Connected to an idle instance.
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'DEFAULT' on host 'linux-ora-01'
Note: ASMLib creates a directory called /dev/oracleasm and ASMFD creates /dev/oraclefd. When we label a disk using ASMFD, a pointer of that will get stored in /dev/oraclefd/disks directory.
Let's start the HAS stack manually:
oracle@linux-ora-01:+ASM>crsctl start has
CRS-4123: Oracle High Availability Services has been started.
LABELING THE DISKS
Label and migrate the existing disks:
oracle@linux-ora-01:+ASM>srvctl stop diskgroup -diskgroup FRADG
oracle@linux-ora-01:+ASM>asmcmd afd_label FRA_DISK1 '/dev/sdd' --migrate
oracle@linux-ora-01:+ASM>srvctl stop diskgroup -diskgroup DATA
oracle@linux-ora-01:+ASM>asmcmd afd_label DATA_DISK1 '/dev/sdb' --migrate
oracle@linux-ora-01:+ASM>asmcmd afd_label DATA_DISK2 '/dev/sdc' --migrate
Rescan the disks and start the diskgroups:
oracle@linux-ora-01:+ASM>asmcmd afd_scan
oracle@linux-ora-01:+ASM>srvctl start diskgroup -diskgroup DATA
oracle@linux-ora-01:+ASM>srvctl start diskgroup -diskgroup FRADG
We can list the disks being used by ASMFD:
oracle@linux-ora-01:+ASM>asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label Filtering Path
================================================================================
FRA_DISK1 ENABLED /dev/sdd
DATA_DISK1 ENABLED /dev/sdb
DATA_DISK2 ENABLED /dev/sdc
oracle@linux-ora-01:+ASM>ls -l /dev/oracleafd/disks/
total 12
-rw-r--r--. 1 oracle dba 9 Feb 23 19:48 DATA_DISK1
-rw-r--r--. 1 oracle dba 9 Feb 23 19:48 DATA_DISK2
-rw-r--r--. 1 oracle dba 9 Feb 23 19:43 FRA_DISK1
These are actually just the text files containing the actual path of the disk:
oracle@linux-ora-01:+ASM>file DATA_DISK2
DATA_DISK2: ASCII text
oracle@linux-ora-01:+ASM>cat DATA_DISK2
/dev/sdc
We have successfully migrated the ASM instance to use ASM Filter Driver.
And now we can also reset the ASM diskstring to scan only AFD disks:
oracle@linux-ora-01:+ASM>asmcmd dsset 'AFD:*'
oracle@linux-ora-01:+ASM>
DESTRUCTION
Now let's test the feature where it won't let any non-oracle I/O to go through and corrupt the disks managed by ASMFD:
oracle@linux-ora-01:+ASM>dd if=/dev/zero of=/dev/sdb bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 1.58866 s, 660 MB/s
oracle@linux-ora-01:+ASM>sqlplus / as sysdba
SQL*Plus: Release 12.1.0.2.0 Production on Mon Feb 23 20:17:39 2015
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Automatic Storage Management option
SQL> col path format a30 trunc
SQL> set lines 120
SQL> select name, path, total_mb, free_mb from v$asm_disk;
NAME PATH TOTAL_MB FREE_MB
------------------------------ ------------------------------ ---------- ----------
FRADG_0000 AFD:FRA_DISK1 51492 50906
DATA_0000 AFD:DATA_DISK1 51492 48829
DATA_0001 AFD:DATA_DISK2 51492 43658
As you can see, the "dd" command did succeed but it didn't actually wipe the disks. Oracle rejected all the I/O which was coming from "dd" command because it was not coming through Oracle software stack.
Same applied to root user as well, even if we run the same "dd" command as root, it will still be bypassed and the disks will remain intact, absolutely amazing!!
Thanks
Daljit Singh
No comments:
Post a Comment