Virtualization, Oracle RAC 11g and more..

1
Virtualization, Oracle RAC 11g and more..

• Vijay Ragunathan
• Guidewire Software Inc
• vragunathan_at_guidewire.com

2
Agenda

• Introduction
• Set up VM and RAC
• Take a look at a few 11g RAC/ASM features
• Introduce Cache fusion and trace Cache Fusion

3
Virtualization

• Abstraction layer between applications and
  hardware
• Methodology to divide the resources of a computer
  hardware into multiple operating environments
• Partition at various levels like Storage,
  Network, Server, Application

4
Why Virtualize

• Consolidate IT infrastructure
• Improve QoS
• Reduce cost
• Provide more reliable working environment for
  development and testing
• Ease replication of environments
• Partition applications to get security,
  reliability and flexibility

5
Where can we Virtualize

• Development and Test
• Virtual Desktops
• PoC deployments
• Disaster Recovery and Business continuity
• Technology refresh

6
Evolution

• Virtualization in place for three decades
• Mainframes
• RISC based systems
• And now commodity hardware

7
Commodity Virtualization Software

• VmWare
• Xen
• Oracle VM (Xen)
• Microsoft Virtual PC
• And probably many more

8
Oracle Real Application Cluster

• Cluster is two or more nodes sharing some common
  resources
• They communicate with each other to check health
  and guarantee data integrity
• For Oracle, it is two or more Oracle instances
  sharing a physical database
• Each instance use their own SGA and background
  processes
• Database components like Datafiles, control files
  and redo log files are shared

9
Real World Oracle RAC Implementations

• RISC or CISC based processers on two or more
  servers
• A SAN or NAS for storing database
• Interconnect mechanisms like Infiniband, Gigabit
  Ethernet, Memory channel

10
Real Application Clusters
Node1
Node2
Storage
11
VM and Oracle RAC

• Hardware for clustering too costly
• Difficult to always build dev and test
  environments with clustering
• Typically Applications built on single instance
  and tested and deployed on RAC
• VM is Beneficial for small, medium and large
  companies and cost effective

12
Real Application Clusters
Virtual Node1
VirtualNode2
Same Disks in Host
Physical Machine
13
Setup Xen
14
Xen Architecture

• Xen hypervisor has management and virtual
  hardware API
• Dom0 hosts the management code and has control
  interfaces to manage other domains
• All guest Virtual machines are referred as domU

15
Packages

• xen-3.0.3-41.el5
• kernel-xen-2.6.18-53.el5
• xen-libs-3.0.3-41.el5
• kernel-xen-devel-2.6.18-53.el5

16
Dom0 Host

• Default Centos 5.1 install
• Virtual Machine Manager GUI
• Command line tool virt-install

17
Virtual Machine Manager

• The GUI VMM displays all the domains and their
  resource usage

18
Create a VM

• File -gt New Machine

19
Name the VM

• We name the first VM as dtrac41

20
Virtualization Method
21
ParaVirtualization

• ParaVirtualization uses a modified kernel
• Not very portable
• Not all hardware is simulated
• Best performing for Network and IO
• Unique address map for each VM
• Easy to implement

22
Fully Virtualized

• Complete Hardware simulation
• Any supported guest OS can be used
• Not supported by all hardware architecture
• VT support may need to be enabled in BIOS for
  some hardware
• Use techniques to trap and emulate instructions

23
Install using PXE

• PXE server enables storing images in central
  location and install large number of machines
• Using centos 5.1 for guestVM
• Kickstart Template file instructs what to install

24
OS and Swap for GuestVM

• Using simple files
• Default path /var/lib/xen/images

25
Network for the GuestVM
26
Memory and CPU

• Do not over allocate as we need resources for the
  dom0 Host and other GuestVMs

27
Summary of the Options and Resources
28
Vm Installation operations

• VMM will start allotting the disk storage
• Create virtual network
• Map memory to the Dom0 Virtual memory map
• Will start installing the Guest OS

29
PXE install looks at dhcp server
30
Xen Utilities

• xm utility
• - can be used to create domains
• - useful for cloning
• virsh
• - can be used to change CPU, memory
• - define config, dump config in xml form for
• cloning
• - start, shutdown and reboot domains

31
Xen Utilities

• xentop
• - similar to top but customized to list all
  virtual resources and domains
• xentop - 215532 Xen 3.1.0-53.el5
• 3 domains 2 running, 1 blocked, 0 paused, 0
  crashed, 0 dying, 0 shutdown
• Mem 15723456k total, 15721284k used, 2172k free
  CPUs 8 _at_ 2992MHz
• NAME STATE CPU(sec) CPU() MEM(k)
  MEM() MAXMEM(k) MAXMEM() VCPUS NETS NETTX(k)
  NETRX(k) VBDS VBD_OO VBD_RD VBD_WR SSID
• Domain-0 -----r 9576 2.9 13498460
  85.8 no limit n/a 8 4 398061
  38880 0 0 0 0 0
• dtrac41 -----r 2755 2.1 1048152
  6.7 1048576 6.7 1 2 883749
  857104 0 0 0 0 0
• dtrac42 --b--- 1302 8.1 1048080
  6.7 1048576 6.7 1 2 850349
  813954 0 0 0 0 0

32
XenStore

• Configuration database for all domain
• Multiple commands to manage this database
• xenstore-ls
• - can be used to list details of all hardware
  components in each domain

33
Add Virtual Hardware

• Add more storage for Oracle clusterware

34
Add Virtual Storage

• Specify Simple file or disk partition (if
  available)
• Min size of 256Mb for Clusterware

35
Add virtual network interface

• For Interconnect use

36
Add virtual network interface

• Choose virtual network as this interconnect does
  not need to talk to the external world

37
Virtual hardware of one guestVM
38
Oracle Install Preparation

• Update sysctl.conf with the new kernel parameters
• rmem_default and wmem_default should be 4MB min
• Update limits.conf
• Install any additional RPMs needed as listed in
  Oracle Install documentation
• Edit /etc/hosts and add localhost and 6 other
  addresses to indicate public, private and virtual
  networks for both guestVMs

39
Create users and groups

• groupadd g 10000 oinstall
• groupadd g 10001 dba
• groupadd g 10002 asmdba
• useradd g oinstall Goinstall,dba,asmdba d
  /home/oracle u 10000 oracle
• passwd oracle

40
Mapping raw devices

• Create something like /etc/rc3.d/S90raw and/or
  /etc/rc5.d/S90raw based on boot run level with
  contents like
• raw /dev/raw/raw1 /dev/xvdb
• raw /dev/raw/raw2 /dev/xvdc
• raw /dev/raw/raw3 /dev/xvdd
• raw /dev/raw/raw4 /dev/xvde
• chown rootdba /dev/raw/raw2
• chmod 660 /dev/raw/raw2
• chown oracledba /dev/raw/raw1
• chmod 660 /dev/raw/raw1
• chown oracle /dev/raw/raw3
• chown oracle /dev/raw/raw4
• chmod 660 /dev/raw/raw3
• chmod 660 /dev/raw/raw4

41
Cloning GuestVM

• In dom0, /etc/xen/dtrac41 has list of devices
  created for this guest VM
• Copy /etc/xen/dtrac41 to /etc/xen/dtrac42
• Change the mac address of the two virtual network
  interfaces in dtrac42 and edit the primary disk
• Modify the name and uuid for the new VM
• Ensure the order of the network is right in the
  config files
• Copy the virtual disk of dtrac41(dtrac41.img) to
  /var/lib/xen/images/dtrac42.img

42
Config file for guestVM

• name "dtrac41"
• uuid "3656ca7c24e378bd2c63d380206a5f8d"
• maxmem 1024
• memory 1024
• vcpus 1
• bootloader "/usr/bin/pygrub"
• kernel "/var/lib/xen/boot_kernel.qFE_y1"
• ramdisk "/var/lib/xen/boot_ramdisk.HlhabB"
• extra "ro root/dev/vgSys/lvRoot rhgb quiet
  consoletty0 consolettyS1,57600n8"
• on_poweroff "destroy"
• on_reboot "restart"
• on_crash "restart"
• vfb "typevnc,vncdisplay0,keymapen-us"
• disk "tapaio/var/lib/xen/images/dtrac41.img,
  xvda,w", "tapaio/var/lib/xen/images/dtrac4x-ocr1
  1g-shared.img,xvdb,w", "tapaio/var/lib/xen/image
  s/dtrac4x-vote-shared.img,xvdc,w",
  "tapaio/var/lib/xen/images/dtrac4x-asm1-shared.i
  mg,xvdd,w", "tapaio/var/lib/xen/images/dtrac4x-a
  sm2-shared.img,xvde,w"
• vif "mac00163e0c81f5,bridgexenbr0,scrip
  tvif-bridge", "mac00163e754184,bridgevirbr
  0,scriptvif-bridge"

43
Cloning GuestVM

• Use xm to create the domain
• xm create dtrac42
• Start the new domain
• xm start dtrac42
• You should see both the GuestVMs in the Virtual
  Machine Manager

44
Setup GuestVMs

• Reboot both the virtual machines
• Ensure fdisk l is showing all the disks
• Edit the network configuration files to setup the
  IP addresses and hostname and start the network
• Ping the IP addresses of the other node and
  confirm that the network works fine

45
User equivalence Setup

• Use ssh-keygen and generate keys
• Concatenate the public keys for both machines and
  store as /.ssh/authorized_keys
• Check you can login to all combinations without
  prompts. For e.g.,
• ssh dtrac41 date
• ssh dtrac42 date
• ssh dtrac41p date
• ssh dtrac42p date
• Run cluster verify tool and do pre CRS inst check

46
Vmware

• Install Vmware server
• Create two virtual machines using Vmware Server
  Console
• For the shared disks, edit the config files
  (suffixed vmx) and add
• disk.locking "false
• scsi01.shared "true"
• scsi02.shared "true"
• scsi03.shared "true"
• scsi04.shared "true"

47
Oracle install media

• Download the clusterware and database software
• Unzip the files
• unzip linux_11gR1_clusterware.zip
• unzip linux_11gR1_database.zip
• If using cdrom/dvd, mount the clusterware media
  first
• Start the Installer
• cd clusterware
• ./runInstaller

48
Clusterware Software Home
49
Cluster Configuration

• Cluster name is dtrac4_cluster
• Edit details of dtrac41
• Add details of dtrac42

50
Network Interface Usage

• Select the right Interface for use as Public and
  Private network
• Here eth0 is chosen as Public and eth1 as private

51
OCR and Voting disk

• Both OCR in the previous installer screen and
  voting disk here should have min of 256MB space
• Specify mirror copies here we use single voting
  disk location

52
Clusterware Install Summary

• Ensure all cluster nodes are seen

53
Root Scripts

• Run orainstRoot.sh and root.sh on all nodes (one
  at a time)

54
Root.sh on node1

• Configures OCR and Voting Disk and starts CRS
  stack on node1

55
Root.sh on node2

• Starts CRS stack on node2 and creates cluster
  resources

56
Config Assistants for Clusterware
57
Clusterware Install Completion

• That completes the clusterware install
• OCR and Voting disk are configured
• Resources like ONS and VIP should be created by
  now and can be seen using crs_stat
• Processes like evmd, crsd, ocssd and oprocd
  should be running now
• Ready to Install RAC software
• cd database
• ./runInstaller

58
Installation Type
59
Node Selection

• Choose all nodes in the cluster both VMs

60
OS Authentication
61
RAC Install Summary

• Check for selection of all Cluster nodes

62
Create RAC database
63
Create ASM Disk Group

• Change disk discovery path and check permissions
  if disks are not seen

64
Starts Cluster database

• Database Creation is complete

65
CRS status

• crs_stat -t
• Name Type Target State
  Host
• --------------------------------------------------
  ----------
• ora....SM1.asm application ONLINE ONLINE
  dtrac41
• ora....41.lsnr application ONLINE ONLINE
  dtrac41
• ora....c41.gsd application ONLINE ONLINE
  dtrac41
• ora....c41.ons application ONLINE ONLINE
  dtrac41
• ora....c41.vip application ONLINE ONLINE
  dtrac41
• ora....SM2.asm application ONLINE ONLINE
  dtrac42
• ora....42.lsnr application ONLINE ONLINE
  dtrac42
• ora....c42.gsd application ONLINE ONLINE
  dtrac42
• ora....c42.ons application ONLINE ONLINE
  dtrac42
• ora....c42.vip application ONLINE ONLINE
  dtrac42
• ora.xenrac.db application ONLINE ONLINE
  dtrac42
• ora....c1.inst application ONLINE ONLINE
  dtrac41
• ora....c2.inst application ONLINE ONLINE
  dtrac42

66
RAC and ASM

• 11g New Features

67
11g New Features

• Automatic Diagnostic Repository should be placed
  in shared disk if using shared oracle home
• Kill sessions from anywhere in the cluster
• ADDM for RAC dbms_addm.analyze_db procedure and
  dbms_addm.get_report function
• Inter-node parallelism can use services to
  determine the instances that it can use
• Grid provisioning and EM changes
• Clone ASM and RAC homes clone.pl

68
11g ASM New Features

• asm_preferred_read_failure_groups parameter
• ASM rolling migration
• alter system start rolling migration to
  ltpnumbergt
• Convert single instance ASM to cluster using
  rconfig/EM
• ASM compatibility attributes
• Variable size extents
• Higher AU values for Coarse striping 1M 64M

69
11g ASM New features

• Ability to drop ASM instances using dbca
• dbca silent deleteasm nodelist
  dtrac41,dtrac42
• ASM fast mirror resync
• SYSASM and OSASM
• ASMcmd command additions cp, md_backup, lsdsk,
  remap
• Mount/drop diskgroup with Force option

70
Cache Fusion
71
Cache Fusion

• Shared cache architecture
• Cache coherency preserve integrity of data
  across different caches for the same data
• Employs high-speed Interconnects
• Created in Oracle OPS 8i to reduce pings
• 8i handled read/write contention
• 9i RAC handles other contentions like write-write

72
Cache Fusion

• Locks mode, role, past image
• E.g lock SG1
• Modes Null, Shared, Exclusive
• Role Local, Global
• Past Image component 0 or 1
• Maximum 3-way protocol (master, holder and
  requestor)

73
Events

• Event 10432 for diagnosing unexpected GCS
  events and lock requests
• Event 10708 shows block request and grants
  (level 7 or 103 is common)
• Event 10046 common sql trace and combines well
  with event 10708

74
Trace commands

• SQLgt alter system set events '10432 trace name
  context forever, level 1510708 trace name
  context forever, level 103'
• System altered.
• SQLgt conn scott/tiger
• Connected.
• SQLgt alter session set events '10046 trace name
  context forever, level 8'
• Session altered.

75
First Read session

• Select on node 1
• 2008-05-15 024036.228
• kclscrs req0 block5/36
  
  
• 2008-05-15 024036.228
  
  kclscrs req0 typimmediate modeS
• WAIT 3 nam'db file sequential read' ela 786
  file5 block36 blocks1 obj16215
  tim1210844436231630
• 2008-05-15 024036.231
  
  KCL P02 cr prewarm
• 
  
  2008-05-15 024036.231
• kclscrs req0 block5/28
  
  
• 2008-05-15 024036.231
  
  kclscrs req0 typimmediate modeS
• 
  
  2008-05-15 024036.231
• kclscrs req1 block5/29
  
  
• 2008-05-15 024036.231
  
  kclscrs req1 typimmediate modeS
• 2008-05-15 024036.232
• kclscrs req7 block5/25
• 2008-05-15 024036.232
• kclscrs req7 typimmediate modeS
• WAIT 3 nam'db file scattered read' ela 7342
  file5 block25 blocks8 obj16214
  tim1210844436239474

76
Second read from same node

• PARSE 9c0,e1478,p0,cr0,cu0,mis1,r0,dep0,
  og1,tim1210847692139409
• EXEC 9c0,e37,p0,cr0,cu0,mis0,r0,dep0,og
  1,tim1210847692139492
• WAIT 9 nam'SQLNet message to client' ela 1
  driver id1650815232 bytes1 p30 obj523
  tim1210847692139514
• FETCH 9c0,e29,p0,cr2,cu0,mis0,r1,dep0,og
  1,tim1210847692139568
• ? No communication to the other node

77
Write Operation Tracing..

• update emp set salsal100 where empno 7654
• END OF STMT
• PARSE 3c20001,e44395,p1,cr55,cu0,mis1,r0,
  dep0,og1,tim1210848457075954
• WAIT 3 nam'enq TM - contention' ela 226
  namemode1414332419 object 16214
  table/partition0 obj-1 tim1210848457076262
• 2008-05-15 034737.076
• kclscrs req0 block5/36
• 2008-05-15 034737.076
• kclscrs bid13202c00200..

78
Tracing Contd..

• Lms trace of remote node
• 2008-05-15 034737.077
• kcl_snd_cur block5/36 time54
• Back to update session trace
• 2008-05-15 034737.077
• kclwcrs got 1 blocks from ksxprcvimd
• WAIT 3 nam'gc current block 2-way' ela 413
  p15 p236 p31 obj16215 tim1210848457077694
• 2008-05-15 034737.077
• kclwcrs req0 typcur wtyp2hop tm489

79
Tracing Contd..

• 2008-05-15 034737.140
• kcllkopb req7 block5/25
• 2008-05-15 034737.140
• kcllkopb bid13202140..
• WAIT 3 nam'gc current multi block request'
  ela 27265 file5 block25 id33554446
  obj16214 tim1210848457226586
• Nothing in the LMS trace
• Back to sessions trace
• 2008-05-15 034737.254
• kcllkopb req0 typcur wtyp2hop tm174615
• WAIT 3 nam'db file scattered read' ela 83230
  file5 block25 blocks3 obj16214
  tim1210848457337661

80
Causing Write-Write Contention

• update emp set salsal100 where empno ! 7654
• PARSE 5c4000,e49748,p1,cr9,cu0,mis1,r0,de
  p0,og1,tim1210848625816852
• 2008-05-15 035025.817
• kclscrs req0 block5/28
• 2008-05-15 035025.817
• kclscrs bid1310250040..

81
Write-Write Contention Tracing..

• Remote LMS
• 2008-05-15 035025.816
• kclgcr req231 block5/28
• 2008-05-15 035025.816
• kclgcr bid13102500..
• 2008-05-15 035025.825
• kclqsnd req231 status2 fscn0b49c2 time8523
• 2008-05-15 035025.826
• kclpdc le1c/50000 rls49

82
Write-Write Contention Tracing..

• Back to local session
• 2008-05-15 035025.825
• kclwcrs got 1 blocks from ksxprcvimd
• WAIT 5 nam'gc cr block busy' ela 8846 p15
  p228 p31 obj16214 tim1210848625825992
• 2008-05-15 035025.825
• kclwcrs got 1 blocks from ksxprcvimd
• WAIT 5 nam'gc cr block busy' ela 8846 p15
  p228 p31 obj16214 tim1210848625825992
• 2008-05-15 035025.826
• kclwcrs req0 typcr wtypbusy tm9178

83
Write - Write contention tracing

• 2008-05-15 035025.826
• kcllkopb req0 block5/28
• 2008-05-15 035025.826
• kcllkopb bid13102600
• LMS trace
• 2008-05-15 035025.826
• kcl_snd_cur block5/28 time64
• Local Session Trace
• 2008-05-15 035025.827
• kcllkopb got 1 blocks from ksxprcvimd
• WAIT 5 nam'gc current block 2-way' ela 270
  p15 p228 p333554433 obj16214
  tim1210848625827163
• 2008-05-15 035025.827
• kcllkopb req0 typcur wtyp2hop tm417

84
Q A