Technology Tips And Tricks

Part 4 of the 5 part series on the step by step installation post focuses on setting up the Sybase cluster. Part 1 has a list and links to the download pages for this software.

The 5 parts of this series are:

• Part 1: System Requirements and Test Setup Concept
• Part 2: Setup of the VMWare Server and the Linux Servers
• Part 3: Install of the Sybase Software
• Part 4: Setup and test the Sybase Cluster
• Part 5: Add a additional node to the cluster (planned release 12/30/2009)

The instructions designed that non-Sybase DBA can perform these tasks. Sybase DBA will find that the steps are very similar to your standard ASE installation.

Please click here to read about how to install the Sybase software.

Sybase Setup

Run uafstartup.sh

Before continuing with the cluster install, each node needs to have a running agent waiting for the connection. Without this agent Sybase Cluster Edition will not install nor run.
On asece1 and asece2 as user sybase run this command.

$SYBASE_UA/bin/uafstartup.sh &

This is the out from running the command. Make sure no errors are displayed.

sybase@asece1:~> $SYBASE_UA/bin/uafstartup.sh &
[1] 12142
sybase@asece1:~>
Node: asece1
Operating System: Linux
OS Architecture: x86_64
OS Version: 2.6.16.60-0.21-default
JAVA Runtime: /sybase/shared/jre150_012
java version "1.5.0_12"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_12-b04)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_12-b04, mixed mode)
Message level: WARN
Creating a UAF nodes subdirectory representing the node asece1
Agent Home: /sybase/UAF-2_5/nodes/asece1
2008-11-25 15:44:53,500 INFO [main] Starting Unified Agent...
2008-11-25 15:44:53,508 INFO [main] Version: 2.5.0(983)
2008-11-25 15:44:53,705 INFO [main] Initializing BootstrapService...
2008-11-25 15:44:53,705 INFO [main] Starting BootstrapService...
2008-11-25 15:44:54,196 INFO [main] Host: asece1.cluster.local
2008-11-25 15:44:54,199 INFO [main] Agent home: /sybase/UAF-2_5/nodes/asece1
2008-11-25 15:44:54,409 INFO [main] Loading 7 primordial services...
2008-11-25 15:44:54,616 INFO [main] Registered Agent Service
2008-11-25 15:44:55,253 INFO [main] Initializing Agent Service...
2008-11-25 15:44:55,257 INFO [main] Starting Agent Service...
2008-11-25 15:44:55,802 INFO [main] Registered Environment Service
2008-11-25 15:44:56,516 INFO [main] Registered Configuration Service
2008-11-25 15:44:57,229 INFO [main] Registered Session Service
2008-11-25 15:44:57,543 INFO [main] Registered Security Service
2008-11-25 15:44:58,180 INFO [main] Registered Service Registration Service
2008-11-25 15:44:58,218 INFO [main] Initializing Service Registration Service...
2008-11-25 15:44:58,219 INFO [main] Starting Service Registration Service...
2008-11-25 15:44:58,656 INFO [main] Validated 9 extended services.
2008-11-25 15:44:58,656 INFO [main] Check circular dependencies...
2008-11-25 15:44:58,657 INFO [main] OK. No circular dependencies are found.
2008-11-25 15:44:58,837 INFO [main] Initializing RMI Service...
2008-11-25 15:44:58,928 INFO [main] Starting RMI Service...
2008-11-25 15:44:59,588 INFO [main] Agent URL in long format: service:jmx:rmi:///jndi/rmi://asece1.cluster.local:9999/agent
2008-11-25 15:44:59,588 INFO [main] Agent URL in short format: rmi://asece1.cluster.local:9999
2008-11-25 15:44:59,589 INFO [main] TDS Service will not be registered because registerOnStartup is false.
2008-11-25 15:44:59,592 INFO [main] SNMP Service will not be registered because registerOnStartup is false.
2008-11-25 15:44:59,593 INFO [main] DeploymentService: Found 1 dependencies.
2008-11-25 15:44:59,593 INFO [main] Dependent service RemoteShellService needs to be registered...
2008-11-25 15:44:59,809 INFO [main] DeploymentService: successfuly resolved dependencies.
2008-11-25 15:44:59,931 WARN [main] The template directories has been preset to: [/sybase/UAF-2_5/nodes/asece1/templates]
2008-11-25 15:45:00,275 INFO [main] SelfDiscoveryService: Found 1 dependencies.2008-11-25 15:45:00,279 INFO [main] Dependent service RMIService already registered.
2008-11-25 15:45:00,280 INFO [main] SelfDiscoveryService: successfuly resolved dependencies.
2008-11-25 15:45:00,801 INFO [main] Initializing Plugin Registration Service...2008-11-25 15:45:00,801 INFO [main] Starting Plugin Registration Service...
2008-11-25 15:45:00,875 INFO [main] Validated 2 plugin directories.
2008-11-25 15:45:01,238 INFO [main] Plugin registered. Updating lookup info...
2008-11-25 15:45:01,392 WARN [main] The Sybase home directory should be specified in the com.sybase.sysam agent-plugin.xml file.
2008-11-25 15:45:01,786 WARN [main] The Sysam home directory should be specified in the com.sybase.sysam agent-plugin.xml file.
2008-11-25 15:45:01,789 WARN [main] Log file could not be located. This may be that the license manager has not been started
2008-11-25 15:45:02,240 INFO [main] Plugin com.sybase.ase.snmp_2.0.0: Found 2 dependencies.
2008-11-25 15:45:02,419 INFO [main] Validated 9 extended services.
2008-11-25 15:45:02,420 INFO [main] Check circular dependencies...
2008-11-25 15:45:02,420 INFO [main] OK. No circular dependencies are found.
2008-11-25 15:45:02,425 INFO [main] TdsService: Found 1 dependencies.
2008-11-25 15:45:02,425 INFO [main] Dependent service RMIService already registered.
2008-11-25 15:45:02,426 INFO [main] TdsService: successfuly resolved dependencies.
2008-11-25 15:45:02,849 INFO [main] Validated 9 extended services.
2008-11-25 15:45:02,850 INFO [main] Check circular dependencies...
2008-11-25 15:45:02,850 INFO [main] OK. No circular dependencies are found.
2008-11-25 15:45:03,770 INFO [main] Plugin com.sybase.ase.snmp_2.0.0: successfuly resolved dependencies.
2008-11-25 15:45:04,059 INFO [main] Plugin registered. Updating lookup info...
2008-11-25 15:45:04,076 INFO [main] Finished loading primordial services.
2008-11-25 15:45:04,076 INFO [main] Bootstrap completed successfully.
sybase@asece1:~>

Run sybcluster

On the node asece1 execute the cluster creation command. This interactive session will be used to create the configuration file that the can be used to create the cluster with a batch file. This way unforeseen problems can be corrected in the batch file and the cluster creation can restart.

> create cluster

There are certain circumstances when the install will fail. Most of the time it is a size mismatch of the raw devices. If this happens just scale down the raw device size down a little bit. It seems that the size check for the master device causes problems if set to the max size of the raw device.

Once the problem is correct in the .xml file, restart the sybcluster tool and enter the following command.

sybase@asece1:~> sybcluster -U uafadmin -P -C mycluster -F "asece1,asece2"
> create cluster mycluster file /home/sybase/mycluster.xml
During this run answer the questions about reuse the configuration with Y to complete this step.
Here's the output from the batch run:
INFO - Creating the Cluster Agent plugin on host address asece1.cluster.local using agent: asece1.cluster.local:9999
A cluster agent plugin at asece1.cluster.local:9999 is already managing a cluster by the name mycluster.
Should this agent plugin be reused for the cluster: [ N ] y
INFO - The Cluster Agent Plugin on agent asece1.cluster.local:9999 will be reused.
INFO - Creating the Cluster Agent plugin on host address asece2.cluster.local using agent: asece2.cluster.local:9999
A cluster agent plugin at asece2.cluster.local:9999 is already managing a cluster by the name mycluster.
Should this agent plugin be reused for the cluster: [ N ] y
INFO - The Cluster Agent Plugin on agent asece2.cluster.local:9999 will be reused.
Would you like to check whether this device supports IO fencing capability (Y/N)? [ Y ] n
INFO - Cluster "mycluster" creation in progress.
INFO - Choosing the first instance to be created using the connected agent...
INFO - The Sybase home directory is /sybase.
INFO - The ASE home directory is /sybase/ASE-15_0.
INFO - Retrieving environment variables from /sybase/SYBASE.sh.
INFO - The first instance created will be ase1.
INFO - Warning: Unable to verify /dev/raw/raw1 device size. Please verify that this
INFO - device is not already in use and that it has sufficient space available.
INFO - Warning: Unable to verify /dev/raw/raw4 device size. Please verify that this
INFO - device is not already in use and that it has sufficient space available.
INFO - Building Adaptive Server 'ase1':
INFO - Writing entry into directory services...
INFO - Directory services entry complete.
INFO - Building master device...
INFO - Master device complete.
INFO - Starting server...
INFO - Server started.
INFO - Building sysprocs device and sybsystemprocs database...
INFO - sybprocs device and sybsystemprocs database created.
INFO - Running installmaster script to install system stored procedures...
INFO - installmaster: 10% complete.
INFO - installmaster: 20% complete.
INFO - installmaster: 30% complete.
INFO - installmaster: 40% complete.
INFO - installmaster: 50% complete.
INFO - installmaster: 60% complete.
INFO - installmaster: 70% complete.
INFO - installmaster: 80% complete.
INFO - installmaster: 90% complete.
INFO - installmaster: 100% complete.
INFO - installmaster script complete.
INFO - Creating two-phase commit database...
INFO - Two phase commit database complete.
INFO - Installing common character sets (Code Page 437, Code Page 850, ISO Latin-1,
INFO - Macintosh and HP Roman-8)...
INFO - Character sets installed.
INFO - Server 'ase1' was successfully created.
INFO - Adding cluster and instance entries to the interfaces file at /sybase.
INFO - Connecting to the dataserver using the host and query port asece1.cluster.local:19786.
INFO - Creating the Local System Temporary device mycluster1_tempdb at /dev/raw/raw6 of size 1000M.
INFO - Creating the Local System Temporary device mycluster2_tempdb at /dev/raw/raw7 of size 1000M.
INFO - ase1: Creating the Local System Temporary database mycluster_tdb_1 on mycluster1_tempdb of size 1000M.
INFO - ase2: Creating the Local System Temporary database mycluster_tdb_2 on mycluster2_tempdb of size 1000M.
INFO - The cluster is now configured. Shutting down this first instance.
The cluster mycluster was successfully created.
INFO -
>

By default the cluster will be down. To start the cluster enter the following commands.

> connect
mycluster> start cluster
.
.
Startup output messages
.
.

Install Auxiliary Servers

Install a Backup Server

sybase@asece1:~> sybcluster -U uafadmin -P -C mycluster -F "asece1,asece2"

> connect

mycluster> create backupserver
Enter the Backup Server name: [ mycluster_BS ]
Enter Backup Server log file path: [ /sybase/ASE-15_0/install/mycluster_BS.log ]
Do you want to create Backup Server for node "asece1.cluster.local"? [ Y ]
Enter the Backup Server port number for node "asece1.cluster.local": 19799
Do you want to create Backup Server for node "asece2.cluster.local"? [ Y ]
Enter the Backup Server port number for node "asece2.cluster.local": 19799
The Backup Server "mycluster_BS" was successfully defined.
mycluster>

It is not necessary to install the backup server on both nodes. The second node is failover node. Initiating a backup on the cluster edition is the same as with the standard ASE backup. Simply execute the commands dump database or dump tran. To restore the database the same commands as with the standard ASE version is used. Load database will reload a database dump to the system.

Install Job Scheduler

Before installing the job scheduler a device named sybmgmtdev must be created. This device must be shared with all nodes.

sybase@asece1:/sybase/ASE-15_0/scripts> isql -Usa -P -Smycluster -i ./installjsdb

(return status = 0)
Starting the setup of sybmgmtdb.
Using device sybmgmtdev for the sybmgmtdb database.
CREATE DATABASE: allocating 86016 logical pages (168.0 megabytes) on disk
'sybmgmtdev' (86016 logical pages requested).
CREATE DATABASE: allocating 6144 logical pages (12.0 megabytes) on disk
'sybmgmtdev' (6144 logical pages requested).
Caution: You have set up this database to include space on disk 5 for both data
and the transaction log. This can make recovery impossible if that disk fails.
Database option 'trunc log on chkpt' turned ON for database 'sybmgmtdb'.
Running CHECKPOINT on database 'sybmgmtdb' for option 'trunc log on chkpt' to
take effect.
(return status = 0)
Database option 'select into/bulkcopy/pllsort' turned ON for database
'sybmgmtdb'.
Running CHECKPOINT on database 'sybmgmtdb' for option 'select
into/bulkcopy/pllsort' to take effect.
(return status = 0)
Executing checkpoint in sybmgmtdb
Database log is 6144 pages, 30% log free is at 1843 pages, the last chance
threshold is at 448 pages.
installing sp_js_logthreshold at 1843 pages.
Adding threshold for segment 'logsegment' at '1843' pages.
DBCC execution completed. If DBCC printed error messages, contact a user with
System Administrator (SA) role.
(return status = 0)
Created js_jobs.
Created js_commands.
Created js_schedules.
Created js_scheduledjobs.
Created js_seqsql.
Created js_callouts.
Created js_history.
Created js_output.
Created js_keys.
Created js_templates.
Created js_xml.
Created js_daynames.
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 0)
(return status = 1)
(return status = 0)
Setup of sybmgmtdb is done.
Password correctly set.
Account unlocked.
New login created.
(return status = 0)
Account locked.
(return status = 0)
Authorization updated.
(return status = 0)
Authorization updated.
(return status = 0)
Option changed.
(return status = 0)
Option changed.
(return status = 0)
New user added.
(return status = 0)
Installing Job Scheduler Task js_seqsql sequencer code
: js_seqsql installed token 0 - BOOTSTRAP FIRMWARE
: js_seqsql installed token 1 - PROBE JS_CALLOUTS
: js_seqsql installed token 2 - PROBE JS_HISTORY
: js_seqsql installed token 3 - CLEAN JS_CALLOUTS
: js_seqsql installed token 4 - POPULATE JS_CALLOUTS
: js_seqsql installed token 5 - GET CALLOUT SIZE
: js_seqsql installed token 6 - LOAD CALLOUT
: js_seqsql installed token 7 - LOAD JS_SCHEDULEDJOBS
: js_seqsql installed token 8 - GET JS_SCHEDULES
: js_seqsql installed token 9 - SET JSC_START
: js_seqsql installed token 10 - GET MAX JSH_EXID
: js_seqsql installed token 11 - GET SYB_JSAGENT
: js_seqsql installed token 12 - SET RUNNABLE
: js_seqsql installed token 13 - SET COMPLETED
: js_seqsql installed token 14 - SET TERMINATED
: js_seqsql installed token 15 - SET EXPIRED
: js_seqsql installed token 16 - EXEC JSAGENT
: js_seqsql installed token 17 - EXEC DIAGJSAG
: js_seqsql installed token 18 - INSERT CALLOUT
: js_seqsql installed token 19 - DELETE CALLOUT
: js_seqsql installed token 20 - ADD JSTASK
: js_seqsql installed token 21 - UPDATE CALLOUT
: js_seqsql installed token 22 - SET JSC_STATE
: js_seqsql installed token 23 - CLEAN JSH_STATE
: js_seqsql installed token 24 - EXPIRE SCHEDULE
: js_seqsql installed token 25 - UNLOCK JSTASK
: js_seqsql installed token 26 - CHECK WAKEUP
: js_seqsql installed token 27 - VERIFY CALLOUT
Sequencer code installed in js_seqsql ...
Created sp_js_check_id
Created sp_js_dates2bits
Created sp_js_bits2dates
Created sp_js_days2bits
Created sp_js_bits2days
Created sp_js_getkey
Created sp_js_putkey
Created sp_js_checksched
Created sp_js_properties
Created sp_createjob
Created sp_createschedule
Created sp_createscheduledjob
Created sp_bindscheduledjob
Created sp_unbindscheduledjob
Created sp_addjobtext
Created sp_dropjobtext
Created sp_sjobmodify
Created sp_sjobcreate
Created sp_sjobcmd
Created sp_sjobcopy
Created sp_sjobcontrol
Created sp_sjobsetstatus
Created sp_sjobdrop
Created sp_sjobhelp
Created sp_sjobhistory
Created sp_modifyscheduledjob
Created sp_modifyschedule
Created sp_dropscheduledjob
Created sp_dropjob
Created sp_dropschedule
Created sp_copyjob
Created sp_copyschedule
Created sp_listscheduledjob
Created sp_listjob
Created sp_listschedule
Created sp_listjobtext
Created sp_controljob
Created sp_js_agent
Created sp_listjobhistory
Created sp_dropjobhistory
Created sp_dropjoboutput
Created sp_readjoboutput
Created sp_writejoboutput
Created sp_createjobtemplate
Created sp_modifyjobtemplate
Created sp_readjobtemplate
Created sp_writejobtemplate
Created sp_dropjobtemplate
Created sp_listjobtemplate
Created sp_sjobclean
Created sp_js_history_thresh
Database data is 86016 pages, installing free data threshold 8601 pages,
sp_js_history_thresh
Adding threshold for segment 'default' at '8601' pages.
DBCC execution completed. If DBCC printed error messages, contact a user with
System Administrator (SA) role.
(return status = 0)
Loading of sybmgmtdb database is complete.
(return status = 0)
sybase@asece1:/sybase/ASE-15_0/scripts>

Install a XP Server

> connect
mycluster> create xpserver
Enter the XP server port number for instance "ase1": 19798
Enter the XP server port number for instance "ase2": 19798
The XP Server was successfully defined for each instance.
mycluster>

Install a Monitor Server

> connect
mycluster> create monitorserver
Enter the Monitor server log file location for instance "ase1": [ /sybase/ASE-15_0/install/ase1_MS.log ]
Enter the Monitor server port number for instance "ase1": 19797
Enter the Monitor server log file location for instance "ase2": [ /sybase/ASE-15_0/install/ase2_MS.log ]
Enter the Monitor server port number for instance "ase2": 19797
Enter a username: sybmon
Enter a password:
mycluster>

Check The Interfaces File

After installing all the auxiliary servers, the interfaces file will look like this:

ase1
master tcp ether asece1.cluster.local 19786
query tcp ether asece1.cluster.local 19786

Test the cluster

Login to the cluster to execute a view commands.

> connect
mycluster> start cluster
.
.
Startup output messages
.
.
mycluster> show cluster status
INFO - Listening for the cluster heartbeat. This may take a minute. Please wait... (mycluster::AseProbe:434)

This is the easiest way to check on the status of the cluster.

** Cluster configuration for "mycluster" **
Interface Path "/sybase"
Trace Flags:
There were are no trace flags.
Maximum Instances "4"
Quorum "/dev/raw/raw2"
Master Device "/dev/raw/raw1"
logfile ase1 /sybase/ASE-15_0/install/ase1.log
run_parameters ase1
logfile ase2 /sybase/ASE-15_0/install/ase2.log
run_parameters ase2
Primary Interconnect "udp"
Server[1] ase1 asece1-ppriv 15100 15119
Server[2] ase2 asece2-ppriv 15120 15139
Secondary Interconnect "udp"
Server[1] ase1 asece1.spriv 15181 15200
Server[2] ase2 asece2.spriv 15201 15220
mycluster>

This completes the Sybase ASE 15 Cluster Edition installation examples. There is one more step to walk you through. This will explain how to add an additional node to an existing cluster.

Part 3 of the 5 part series on the step by step installation post focuses on installing the Sybase software. Part 1 has a list and links to the download pages for this software.

The 5 parts of this series are:

• Part 1: System Requirements and Test Setup Concept
• Part 2: Setup of the VMWare Server and the Linux Servers
• Part 3: Install of the Sybase Software
• Part 4: Setup and test the Sybase Cluster
• Part 5: Add a additional node to the cluster (planned release 12/30/2009)

The instructions designed that non-Sybase DBA can perform these tasks. Sybase DBA will find that the steps are very similar to your standard ASE installation.

Please click here to read how to setup the Linux environment.

Sybase Software Installation

System preparation

Sybase uses a system SySAM to manage the licensing. This is no important during the installation process. The server wil lstart with a temporary licenses, but if this step gets omitted the server will all a sudden stop working.

Please follow the SySAM installation steps in the installation guide.

User Account Setup

Before installing the software into the NFS share $SYBASE make sure that both nodes asece1 and asece2 have the user sybase and the group sybase added to the user accounts. This is important to manage the NFS shares. For simplicity add the sybase user the NFS server asecenfs too.

groupadd sybase
useradd -u 1000 -g sybase sybase
passwd sybase

This is the .bashrc of the sybase user.

# Sample .bashrc for SuSE Linux
# Copyright (c) SuSE GmbH Nuernberg
test -s ~/.alias && . ~/.alias || true
export SYBASE=/sybase
. $SYBASE/SYBASE.sh

Network Setup

To make it easier to navigate through all these settings a simple naming and network address pattern has been chosen. Nothing fancy, but it does the trick.

asece1:
hostname: asece1.localhost.org
IP Address eth0: 192.168.1.211 (public address)
Default Gateway eth0: 192.168.1.1 (public address)
IP Address eth1: 192.168.2.211 (primary private address)
Default Gateway eth1: none
IP Address eth2: 192.168.3.211 (secondary private address)
Default Gateway eth2: none
asece2:
hostname: asece2.localhost.org
IP Address eth0: 192.168.1.212 (public address)
Default Gateway eth0: 192.168.1.1 (public address)
IP Address eth1: 192.168.2.212 (primary private address)
Default Gateway eth1: none
IP Address eth2: 192.168.3.212 (secondary private address)
Default Gateway eth2: none

It is important to include both nodes, plus the nfs server in all /etc/hosts files. You must also configure the public, primary private and secondary private ip address space.

The /etc/hosts file looks like this:

#
# hosts This file describes a number of hostname-to-address
# mappings for the TCP/IP subsystem. It is mostly
# used at boot time, when no name servers are running.
# On small systems, this file can be used instead of a
# "named" name server.
# Syntax:
#
# IP-Address Full-Qualified-Hostname Short-Hostname
#
127.0.0.1 localhost
# special IPv6 addresses
::1 localhost ipv6-localhost ipv6-loopback
fe00::0 ipv6-localnet
ff00::0 ipv6-mcastprefix
ff02::1 ipv6-allnodes
ff02::2 ipv6-allrouters
ff02::3 ipv6-allhosts
# Public IP Addresses
192.168.1.210 asecenfs.localhost.org asecenfs
192.168.1.211 asece1.localhost.org asece1
192.168.1.212 asece2.localhost.org asece2
# Primary Private Network
192.168.2.211 asece1-ppriv.localhost.org asece1-ppriv
192.168.2.212 asece2-ppriv.localhost.org asece2-ppriv
# Secondary Private Network
192.168.3.211 asece1-spriv.localhost.org asece1-spriv
192.168.3.212 asece2-spriv.localhost.org asece2-spriv

NFS setup

The NFS configuration enables sharing the files between the nodes. For test purposes one of the nodes could have been used as the NFS server, but this would not be a close to reality scenario. Having a dedicated NFS server is just good practice. A better practice would be a clustered NFS server, but that’s overkill for this purpose.

This is the NFS configuration for the Sybase Cluster Edition test configuration:

On the NFS server the shared directories for the files need to be created.

mkdir /sybase

Add the following lines to the /etc/exports file.

/sybase *(rw,sync,no_wdelay,insecure_locks,no_root_squash)

Start the NFS services on the NFS server.

/sbin/chkconfig nfs on
/sbin/service nfs restart

This is very important. Because of the VMware environment the NFS setup is not very stable. At least not on my system. Before starting the Sybase nodes, the status of the NFS server has to be confirmed. Executing the commands above on the NFS server will prevent data corruption on one of the Sybase nodes due to a NFS issue.

On each Sybase nodes, asece1 and asece2 the local folder structure has to be established and stamped with the correct user permissions.

mkdir -p /sybase
chown -R sybase:sybase /sybase

On each Sybase nodes, asece1 and asece2, add the following lines to the /etc/fstab file to automatically mount the NFS shares.

asecenfs:/sybase /sybase nfs rw,bg,hard,nointr,tcp,vers=3,timeo=300,rsize=32768,wsize=32768,actimeo=0 0 0

To get immediate access to the NFS shares without rebooting the nodes, manually mount the NFS shares on both servers.

mount /sybase

Raw Device Setup

In this setup 8 raw devices are created.

/dev/sdb master device 300 Mb
/dev/sdc quorum device 20 Gb
/dev/sdd data device 10 Gb
/dev/sde sybsystemprocs device 200 Mb
/dev/sdf system device 100 Mb
/dev/sdg sysmgmt device 200 Mb
/dev/sdh tempdb asece1 1 Gb
/dev/sdi tempdb asece2 1 Gb

For all 8 (sdb – sdi) devices repeat the following fdisk command sequence. Run this command on asece1 only. When following the example simply replace the device name.

asece1:~ # fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-200, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-200, default 200):
Using default value 200
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
asece1:~ #

Add the following entries into the file /etc/raw at both nodes asece1 and asece2.

asece1:/etc # cat raw
# /etc/raw
#
# sample configuration to bind raw devices
# to block devices
#
# The format of this file is:
# raw<N>:<blockdev>
#
# example:
# ---------
# raw1:hdb1
#
# this means: bind /dev/raw/raw1 to /dev/hdb1
#
# ...
raw1:sdb1
raw2:sdc1
raw3:sdd1
raw4:sde1
raw5:sdf1
raw6:sdg1
raw7:sdh1
raw8:sdi1
asece1:/etc #

Execute the command below to create the raw devices at both nodes asece1 and asece2.

asece1:/etc # cd /etc/init.d
asece1:/etc/init.d # sh raw start
bind /dev/raw/raw1 to /dev/sdb1... done
bind /dev/raw/raw2 to /dev/sdc1... done
bind /dev/raw/raw3 to /dev/sdd1... done
bind /dev/raw/raw4 to /dev/sde1... done
bind /dev/raw/raw5 to /dev/sdf1... done
bind /dev/raw/raw6 to /dev/sdg1... done
bind /dev/raw/raw7 to /dev/sdh1... done
bind /dev/raw/raw8 to /dev/sdi1... done
asece1:/etc/init.d # /sbin/chkconfig raw on
asece1:/etc/init.d #

Make sure that the user sybase owns the raw devices at both nodes asece1 and asece2.

asece1:/etc/init.d # cd /dev/raw
asece1:/dev/raw # chown sybase raw[1-8]
asece1:/dev/raw # ls -al
total 0
drwxr-xr-x 2 root root 200 Nov 25 13:56 .
drwxr-xr-x 10 root root 7320 Nov 25 13:56 ..
crw-rw---- 1 sybase disk 162, 1 Nov 25 13:56 raw1
crw-rw---- 1 sybase disk 162, 2 Nov 25 13:56 raw2
crw-rw---- 1 sybase disk 162, 3 Nov 25 13:56 raw3
crw-rw---- 1 sybase disk 162, 4 Nov 25 13:56 raw4
crw-rw---- 1 sybase disk 162, 5 Nov 25 13:56 raw5
crw-rw---- 1 sybase disk 162, 6 Nov 25 13:56 raw6
crw-rw---- 1 sybase disk 162, 7 Nov 25 13:56 raw7
crw-rw---- 1 sybase disk 162, 8 Nov 25 13:56 raw8
crw-rw---- 1 root disk 162, 0 Nov 25 13:56 rawctl
asece1:/dev/raw #

The Linux system creates these raw devices as root user at every boot. There should be a start-up command added that will change permission automatically. If the owner of these raw devices is not sybase the database start-up fails.

Shared Memory Setup

The Linux kernel needs just one modification.

The file /etc/sysctl.conf needs the following line added.

kernel.shmmax=1073741824

This will set the shared memory to 1GB.

To activate the settings execute the following command:

/sbin/sysctl -p

Install Sybase ASE Cluster Edition software

As the sybase user go to the folder where the Sybase ASE Cluster Edition software is staged and execute the setup command. This will install the software into the $SYBASE directory. This command needs to be executed only once, on asece1.

./setup
InstallShield Wizard
Initializing InstallShield Wizard...
Searching for Java(tm) Virtual Machine...
........
Running InstallShield Wizard...

The installer will start and you will see this screen:

Hit next to start the installation process.

Select the country and accept the license agreement. Then click next.

Make sure that you select the directory that corresponds with your $SYBASE setting. Hit next to continue.

Select the full installation. Then hit Next.

We don’t want to setup the license server. Select No and hit Next.

Then you will see this screen.

Hot Ok to continue.

We don’t want to setup the mail services. Select No and hit Next.

I believe that’s the correct license setting. Hit Next to continue.

That’s the list of what will be installed. Review the list and hit Next to start the software install.

This is the progress screen of the software install.

The installer confirms that the software is installed. Hit Next to continue.

This is the last screen of the software installation. Make sure it reads successful and the hit Finish to close the installer.

Important: Before you continue make sure that the SySAM license is copied to the $SYBASE/SYSAM-2_0/licenses directory.

This completes the Sybase installation. Next you will see how to setup your cluster. Please click here to read the next part.

Part 2 of the 5 part series on the step by step installation post focuses on the setup of VMWare Server and the SuSe Linux system. Part 1 has a list and links to the download pages for this software.

The 5 parts of this series are:

• Part 1: System Requirements and Test Setup Concept
• Part 2: Setup of the VMWare Server and the Linux Servers
• Part 3: Install of the Sybase Software
• Part 4: Setup and test the Sybase Cluster
• Part 5: Add a additional node to the cluster (planned release 12/30/2009)

The instructions designed that non Linux admins can easily reproduce the steps. Any tuning, hardening, high-availability and other Unix management tasks have been omitted for simplicity.

Please click here to read how to setup the VMWare environment.

Linux Setup

This post will walk you through the Linux setup of our 3 virtual machines that have been setup in Part 2 of the Sybase ASE Cluster Edition Step-By-Step Installation. These posts are huge and for better readability they have been split in several parts. All parts are linked together. So you will always find the start of the series as well as each individual part.

Please click here to read how to setup the virtual machines. To start the series from the first post, please click here.

Setting up the Linux systems is a fully menu driven approach and requires mounting of the ISO file first to load the boot disk to the virtual machines.

First we need to add a new datastore to the VMWare Server to allow access to the location where the ISO files are saved. This is most likely on your desktop or laptop and very easy to do. In this example the new datastore is located on a network drive. It takes a little bit more parameters to configure.

Click on the icon next to the CD/DVD Drive label and select edit.

Select the ISO Image option for the CD mounts and click on Browse.

Select the SuSe Linux ISO file and click OK. Then click OK again to complete the mount process.

These are the steps to mount an ISO file as CD drive to any virtual machine. Now you can repeat these steps to mount the ISO file to the other virtual machines. Unlike a physical CD, you can mount the same ISO file to many virtual machines at the same time.
The test environment diagram outlines that there will be 3 Linux server installed; asece1, asece2 and asecenfs. The next steps will illustrated how to install the base Linux system on asecenfs. The only difference between this server and asece1 and 2 are the 3 network cards used and the shared disks.
These are the network addresses used for the 3 servers.

# asecenfs
192.168.1.210   	asecenfs.localhost.org asecenfs

# asece1
192.168.1.211   	asece1.localhost.org asece1
192.168.2.211 	asece1-ppriv.localhost.org asece1-ppriv
192.168.3.211 	asece1-spriv.localhost.org asece1-spriv

# asece2
192.168.1.212 	asece2.localhost.org asece2
192.168.2.212 	asece2-ppriv.localhost.org asece2-ppriv
192.168.3.212 	asece2-spriv.localhost.org asece2-spriv

This information is crucial when configuring the network interfaces during the installation process.

Booting the Install Disk
Now it is time to boot up the virtual machine. Click on the console tab of the virtual machine you want to start. In this case we will start the ASECENFS server and then click anywhere in the black screen to boot the system. If you don’t see the big play symbol, you need to install the software. Just follow the instructions on the screen. Once completed, you will come back to this screen.

Once the power on process is complete you want to open to console to see console in a new window. It is important that you take control over the console as soon as possible. SuSe Linux has a default menu that shows up when booted the first time. Just click anywhere in the console window and move the cursor down to select “Installation”. This will stop the timer.

Now hit enter to start the installation process.

This is the first screen of the installation for SuSe Linux. Selecting the language then hit Next. On the left hand menu you will see the progress of the installation. Like the VMWare virtual machine setup, you always know which step you’re performing and how many more steps to go.

Accept the license agreement and click Next. If you don’t want to accept the license agreement, then click Abort. In this case you need to find an alternative way to install an active Linux system onto your virtual servers.

Skip the media check and click Next

Select the “New Installation” option, which is the only one available and hit Next.

Chose your Clock and Timezone settings and hit Next.

Now you will see a summary of the selection. If you are a Linux savvy admin, you could go ahead and click on the Expert tab to further fine tune the selection. But for this example we simply accept the default and let the system install all the necessary parts.

This is your last chance to abort this installation. Just hit install and the system will install SuSe Linux onto the virtual machine. This can take a while.

The system will reboot itself and after all is set and done, you will see this screen.

Just enter the root password and click on Next.

The next screen asks for the hostname and domain. This example creates asecenfs, therefore the hostname is asecenfs and for simplicity the domain is localhost.org, which is a fictive domain name.
DHCP is a big no-no in a cluster environment. Every time you see the word DHCP, make sure the checkbox next to it is unchecked.

Changing the network settings is important. Not only do you need to turn off any DHCP settings, but you also need to turn off any firewall settings. Both are turned on by default. On the next screen will click on the word enabled next to the word firewall and the firewall is turned off.

Now you need to configure the network. Just click on the Network Interfaces link.

Click Edit at the next screen.

Now you are in the network configuration mode. To make it all work the network will be configured with static ip addresses.

There are two more settings to either check or change. Add the appropriate name server and routing address for your network.

After clicking Next a couple of times you will end up at the entry screen again and it should look something like this.

After passing the network test you will see this screen. Don do anything with it for now. Select “Configure Later” and click Next.

Skip the service configuration.

Use local user configuration.

Create the Sybase user.

Click Next on the release notes and then the system checks the hardware configuration. The screen will flicker and resize several times. Once this process is completed, click Next on the screen below.

Congratulations!
You just installed your first SuSe Linux system. This wasn’t too hard? Click on Finish to exit the installation program.

Now it is time to configure the 2 main cluster nodes. Follow the steps outlined above to get the SuSe Linux software installed and the base system configured.
The main differences will be with the network interfaces. The following screens will illustrated the differences.
asece1
Now it is time to address the 3 network connections for asece1. Here are the ip addresses again.

# asece1
192.168.1.211   	asece1.localhost.org asece1
192.168.2.211 	asece1-ppriv.localhost.org asece1-ppriv
192.168.3.211 	asece1-spriv.localhost.org asece1-spriv

Click on Network Interfaces to configure all 3 networks at once.

Select the first network to configure the public network and click on edit.

Add the ip address and select the static address setup option.

Set the name server and the routing address once done click on Next.

Now go ahead and select the second network from the list. This will be the primary private network that doesn’t share the subnet with anybody else. No name server or routing will be selected.
Here’s the ip setup screen.

Set the routing then repeat the same steps for the last network, which will be the secondary private network.

asece2

When you power on this server you will notice a new window asking you if you copied or moved this virtual machine. This is because you copied ascece1 to create asece2. Select “I copied it” and continue.

Repeat the same steps you already did for asece1 and asecenfs.
These are the ip addresses used for asece2.

# asece2
192.168.1.212 asece2.localhost.org asece2
192.168.2.212 asece2-ppriv.localhost.org asece2-ppriv
192.168.3.212 asece2-spriv.localhost.org asece2-spriv

Now all Linux servers for the Sybase cluster are set-up and ready to go.

The next step is to setup Sybase in part 3 of the series. Click here to read part 3 and click here to start all over from the beginning.