Disaster Recovery Procedures:

Several web sites and or web applications are hosted on or redirected from //apps <10.10.5.5/24>. The //apps server has two separate RAID1 system and application partitions. The web and application environments as well as the mysql databases are mirrored and or backed up to three locations.

1. //vapps <192.168.2.239/24>: The //apps LAMP environment and system configuration is mirrored to //vapps daily at 03:37 AM by a script running as datbak@vapps, script path: '//vapps/home/datbak/script/sync_apps.sh'. In case of an //apps system failure this VM could be configured to to assume it's web hosting responsibilities by completing the following steps, which require root privileges:

a. On the VM, edit the file '/apps/www/portal/public_html/sites/default/settings.php', change the line "$base_url = 'http://testportal.modeldriven.org';" to read "$base_url = 'http://portal.modeldriven.org';", minus the double quote marks.

b. Also on the VM, comment out the clone script by editing the cron job for datbak: 'su - datbak'->'crontab -e'->add a pound character to the beginning of this line as shown '#37 03 * * * ~/script/sync_apps.sh > /dev/null 2>&1'->save ':wq' + <ENTER>.

c. Edit the 'hosts' and 'HOSTNAME' files, this is necessary for sending e-mail to some domains: 'vi /etc/hosts'->change the line '127.0.0.2 vapps.datcentral.local vapps' to '127.0.0.2 apps.modeldriven.org apps' minus the quote marks->also change the line '192.168.2.239 vapps' to read '192.168.2.239 apps' again minus the quote marks->save ':wq'+<ENTER>->vi /etc/HOSTNAME->change the line 'vapps.datcentral.local' to read 'apps.modeldriven.org' minus the quote marks->save ':wq' + <ENTER>->restart system->'shutdown -r now'.

d. Change the NAT mapping on the Fortinet 100a firewall: Fortgate 100a->Firewall->Virtual IP->MDWiki->Edit->Mapped IP Address/Range->192.168.2.239->Ok.

2. //apps <10.10.5.5/24> & //devbackup.enterprisecomponent.com <70.184.241.162/0>: All portal content as well as other hosted applications and system data are also backed up to a USB hardrive mounted on //apps by another script running as datbak@apps everyday at 12:33 AM, script path: '//apps/home/datbak/script/bakupapps.sh', backup path: //apps//usbhd0p1/appbakup'. Additionally, this script will perform an offiste backup job copy on Sundays to the following location: //devbackup.enterprisecomponent.com/BAKUPSTORE/APPSBAKUP.

Subversion:

The main source repository is located on the //dev system, internal IP: <10.10.3.2/24>, external IP: <70.182.188.5/0>. Currently, it is being backed up and or mirrored to three different locations. The combined system and application partitions are in RAID5 array configuration with a standby hot spare drive available in case of an array member drive failure.

1. //dev-clone <192.168.2.28/24>: This is a SuSE linux VM running on //buildhost <192.168.2.230/24>, which is a clone of //dev both in system configuration and subversion repositories. The cloning procedure is accomplished by a script runnng as datbak on the VM, it runs daily at 1:07 AM, script path: '//dev-clone/home/datbak/script/svn_backup2.sh'. In case of a //dev system failure //dev-clone can be configured to take over dev's functionality by changing the firewall NAT mapping. Fortgate 100a->Firewall->Virtual IP->DEV->Edit->Mapped IP Address/Range->192.168.2.28->Ok

2. //devbackup.enterprisecomponent.com <70.184.241.162/0>: This remote system is setup as a read only SVN mirror of //dev and is currently only accessible from inside the MDS internal network. Its purpose is to provide an offsite copy of the production repositories in case of //dev system failure or catastrophic environmental failures such as building fire or theft. The mirroring process utilizes SVN's native mirroring utility 'svnsync' and is initiated by a script which runs daily at 11:47 PM as the user datkbak@dev, script path: '//dev/home/datbak/script/svn_sync.sh'. The read only repository is accessible using the following URL https://devbackup.enterprisecomponent.com/repository/ from inside MDS' internal network.

3. //devbackup.enterprisecomponent.com <70.184.241.162/0>: The source repositories are also backed up remotely to a physical drive separate from the one housing the mirrored repositories in the //devbackup system . This backup only contains a copy of the source repositories and is not accessible via the web. The backup job runs daily at 12:33 AM as the user datbak@dev using a partial dump, SCP transfer and load strategy, script path: '//dev/home/datbak/script/svn_backup.sh', backup path: '//devbackup.enterprisecomponent.com/BAKUPSTORE/svnroot'.

4. //demo <192.168.2.215/24>: This system does not host any of the actual backup processes for dev's source repositories but, does process two important monitoring scripts, which provide a SVN 'youngest' revsion report to check the subversion backup status on the various systems and a file space useage report to validate backup space. Both scripts run daily as datbak@demo at 01:03 PM and 06:03 AM respectfully. The corresponding reports are sent to the sysadmin@modeldriven.com distribution list. Respective script paths: '//demo/home/datbak/script/revisionrpt.sh', '//demo/home/datbak/script/chkspace.pl'.

revsion report example:

dev source repositories: //10.10.3.2/svnroot/repository/
GSA     MDLibrary       MDS     projects        repos
18806   19812           18593   18233           18720
dev mirror repositories: //devbackup.enterprisecomponent.com/svnroot/repository/
GSA     MDLibrary       MDS     projects        repos
18806   19810           18589   18233           18718
dev backup repositories: //devbackup.enterprisecomponent.com/BAKUPSTORE/svnroot/repository/
GSA     MDLibrary       MDS     projects        repos
18806   19810           18589   18233           18718
dev clone repositories: //192.168.2.28/svnroot/repository/
GSA     MDLibrary       MDS     projects        repos
18806   19810           18589   18233           18718
demo source repositories: //192.168.2.215/svnroot/repository/
bin
20
demo backup repositories: //10.10.5.5/usbhd0p2/demobakup/repository/
bin
20

file space report example:

The file space of usage the following systems was found to be within acceptable limits: < 90%

devbackup : <devbackup.enterprisecomponent.com>

dev : <10.10.3.2>

demo : <192.168.2.215>

apps : <10.10.5.5>

dev-clone : <192.168.2.28>

modelpro_ci : <192.168.2.19>

buildhost : <192.168.2.230>

vapps : <192.168.2.239>

The Large/Binary file SVN repository is hosted on //demo <192.168.2.215/24> and is only accessible on the LAN or with a VPN connection using the same login credentials as //dev though the following URL: http://192.168.2.215/repository/ . This repository is backed up as necessary to a USB hard drive mounted on //apps <10.10.5.5/24> by a script running as datbak@demo at 11:03 PM, script path: '//demo/home/datbak/script/svnbackup.sh', backup path: '//apps/usbhd0p2/demobakup/repository/bin'. The svn access password files are copied via the rsync command daily at 5:17 PM by another script running as datbak@demo, script path: '//demo/home/datbak/script/htpasswd_sync.sh'.

ModelDriven.org Portal:

The ModelDriven.org portal running on //apps internal IP: <10.10.5.5/24>, external IP: <70.182.188.7/0> is also backed up daily at multiple intervals by a separate system level script.

1. //apps <10.10.5.5/24>: All portal content including files, MySQL data, apache and LAMP configuratrions is backed up to a USB hard drive mounted on //apps by a script running as datbak@apps six times a day at 04:03 AM, 10:03 AM, 04:03 PM and 10:03 PM, script path: '//apps/home/datbak/script/mdobakup.sh', backup path: //apps/usbhd0p2/mdobakup'.

MS Exchange:

The Windows Exchange server \\DATCENTRALSRV has it's mailbox store backed up daily to three different locations both local and remote.

\\DATCENTRALSRV <10.10.2.3/24>: The Exchange mailbox store is backed up daily to both local and remote file shares by a script running as the local administrator, script path: '\\DATCENTRALSVR\K$\ExchangeDailyBackups\script\winbackup.pl'. The perl script, when launched by the Windows scheduler, executes the NT Backup utility with shadow copy enabled to accomplish this. Full backups occur on Fridays at 11:33 PM and differential backups are run on Saturday though Thurday at 10:17 PM. All differential backup sets are deleted upon sucessful completion of Firday's full backup job. The resulting backup files '*.bkf' are zipped and or copied to the following locations.

1. \\DATCENTRALSRV\K$\ExchangeDailyBackups\: A local Windows file share on the system.

2. \\DATCENTRALSRV\G$\Datcentral\Exch_Backups\: A USB mounted hard drive on the local system.

3. //devbackup.enterprisecomponent.com/BAKUPSTORE/DATCENTRAL/: A remote UNIX file system.

Sugar CRM & TimeTracker:

The Sugar Customer Relationship Manager and local TimeTracker LAMP applications are hosted on //demo <192.168.2.215/24>. The //demo system has a RAID1 system partition and a separate RAID5 application partition.

//demo <192.168.2.215/24>: The backups of file, mysql and system configuration data is completed daily by a script running as datbak@demo at 02:37 AM. The locally created backup tar files are subsequently moved to a USB hard drive mounted on //apps <10.10.5.5/24> using the 'rsync' command, script path: '//demo/home/datbak/script/file-bakup.sh', backup path: //apps/usbhd0p2/demobakup/file-bakup'.

Virtual Machines:

There are two production VMware v1.0.8 servers, //demo <192.168.2.215/24> & //buildhost <192.168.2.230/24>, and one experimental VMware ESX-3i server, //esx3i-1 <192.168.2.176/24>. Both the //buildhost and //esx3i-1 systems are capable of hosting 64 bit guest OS', the //demo system can only host 32 bit guests. The //demo system has a RAID1 system partition and a separate RAID5 application partition. The //buildhost system has a non RAID system partition and a RAID0 application patition, both //buildhost and //esx3i-1 do not have any redundant, hard drive, failover capability.

//demo <192.168.2.215>: A script running as dakbak@demo backs up running and shutdown VMs on both the //demo and //buildhost systems to a locally mounted USB hard drive, it is currently in development and not scheduled, script path: '//demo/datbak/script/vmsbakup.pl', backup path: '//demo/vmsbakup/192.168.2.215' & '//demo/vmsbakup/192.168.2.230'. Because of the different VMware API implementations this script will not currently work on either VMware server v2.0.x or the ESX-3i server versions. The original intent was to upgrade both production systems to VMware Server 2.0.x which has similiar API fuctionality to the ESX-3i version but, this release proved to be unstable.

System Level Backups:

//apps <10.10.5.5/24>: Refer to section: 'ModelDriven.org Portal:, 3.'

//demo <192.168.2.215/24>: Refer to section: 'Sugar CRM & TimeTracker:, //demo'

//dev <10.10.3.2/24>: A scrpt running as datbak@demo performs a daily file level backup of //dev at 04:03 AM and copies it to a USB hard drive mounted on //apps. This backup job excludes the SVN repositories, script path: '//demo/home/datbak/script/dev-file-bakup.sh', backup path: '//apps/usbhd0p1/devbakup'.

\\DATCENTRALSRV <10.10.2.3/24>: The bkprunner.exe utility is executed daily by the Windows scheduler at 02:30 AM, backup location: '\\DATCENTRALSRV\G$\Datcentral\Backup Files'.

Latest News

Disaster Recovery Procedures: