SVN backup script

Here something to drop to an SVN server:

#!/bin/bash
SUFIX=`date +%Y%m%d`
DIR_BASE=/tmp/`hostname`_svn_
DIR_WRK=$DIR_BASE$SUFIX
DIR_REPOS=/path/to/a/directory/with/svn/repositories
REPOS="SVNRepo1 SVNRepo2"
STORAGE=backup@int.storage.server:/path/to/backup/location

# -- init working directory
echo "`date` - Preparing backup directory"
rm -rf $DIR_BASE*
mkdir -p $DIR_WRK
echo "`date` - Changing into backup directory '$DIR_WRK'"
cd $DIR_WRK

for repo in $REPOS
do
 path=$DIR_REPOS$repo
 if [ -d $path ]; then
  mkdir $repo
  echo "`date` - Backing up '$repo' to '$repo/repository.dump'"
  svnadmin dump $path > $repo/repository.dump
  cp $path/access-policy $repo/.
  cp $path/svn-auth-file $repo/.
  echo "`date` - Compressing '$repo' to '$repo.tgz'"
                tar -czf $repo.tgz $repo
                echo "`date` - Deleting '$repo'"
                rm -rf $repo
 else
  echo "`date` - Repository '$path' does not exist"
  continue
 fi
done

# -- transfer to a safe place
echo "`date` - Transfering backup directory to a remote location"
rsync -re ssh $DIR_WRK $STORAGE

echo "`date` - Cleaning up backup directory"
rm -rf $DIR_WRK
echo "`date` - Done"

MySQL backup script

Nothing extraordinary, just a backup script which runs on our production db servers each night:

#!/bin/bash
SUFIX=`date +%Y%m%d`
DIR_BASE=/tmp/`hostname`_db_
DIR_WRK=$DIR_BASE$SUFIX
DB_USER=backup
DB_PASSWD=abcde
STORAGE=backup@int.storage.server:/path/to/backup/location

# -- init working directory
echo "`date` - Preparing backup directory"
rm -rf $DIR_BASE*
mkdir -p $DIR_WRK
echo "`date` - Changing into backup directory '$DIR_WRK'"
cd $DIR_WRK

# -- backup all databases
for db in `mysql -u $DB_USER -p$DB_PASSWD -e 'SHOW DATABASES;'`
do
case $db in
 Database|information_schema) continue;;
 *)
  echo "`date` - Backing up '$db' to '$db.sql'"
  mysqldump -u $DB_USER -p$DB_PASSWD --opt $db > $db.sql
  echo "`date` - Compressing '$db.sql' to '$db.tgz'"
  tar -czf $db.tgz $db.sql
  echo "`date` - Deleting '$db.sql'"
  rm $db.sql
  ;;
esac
done

# -- transfer to a safe place
echo "`date` - Transfering backup directory to a remote location"
rsync -re ssh $DIR_WRK $STORAGE

echo "`date` - Cleaning up backup directory"
rm -rf $DIR_WRK
echo "`date` - Done"

Automatic deployment part 5

Here is the current version of the build script:

<?xml version="1.0" encoding="UTF-8"?>
<project name="my_project" default="deploy">
<property name="config.file" value="builder/build.properties.${dest}" />
<property name="dist.dir" value="${ant.project.name}-r${svn_revision}-b${build_number}" />
<property name="build.dir" value="${workspace}/${dist.dir}" />

<record name="${build.dir}.log" />

<target name="init">
<available file="${config.file}" property="config.exists"/>
<fail message="Config file '${config.file}' not found." unless="config.exists" />
<echo>Config file '${config.file}' exists.</echo>
<property file="${config.file}" />

<available file="${workspace}" property="workspace.exists"/>
<fail message="Workspace '${workspace}' not found." unless="workspace.exists" />
<echo>Workspace '${workspace}' exists.</echo>
<delete dir="${build.dir}" />
<mkdir dir="${build.dir}" />
</target>

<target name="prepare" depends="init">
<copy todir="${build.dir}">
<fileset dir=".">
<exclude name="**/.*" />
<exclude name="build*" />
<exclude name="bkp/" />
<exclude name="builder/" />
<exclude name="cache/" />
<exclude name="scripts/" />
<exclude name="sources/" />
</fileset>
</copy>
<copy file="builder/.htaccess.${dest}" tofile="${build.dir}/.htaccess" verbose="true" />
</target>

<target name="transfer" depends="prepare">
<scp todir="${dest_user}@${dest_host}:${dest_path}/" password="${dest_passwd}" trust="true">
<fileset dir="${workspace}">
<include name="${dist.dir}/"/>
</fileset>
</scp>
</target>

<target name="remote-deploy" depends="transfer">
<sshexec
host="${dest_host}"
username="${dest_user}"
password="${dest_passwd}"
command="
cd ${dest_path};
rm current;
ln -s ${dist.dir} current;"
/>
</target>

<target name="deploy" depends="remote-deploy">
</target>
<project>

The script needs to be invoked like this:
ant -Dworkspace=/tmp -Ddest=dev -Dsvn_revision=123 -Dbuild_number=123

Automatic deployment part 4

The standard installation of Hudson or Ubuntu does not have all the optional tasks and dependencies. In manage Hudson - configure - I had to create another installation of ant. Then I changed configuration of the project to use that specific installation of the Ant.

On next build Hudson downloaded and installed Ant. ...Which again didn't contain optional tasks. I used command line to get to the Hudson installation directory and in there to the new Ant installation. over there as a hudson user (this is important) I ram ./bin/ant -f fetch.xml -Ddest=system

Then I reloaded configuration of Hudson through manage configuration. Since then I was able to use optional tasks as scp and sshexec.

Automatic deployment part 3

I finished writing the Ant build script for first of our PHP projects. I made plenty of tests. At the end script consists of four parts. I send several parameters to Ant. The most important is destination - devel, test, prod. For each destination I have separate configuration file. The Ant build script then does following:

1. Init - prepares build directory structure.
2. Prepare - copies all files to build directory. In this step an .htaccess for proper destination is copied at proper place in the build directory structure.
3. Transfer - transfers whole build directory to the destination.
4. Remote-deploy - executes series of commands over sshexec. Most of them creates links to newly uploaded build.

So far it works. I'd like to figure out dependencies on other projects as this project itself needs our framework....

The biggest problem for me was to figure out why copy target did not exclude certain directories. But after few hours I figured that out also.

Automatic deployment part 2

So I've installed a Hudson server on one of my virtual machines. It was a snap. Just followed the instructions.

I started to work on Ant build script for one of the projects and figured out following. Aptana Studio 2 doesn't have Ant build as I was used to in Eclipse. I downloaded Eclispe and installed all necessary components for me to find out that installation went bad. Snow Leopard's default Ant does not have optional tasks. No way I was able to enable them.

The solution I opted for was to use MacPorts with Porticus and istalled all Ant related features. After that I ran fetch.xml build script as instructed in installation manual and I have Ant 1.8.1 fully functional.

Automatic deployment part 1

I want to have a deployment process not to be manual anymore. After research I settled for Hudson as an deployment server and I am heading towards continuous integration. I am not sure at the moment if Subversion is righ CRM for the task and thinking of switching to Git. Deploy scripts are to be written in Ant.

The process as I think of is following. Programmers will work on their working copies of projects. Once they are satisfied with progress they commit to the trunk of CRM for the project. Hudson picks up the commit and starts deployment using Ant build script of the project to the testing server. Once we reach set of features and stability required, we mark build as staging ready and Hudson will pick it up and replay it to the staging server to see if the build on production will be functional. If so we mark the build as production build and let Hudson to replay it to the production environment.

There is many things which bothers me. Our projects are usually dependent on other projects of ours. One change in a DB backend required for one project affects few other projects. Deployment needs to account for that. Also tagging in Subversion seems to be a headache to put in symbiosis with Hudson.

VMware Tools and Ubuntu 9.10 problems

With update of VMware ESXi 4.0 to VMware Hypervisor 4.1 I had to reinstall all vmware-tools on all my Ubuntu 9.10 virtual machines. There was no way for me to figure out why it does run on one of them automagically and not on any other machine.

I googled the internet left and right and no help. So I decided to figure out the thing by myself. The thing is that there was a link missing for start-up script. So here is the whole thing needed to be done:

cd /etc/rcS.d
sudo ln -s ../init.d/vmware-tools S38vmware-tools

After that everything works OK again.

VMWare...

Recently I went from using stand alone machines to VMWare ESXi server. It gives me much more flexibility. We purchased HP ML350 with 16GB Ram and one processor so far. Will try to upgrade it soon.

Anyway, the first major thing was, I didn't want to install new machine from the scratch over and over. We have several Ubuntu servers and I just needed one install which I would reuse as a base for different servers.

The way to do it in ESXi is:

1. Prepare one virtual machine for cloning. In my case it was to make a basic installation of Ubuntu server. Don't forget to install VMWare Tools, so you don't have to do it for each of your clones later.
   
2. Then go to your datastare and make a directory for a new virtual machine. Go back to the directory of your Ubutu server installation from previous step. Copy all *.wmx and *.wmdk files over to your new directory.
3. When done go to the new directory, right click on *.wmx file and choose "Add to Inventory". Finish the wizard.
4. Start your new virtual machine.
5. Go to console and edit /etc/hostname file to have the name you want your virtual machine to have.
6. Go to console and edit /etc/hosts. Change hostname from the name of your cloned machine to the one form previous step.
   
7. Go to console and edit /etc/udev/rules.d/70-persistent-net.rules You just need to comment first line and change eth1 to eth0 in the second line.
8. In console edit /etc/network/interfaces and change IP of your machine.
   

After those steps your "cloned" virtual machine is ready. Now you can install whatever software you need for the particular case.