Deploying Python applications

I have to interrupt the ongoing series on Mercurial, Fabric, Liquibase and Kohana. There is stuff which I had to take care of immediately. However the problems I am solving right now are the ones I was writing recently anyway.

Mercurial, Fabric, Liquibase, Kohana (part 4)

Today I am focusing on making sure all database related stuff from previous parts can be executed by me and any of my teammates on their local machines using Fabric.

Mercurial, Fabric, Liquibase, Kohana (part 3)

My new application is going to require an authentication. I am not looking for anything fancy. For starters the ORM authentication available in Kohana 3.3 will do. I may extend it later. This means that I am going to use database to store user accounts. As I am a big fan of database code versioning I'll be using LiquiBase to do that. So how to go about it?

Mercurial, Fabric, Liquibase, Kohana (part 2)

I will be using Mercurial to keep track of changes to a project. I create a project directory and initialize a Mercurial repository in it:

shell> hg init

I do like to have an option to switch Kohana with new version easily in my projects. To achieve that I unpack downloaded Kohana into kohana directory of my project. I then copy application directory from kohana folder into root of my project, so it is on the same level with kohana directory. The same needs to be done with index.php and .htacess. Structure of my project looks like this after that:

project
  |- application
  |- kohana
  |- .htaccess
  |- index.php

Mercurial, Fabric, Liquibase, Kohana (part 1)

I'd like to to start writing about my usage of these tools, but inevitably I would have to write why I am using those specific tools. So not to return to this later I start with my reasons.

Database documentation

It is so frustrating to guess database structure intention without any information about it. Liquibase has a great feature, which helps to prevent a lot of guessing. Liquibase can generate an html documentation of db. However the documentation is pretty sparse. But be not frustrated, there is a light over there at the end of the tunnel...

Graphviz - dot source for my graph

For those who wandered what is the input file for my graph... The input files for graphviz are called "dot" files. Here is the input "dot" file for my graph from the last post:

Graghviz in action

Graphviz - data visualization

I had a problem to follow a logic in someones programming and I had to deliver... I had to come up with a solution to push the brick forward. I started to look for a visualization tool for my problem. And to my surprise I found a graphviz - a tool for data visualization.

After some time I was able to come up with this:

Graphviz in action

LiquiBase - the way to handle a database

I am in a middle of rework of an old script. Whole structure of a solution was cumbersome a lot so I had to step in to correct the stuff. I knew that the script has some data in a MySQL database. It uses few tables in a large db with tens and tens of other tables.

Long time no ... write....

I see I didn't put a link over here for almost two years. Yeah, a lot changed since my last update... I'll start to cover it soon topic by topic.

SVN backup script

Here something to drop to an SVN server:

#!/bin/bash
SUFIX=`date +%Y%m%d`
DIR_BASE=/tmp/`hostname`_svn_
DIR_WRK=$DIR_BASE$SUFIX
DIR_REPOS=/path/to/a/directory/with/svn/repositories
REPOS="SVNRepo1 SVNRepo2"
STORAGE=backup@int.storage.server:/path/to/backup/location

# -- init working directory
echo "`date` - Preparing backup directory"
rm -rf $DIR_BASE*
mkdir -p $DIR_WRK
echo "`date` - Changing into backup directory '$DIR_WRK'"
cd $DIR_WRK

for repo in $REPOS
do
 path=$DIR_REPOS$repo
 if [ -d $path ]; then
  mkdir $repo
  echo "`date` - Backing up '$repo' to '$repo/repository.dump'"
  svnadmin dump $path > $repo/repository.dump
  cp $path/access-policy $repo/.
  cp $path/svn-auth-file $repo/.
  echo "`date` - Compressing '$repo' to '$repo.tgz'"
                tar -czf $repo.tgz $repo
                echo "`date` - Deleting '$repo'"
                rm -rf $repo
 else
  echo "`date` - Repository '$path' does not exist"
  continue
 fi
done

# -- transfer to a safe place
echo "`date` - Transfering backup directory to a remote location"
rsync -re ssh $DIR_WRK $STORAGE

echo "`date` - Cleaning up backup directory"
rm -rf $DIR_WRK
echo "`date` - Done"

MySQL backup script

Nothing extraordinary, just a backup script which runs on our production db servers each night:

#!/bin/bash
SUFIX=`date +%Y%m%d`
DIR_BASE=/tmp/`hostname`_db_
DIR_WRK=$DIR_BASE$SUFIX
DB_USER=backup
DB_PASSWD=abcde
STORAGE=backup@int.storage.server:/path/to/backup/location

# -- init working directory
echo "`date` - Preparing backup directory"
rm -rf $DIR_BASE*
mkdir -p $DIR_WRK
echo "`date` - Changing into backup directory '$DIR_WRK'"
cd $DIR_WRK

# -- backup all databases
for db in `mysql -u $DB_USER -p$DB_PASSWD -e 'SHOW DATABASES;'`
do
case $db in
 Database|information_schema) continue;;
 *)
  echo "`date` - Backing up '$db' to '$db.sql'"
  mysqldump -u $DB_USER -p$DB_PASSWD --opt $db > $db.sql
  echo "`date` - Compressing '$db.sql' to '$db.tgz'"
  tar -czf $db.tgz $db.sql
  echo "`date` - Deleting '$db.sql'"
  rm $db.sql
  ;;
esac
done

# -- transfer to a safe place
echo "`date` - Transfering backup directory to a remote location"
rsync -re ssh $DIR_WRK $STORAGE

echo "`date` - Cleaning up backup directory"
rm -rf $DIR_WRK
echo "`date` - Done"

Automatic deployment part 5

Here is the current version of the build script:

<?xml version="1.0" encoding="UTF-8"?>
<project name="my_project" default="deploy">
<property name="config.file" value="builder/build.properties.${dest}" />
<property name="dist.dir" value="${ant.project.name}-r${svn_revision}-b${build_number}" />
<property name="build.dir" value="${workspace}/${dist.dir}" />

<record name="${build.dir}.log" />

<target name="init">
<available file="${config.file}" property="config.exists"/>
<fail message="Config file '${config.file}' not found." unless="config.exists" />
<echo>Config file '${config.file}' exists.</echo>
<property file="${config.file}" />

<available file="${workspace}" property="workspace.exists"/>
<fail message="Workspace '${workspace}' not found." unless="workspace.exists" />
<echo>Workspace '${workspace}' exists.</echo>
<delete dir="${build.dir}" />
<mkdir dir="${build.dir}" />
</target>

<target name="prepare" depends="init">
<copy todir="${build.dir}">
<fileset dir=".">
<exclude name="**/.*" />
<exclude name="build*" />
<exclude name="bkp/" />
<exclude name="builder/" />
<exclude name="cache/" />
<exclude name="scripts/" />
<exclude name="sources/" />
</fileset>
</copy>
<copy file="builder/.htaccess.${dest}" tofile="${build.dir}/.htaccess" verbose="true" />
</target>

<target name="transfer" depends="prepare">
<scp todir="${dest_user}@${dest_host}:${dest_path}/" password="${dest_passwd}" trust="true">
<fileset dir="${workspace}">
<include name="${dist.dir}/"/>
</fileset>
</scp>
</target>

<target name="remote-deploy" depends="transfer">
<sshexec
host="${dest_host}"
username="${dest_user}"
password="${dest_passwd}"
command="
cd ${dest_path};
rm current;
ln -s ${dist.dir} current;"
/>
</target>

<target name="deploy" depends="remote-deploy">
</target>
<project>

The script needs to be invoked like this:
ant -Dworkspace=/tmp -Ddest=dev -Dsvn_revision=123 -Dbuild_number=123

Automatic deployment part 4

The standard installation of Hudson or Ubuntu does not have all the optional tasks and dependencies. In manage Hudson - configure - I had to create another installation of ant. Then I changed configuration of the project to use that specific installation of the Ant.

On next build Hudson downloaded and installed Ant. ...Which again didn't contain optional tasks. I used command line to get to the Hudson installation directory and in there to the new Ant installation. over there as a hudson user (this is important) I ram ./bin/ant -f fetch.xml -Ddest=system

Then I reloaded configuration of Hudson through manage configuration. Since then I was able to use optional tasks as scp and sshexec.

Automatic deployment part 3

I finished writing the Ant build script for first of our PHP projects. I made plenty of tests. At the end script consists of four parts. I send several parameters to Ant. The most important is destination - devel, test, prod. For each destination I have separate configuration file. The Ant build script then does following:

1. Init - prepares build directory structure.
2. Prepare - copies all files to build directory. In this step an .htaccess for proper destination is copied at proper place in the build directory structure.
3. Transfer - transfers whole build directory to the destination.
4. Remote-deploy - executes series of commands over sshexec. Most of them creates links to newly uploaded build.

So far it works. I'd like to figure out dependencies on other projects as this project itself needs our framework....

The biggest problem for me was to figure out why copy target did not exclude certain directories. But after few hours I figured that out also.

Automatic deployment part 2

So I've installed a Hudson server on one of my virtual machines. It was a snap. Just followed the instructions.

I started to work on Ant build script for one of the projects and figured out following. Aptana Studio 2 doesn't have Ant build as I was used to in Eclipse. I downloaded Eclispe and installed all necessary components for me to find out that installation went bad. Snow Leopard's default Ant does not have optional tasks. No way I was able to enable them.

The solution I opted for was to use MacPorts with Porticus and istalled all Ant related features. After that I ran fetch.xml build script as instructed in installation manual and I have Ant 1.8.1 fully functional.

Automatic deployment part 1

I want to have a deployment process not to be manual anymore. After research I settled for Hudson as an deployment server and I am heading towards continuous integration. I am not sure at the moment if Subversion is righ CRM for the task and thinking of switching to Git. Deploy scripts are to be written in Ant.

The process as I think of is following. Programmers will work on their working copies of projects. Once they are satisfied with progress they commit to the trunk of CRM for the project. Hudson picks up the commit and starts deployment using Ant build script of the project to the testing server. Once we reach set of features and stability required, we mark build as staging ready and Hudson will pick it up and replay it to the staging server to see if the build on production will be functional. If so we mark the build as production build and let Hudson to replay it to the production environment.

There is many things which bothers me. Our projects are usually dependent on other projects of ours. One change in a DB backend required for one project affects few other projects. Deployment needs to account for that. Also tagging in Subversion seems to be a headache to put in symbiosis with Hudson.

VMware Tools and Ubuntu 9.10 problems

With update of VMware ESXi 4.0 to VMware Hypervisor 4.1 I had to reinstall all vmware-tools on all my Ubuntu 9.10 virtual machines. There was no way for me to figure out why it does run on one of them automagically and not on any other machine.

I googled the internet left and right and no help. So I decided to figure out the thing by myself. The thing is that there was a link missing for start-up script. So here is the whole thing needed to be done:

cd /etc/rcS.d
sudo ln -s ../init.d/vmware-tools S38vmware-tools

After that everything works OK again.

VMWare...

Recently I went from using stand alone machines to VMWare ESXi server. It gives me much more flexibility. We purchased HP ML350 with 16GB Ram and one processor so far. Will try to upgrade it soon.

Anyway, the first major thing was, I didn't want to install new machine from the scratch over and over. We have several Ubuntu servers and I just needed one install which I would reuse as a base for different servers.

The way to do it in ESXi is:

1. Prepare one virtual machine for cloning. In my case it was to make a basic installation of Ubuntu server. Don't forget to install VMWare Tools, so you don't have to do it for each of your clones later.
   
2. Then go to your datastare and make a directory for a new virtual machine. Go back to the directory of your Ubutu server installation from previous step. Copy all *.wmx and *.wmdk files over to your new directory.
3. When done go to the new directory, right click on *.wmx file and choose "Add to Inventory". Finish the wizard.
4. Start your new virtual machine.
5. Go to console and edit /etc/hostname file to have the name you want your virtual machine to have.
6. Go to console and edit /etc/hosts. Change hostname from the name of your cloned machine to the one form previous step.
   
7. Go to console and edit /etc/udev/rules.d/70-persistent-net.rules You just need to comment first line and change eth1 to eth0 in the second line.
8. In console edit /etc/network/interfaces and change IP of your machine.
   

After those steps your "cloned" virtual machine is ready. Now you can install whatever software you need for the particular case.

iPhone menace....

OK. SO it is next 12 hours (in which I slept for some 7 hours). While I was using the phone in other 5 hours (for some 30 minutes), iPhone rebooted on me 3 additional times, once in a phone call....
I am finding the device unsatisfactory and calling provider to replace it with any BB model... It is not worth my time....