Here's a backup script that we use to backup our user's IMAP directories on the server to another server. It makes use of "rdiff-backup" and lets us keep 27 weeks of snapshots without using up a ton of disk space.
We process the folders on a account-by-account basis, which reduces the amount of files that rdiff-backup has to keep track of and lowers the memory requirements. It uses a brute-force method of looking for a "subscriptions" file which appears in the root of the MailDir folder for each user. If your installation doesn't have that file, you may need to search for files like "dovecot.index" or ".Junk".
The 3rd line in the "DIRS=" statement is a little one-line perl that will randomize the list of directories. For a backup that runs each day, processing the directories in a random order gives a better chance that all directories will eventually be backed up - even if there is a directory that sometimes causes the script to break. If the script always went in alphabetical order, and the script always breaks at the mid-point, then the directories towards the end of the alphabet will never be backed up. If you don't want that optimization, you can simply replace that section with $SORT and the directories will be processed in alphabetical order.
BKPHOST & BKPBASE control where the files get backed up to. The BASE argument tells the script where to find the IMAP folders on the current server.
Note 1: In order to backup to a remote system like this, it works best if you setup SSH keys and a non-admin / non-root account on the destination server with limited access.
Note 2: "rdiff-backup" works best over LAN networks. If the transfer is aborted due to a link going down, then it will back out completely from the transaction and the folder will not actually be backed up. If the link is too unreliable, this means that rdiff-backup might never accomplish anything at all. One workaround is to rdiff-backup to a local directory and then rsync (with --partial) to a remote host over the unreliable link.
#!/bin/bash
FIND=/usr/bin/find
GREP=/bin/grep
RM=/bin/rm
SED=/bin/sed
SORT=/bin/sort
# source directory
BASE="/var/vmail/"
# destination
BKPHOST=backup-host.example.com
BKPBASE="/backup/mail/vmail/"
echo ""
echo "Backup $BASE to $BKPHOST"
echo "Started at: " `date`
echo ""
# since RHEL5/CentOS5 don't have "sort -R" option to randomize,
# use the following example
# echo -e "2\n1\n3\n5\n4" | perl -MList::Util -e 'print List::Util::shuffle <>'
DIRS=`$FIND $BASE -maxdepth 3 -name subscriptions | \
$GREP '/var/vmail' | $SED 's:^/var/vmail/::' | $SED 's:subscriptions$::' | \
perl -MList::Util -e 'print List::Util::shuffle <>'`
# keep track of directories processed so far (debug purposes)
DCNT=0
for DIR in ${DIRS}
do
echo ""
echo "`date` - Backup: $DIR"
rdiff-backup -v3 --print-statistics --create-full-path \
/var/vmail/$DIR ${BKPHOST}::${BKPBASE}${DIR}
rdiff-backup -v3 --force --remove-older-than 27W \
${BKPHOST}::${BKPBASE}${DIR}
# the following is debug code, to stop the script after N directories
DCNT=$(($DCNT+1))
#echo "DCNT: $DCNT"
#if [[ $DCNT -ge 10 ]]; then exit 0; fi
done
echo ""
echo "Backup finished at:" `date`
echo ""
Peformance for rdiff-backup is limited by the speed of the disks, then the CPU, and possibly the SSH overhead. On older 2GHz Opteron servers, I see throughput of 4-8Mbps over a gigabit LAN. Not that great for an initial backup, but MailDir folders have thousands of individual files. Since most files are mail messages (text) they compress well with the SSH compression. Things go much faster on later runs as only the changes get transferred from the host to the backup server.
A directory of about 810MB and 105,000 files took just under 11 minutes for an initial backup. The SSH compression sped things up quite a bit because the actual transfer speed was around 4.4 megabytes per second, while the network interface never went much above 5-6 megabits per second. That puts the net throughput at around 15-60 gigabytes per hour.
Posted by Thomas at 14:20 0 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2011, Backups, IMAP
Saturday, January 10, 2009
Subversion Backups - Finding SVN repositories
The first trick when backing up SVN repositories is finding them so that you can run the svnadmin hotcopy command. Well, you *could* just setup a list of export DIRS in your backup script - but as you add new SVN repositories, you have to constantly edit that script.
Caveat #1: This setup probably only works for FSFS repositories. I don't use BerkleyDB repositories (a.k.a. BDB) so I can't guarantee that it correctly locates them. I've chosen to look for folders that contain the "db/uuid" file as our "marker" file. Which should result in zero false-positives or mis-identified repositories.
Caveat #2: I'm making the assumption that all of your repositories are stored in a central location (/var/svn), but they do *not* all have to be at the same depth.
Step #1 - Find the uuid files
This is pretty simple, we're just going to use find and grep.
# find /var/svn -name uuid | grep "db/uuid"
/var/svn/tgh-photo/db/uuid
/var/svn/tgh-dev/db/uuid
/var/svn/tgh-web/db/uuid
Step #2 - Clean up the pathnames
Even better, we can tack a sed command onto the end to trim off the "/db/uuid" portion, which gets us exactly what we need for passing to the "svnadmin hotcopy" command.
# find /var/svn -name uuid | grep "db/uuid" | sed 's/\/db\/uuid//'
/var/svn/tgh-photo
/var/svn/tgh-dev
/var/svn/tgh-web
(Make sure that you get all of the "\" and "/" in the right places.)
Step #3 - Strip off the base directory
Since I'm going to create a script variable called "BASE" that equals "/var/svn/", I'm also going to strip that off of the front of the paths.
However, we'll need to convert the BASE variable into something that sed can properly deal with. Otherwise the slashes won't be escaped properly for the sed replacement.
# echo "/var/svn/" | sed 's/\//\\\//g'
\/var\/svn\/
# find /var/svn -name uuid | grep "db/uuid" | sed 's/\/db\/uuid//' | sed 's/^\/var\/svn\///'
tgh-photo
tgh-dev
tgh-web
Or, even better, we can use a different delimiter for sed. That gives us a search line of:
DIRS=`find ${BASE} -name uuid | grep 'db/uuid$' | sed 's:/db/uuid$::' | sed 's:^/var/svn/::'`
Which puts a list of directory names into the DIRS varaible in our bash script.
Step #4 - Putting it all together
Here's our basic script.
#!/bin/bash
BASE="/var/svn/"
HOTCOPY="/var/svn-hotcopy/"
DIRS=`find ${BASE} -name uuid | grep 'db/uuid$' | sed 's:/db/uuid$::' | sed 's:^/var/svn/::'`
for DIR in ${DIRS}
do
echo "svnadmin hotcopy ${BASE}${DIR} to ${HOTCOPY}${DIR}"
rm -r ${HOTCOPY}${DIR}
svnadmin hotcopy ${BASE}${DIR} ${HOTCOPY}${DIR}
done
However, we're not quite done yet because the svn hotcopy command doesn't like it when the destination folder does not exist. So let's add the following scrap of code into the loop.
if ! test -d ${HOTCOPY}${DIR}
then
mkdir -p ${HOTCOPY}${DIR}
fi
The final script
#!/bin/bash
BASE="/var/svn/"
HOTCOPY="/var/svn-hotcopy/"
FIND=/usr/bin/find
GREP=/bin/grep
RM=/bin/rm
SED=/bin/sed
SVNADMIN=/usr/bin/svnadmin
DIRS=`find ${BASE} -name uuid | $GREP 'db/uuid$' | $SED 's:/db/uuid$::' | $SED 's:^/var/svn/::'`
for DIR in ${DIRS}
do
echo "svnadmin hotcopy ${BASE}${DIR} to ${HOTCOPY}${DIR}"
if ! test -d ${HOTCOPY}${DIR}
then
mkdir -p ${HOTCOPY}${DIR}
fi
$RM -r ${HOTCOPY}${DIR}
$SVNADMIN hotcopy ${BASE}${DIR} ${HOTCOPY}${DIR}
done
# insert rdiff-backup line here
Hopefully that works out. Note the use of "rm -r", which could cause data loss if there are errors in the script. You will want to be very careful while working on the script.
Posted by Thomas at 07:59 0 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2009, Backups, bash, SubVersion
PostgreSQL - Backup Speed Tests
Our backup script for pgsql dumps the databases out in plain text SQL format (my preferred method for a variety of reasons). The question was, do we leave it as plain text and/or which compressor do we use?
...
Here are sample times for a plain-text SQL backup.
real 3m30.523s
user 0m14.053s
sys 0m5.132s
The raw database cluster is 22GB (22150820 KB), but that includes indexes. The resulting size of the backups is 794MB (812436 KB). The specific command line used is:
pg_dump -a -b -O -t $s.$t -x $d -f $DIR/$d/$s.$t.sql
($s, $t, $d and $DIR are variables denoting the schema, table, database, and base backup directory)
...
gzip (default compression level of "6")
pg_dump -a -b -O -t $s.$t -x $d | gzip -c > $DIR/$d/$s.$t.sql
CPU usage is pegged at 100% on one of the four CPUs in the box during this operation (due to gzip compressing the streams). So we are bottlenecked by the somewhat slow CPUs in the server.
real 3m0.337s
user 2m17.289s
sys 0m6.740s
So we burned up a lot more CPU time (user 2m 17s) compared to the plain text dump (user 14s). But the overall operation still completed fairly quickly. So how much space did we save? The resulting backups are only 368MB (376820 KB), which is a good bit smaller.
(It would be better, but a large portion of our current database is comprised of the various large "specialized" tables, which are extremely random and difficult to compress. I can't talk about the contents of those tables, but the data in them is generated by a PRNG.)
...
gzip (compression level of "9")
pg_dump -a -b -O -t $s.$t -x $d | gzip -c9 > $DIR/$d/$s.$t.sql.gz
We're likely to be even more CPU-limited here due to telling gzip to "try harder". The resulting backups are 369MB (376944 KB), which is basically the same size.
real 9m39.513s
user 7m28.784s
sys 0m12.585s
So we burn up 3.2x more CPU time, but we don't really change the backup size. Probably not worth it.
...
bzip2
pg_dump -a -b -O -t $s.$t -x $d | bzip2 -c9 > $DIR/$d/$s.$t.sql.bz2
real 19m45.280s
user 3m52.559s
sys 0m11.709s
Interesting, while bzip2 is about twice as slow as gzip (the default compression level), it didn't perform as badly as the maximum compression option of gzip. The resulting backup files are only 330MB (337296 KB), which is a decent improvement over the gzip compression level.
Now, the other interesting thing is that bzip2 took a lot longer to run then gzip, but the server is pretty busy at the moment.
...
Ultimately, we ended up going with bzip2 for a variety of reasons.
- Better compression
- The additional CPU usage was not an issue
- We could change to a smaller block size (-c2) to be more friendly to rsync
Posted by Thomas at 07:45 0 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2009, Backups, PostgreSQL
Thursday, January 08, 2009
PostgreSQL - Basic backup scheme
Here's a basic backup scheme. We're using pg_dump in plain-text mode, compressing the output with bzip2, and writing the results out to files named after the database, schema and table name. It's not the most efficient method, but allows us to go back to:
- any of the past 7 days
- any Sunday within the past month
- the last week of each month in the past quarter
- the last week of each quarter within the past year
- the last week of each year
Which is about 24-25 copies of the data, stored on the hard drive. So you'll need to make sure that you have enough space on the drive to handle all of these copies.
Most of the grunt work is handled by the include script, the daily / weekly / monthly backup scripts simply setup a few variables and then call the main include script.
backup_daily.sh
#!/bin/bash
# DAILY BACKUPS (writes to a daily folder each day)
DAYNR=`date +%w`
echo $DAYNR
DIR=/backup/pgsql/daily/$DAYNR/
echo $DIR
source ~/bin/include_backup_compressed.sh
backup_weekly.sh
#!/bin/bash
# WEEKLY BACKUPS
# Backups go to a five directories based on the day of the month
# converted into 1-5 based on modulus arithmetic. The fifth week
# will sometimes be left over for a few months depending on how
# many weeks there are in the year.
WEEKNR=`date +%d`
echo $WEEKNR
let "WEEKNR = (WEEKNR+6) / 7"
echo $WEEKNR
DIR=/backup/pgsql/weekly/$WEEKNR/
echo $DIR
source ~/bin/include_backup_compressed.sh
backup_monthly.sh
#!/bin/bash
# MONTHLY BACKUPS
# Backups go to three directories based on the month of year
# converted into 1-3 based on modulus arithmetic.
MONTHNR=`date +%m`
echo $MONTHNR
let "MONTHNR = ((MONTHNR -1) % 3) + 1"
echo $MONTHNR
DIR=/backup/pgsql/monthly/$MONTHNR/
echo $DIR
source ~/bin/include_backup_compressed.sh
backup_quarterly.sh
#!/bin/bash
# QUARTERLY BACKUPS
# Backups go to a four directories based on the quarter of the year
# converted into 1-4 based on modulus arithmetic.
QTRNR=`date +%m`
echo $QTRNR
let "QTRNR = (QTRNR+2) / 3"
echo $QTRNR
DIR=/backup/pgsql/quarterly/$QTRNR/
echo $DIR
source ~/bin/include_backup_compressed.sh
backup_yearly.sh
#!/bin/bash
# ANNUAL BACKUPS
YEARNR=`date +%Y`
echo $YEARNR
DIR=/backup/pgsql/yearly/$YEARNR/
echo $DIR
source ~/bin/include_backup_compressed.sh
include_backup_compressed.sh
#!/bin/bash
# Compressed backups to $DIR
echo $DIR
DBS=$(psql -l | grep '|' | awk '{ print $1}' | grep -vE '^-|^Name|template[0|1]')
for d in $DBS
do
echo $d
DBDIR=$DIR/$d
if ! test -d $DBDIR
then
mkdir -p $DBDIR
fi
SCHEMAS=$(psql -d $d -c '\dn' | grep '|' | awk '{ print $1}' \
| grep -vE '^-|^Name|^pg_|^information_schema')
for s in $SCHEMAS
do
echo $d.$s
TABLES=$(psql -d $d -c "SELECT schemaname, tablename FROM pg_catalog.pg_tables WHERE schemaname = '$s';" \
| grep '|' | awk '{ print $3}' | grep -vE '^-|^tablename')
for t in $TABLES
do
echo $d.$s.$t
if [ $s = 'public' ]
then
pg_dump -a -b -O -t $t -x $d | bzip2 -c2 > $DIR/$d/$s.$t.sql.bz2
else
pg_dump -a -b -O -t $s.$t -x $d | bzip2 -c2 > $DIR/$d/$s.$t.sql.bz2
fi
done
done
done
We tried using gzip instead of bzip2, but found that bzip2 worked a little better even though it uses up more CPU. We use a block size of only 200k for bzip2 in order to be more friendly to an rsync push to an external server.
Posted by Thomas at 08:24 0 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2009, Backups, PostgreSQL
Wednesday, January 07, 2009
Very basic rsync / cp backup rotation with hardlinks
Here's a very basic script that I use with RSync that makes use of hard links to reduce the overall size of the backup folder. The limitations are:
- Every morning, a server copies the current version of all files across SSH (using scp) into a "current" folder. There are two folders on the source server that get backed up daily (/home and /local).
- Later on that day, we run the following script to rsync any new files into a daily folder (daily.0 through daily.6).
- In order to bootstrap those daily.# folders, you have to use "cp -al current/* daily.2/" on each, which fills out the seven daily backup folders with hardlinks. Change the number in "daily.2" to 0-6 and run the command once for each of the seven days. Do this after the "current" folder has been populated with data pushed by the source server.
- Ideally, the source server should be pushing changes to the "current" folder using rsync. But in our case, the current server is an old Solaris 9 server without rsync. Which means that our backups are likely to be about 2x to 3x larger then they should be.
- RDiff-Backup may have been a better solution for this particular problem (and we may switch).
- This shows a good example of how to calculate the current day of week number (0-6) as well as calculating what the previous day number was (using modulus arithmetic).
- I make no guarantees that permissions or ownership will be preserved. But since the source server strips all of that information in the process of sending the files over the wire with scp, it's a moot point for our current situation. (rdiff-backup is probably a better choice for that.)
#!/bin/bash
# DAILY BACKUPS (writes to a daily folder each day)
DAYNR=`date +%w`
echo DAYNR=${DAYNR}
let "PREVDAYNR = ((DAYNR + 6) % 7)"
echo PREVDAYNR=${PREVDAYNR}
DIRS="home local"
for DIR in ${DIRS}
do
echo "----- ----- ----- -----"
echo "Backup:" ${DIR}
SRCDIR=/backup/cfmc1/$DIR/current/
DESTDIR=/backup/cfmc1/$DIR/daily.${DAYNR}/
PREVDIR=/backup/cfmc1/$DIR/daily.${PREVDAYNR}/
echo SRCDIR=${SRCDIR}
echo DESTDIR=${DESTDIR}
echo PREVDIR=${PREVDIR}
cp -al ${PREVDIR}* ${DESTDIR}
rsync -a --delete-after ${SRCDIR} ${DESTDIR}
echo "Done."
done
It's not pretty, but it will work better once the source server starts pushing the daily changes via rsync instead of completely overwriting the "current" directory every day.
The code should be pretty self explanatory but I'll explain the two key lines.
cp -al ${PREVDIR}* ${DESTDIR}
This overwrites all files in ${DESTDIR}, which is today, with the files from yesterday, but does it by creating hard links of all files. Old files which were deleted since last week will be left behind until the rsync step.
rsync -a --delete-after ${SRCDIR} ${DESTDIR}
This then brings today's folder up to date with any changes as compared to the source directory (a.k.a. "current"). It also deletes any file in today's folder that don't exist in the source directory.
References:
Easy Automated Snapshot-Style Backups with Linux and Rsync
Local incremental snap shots with rsync
Posted by Thomas at 05:12 0 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2009, Backups, RSync
Friday, May 25, 2007
iSCSITarget on CentOS5
Setting up our test iSCSI SAN box this week. The original plans were to run this on top of Gentoo (which is very powerful and flexible) but after 3 years, I'm not very pleased with Gentoo as a server OS. Which is a whole different topic. So we've migrated over to using CentOS5, which is derived from Red Hat Enterprise Linux 5, a distro that is more suited for corporate use.
There's not much to talk about in terms of the base system. It's a pretty vanilla 64bit CentOS5 install (from DVD) running on top of a dual-CPU dual-core pair of Socket F Opterons. The primary packages that I've installed so far are "Yum Extender" (from stock repositories) and "rdiff-backup" (downloaded as an RPM). The OS runs on top of a 3-disk RAID1 (mirror, all drives active) Software RAID for safety.
I use a semi-customized partition layout on the (3) operating system disks. I have:
a) /boot
b) / (root, the primary OS install area)
c) swap
d) a backup root partition (which is basically a clone of the primary, except for a small change in /etc/fstab) designed for quick recovery from a situation that would hose the primary root partition
e) /var/log (broken out to its own area)
f) /backup/system (a place to store system backups)
g) LVM area (no allocated areas yet)
I mention all that because the first step before installing iscsitarget is to make sure I can recover if things go awry. Since installing iscsitarget involves mucking with the running kernel, I want a good backup of /boot along with making sure GRUB offers me options to boot an older kernel. I'll also freshen my root backup partition.
Step 1 - Backing up /, /boot, and the existing kernel
Simplicity is often best when dealing with the base OS. My methods are crude, but designed to get me back up and running without needing much in the way of software. The primary requirement is a bootable USB pen drive or bootable LiveCD (such as RIPLinuX) with the necessary tools. You could also use the CentOS5 boot DVD.
I'll run with the CentOS install DVD since that's what I have sitting in the optical drive at the moment. When CentOS boots up, enter "linux rescue" at the boot prompt. Note, if you have multiple NICs installed, it's probably better to not start networking (because the CentOS rescue mode takes forever to initialize unconnected NICs).
Select "Skip" when asked about mounting the existing install at /mnt/sysimage. We'll be doing things our own way instead.
Start up Software RAID on the key partitions (/boot, /, the backup /, and the backup partition). The following commands will (usually) startup your existing RAID devices automatically.
# mdadm --examine --scan >> /etc/mdadm.conf
# mdadm --assemble --scan
In my case "md0" is /boot, "md2" is my base CentOS install, "md3" is the backup root partition, and "md5" is where I can store image files. So let's double-check that.
# mkdir /mnt/root ; mount /dev/md3 /mnt/root
# mkdir /mnt/backuproot ; mount /dev/md3 /mnt/backuproot
If we then examine the output of "df -h" or by using "ls" on the mounted volumes we can verify that we know which is which. Let's mount our backup area and create image files. I prefer to kick off the "dd" commands in the background so that I can monitor progress and keep multiple CPUs busy.
# mkdir /mnt/backup ; mount /dev/md5 /mnt/backup
# cd /mnt/backup ; mkdir images ; cd images
# dd if=/dev/md0 | gzip > dd-md0-boot-20070525.img.gz &
# dd if=/dev/md2 | gzip > dd-md2-root-20070525.img.gz &
# dd if=/dev/md3 | gzip > dd-md3-bkproot-20070525.img.gz &
We should also backup the master boot records on each of the hard drives in the unit.
# for i in a b c; do dd if=/dev/sd$i count=1 bs=512 of=dd-sd$i-mbr-20070525.img; done
Unfortunately, the CentOS5 DVD doesn't include tools like "G4L" (Ghost for Linux) or I'd make a second set of backup files using that. I may boot my RIPLinuX CD and see what tools are there. (Because you can never have too many backups.)
Now I can dump the contents of "md2" (the original root) to "md3" (our backup root).
# dd if=/dev/md2 of=/dev/md3
Now for some cleanup stuff...
# mount /dev/md3 /mnt/backuproot
# vi /mnt/backuproot/etc/fstab
We'll need to change any references of "md2" to "md3". Basically flip them around so that "md3" is the official root when /etc/fstab gets processed. I also like to change the prompt and system name to remind myself that I'm using the emergency system. Again, our primary goal is to be able to get a box back up and operational in the case where the primary root partition is hosed. Get it up quickly, then schedule some downtime to deal with it properly.
Now would also be a good time to tune the ext3 file system on your partitions.
The last thing we need to do is edit GRUB's configuration so that we can select our backup root OS from the selection menu.
# mkdir /mnt/boot
# mount /dev/md0 /mnt/boot
# vi /mnt/boot/grub/grub.conf
Things that we'll want to do here (you could also accomplish this by booting the server in normal mode and editing grub.conf there using a more comfortable text editor):
a) Change the timeout=5 value to timeout=15 (or 30 or 60). By default, CentOS doesn't give you very long to pick an alternate boot. I find 5 seconds to be too short of a window, especially on a unit where the storage controller takes a minute or two to scan and setup the drives.
b) Copy the latest "title" section and change "root=/dev/md2" to "root=/dev/md3". I always make the "EMERGENCY" boot option the 2nd one in the list.
# mkdir /mnt/backuproot
# mount /dev/md3 /mnt/backuproot
# vi /mnt/backuproot/etc/sysconfig/network
I like to change the hostname to have "-emergency" tacked onto the end. Which should make it fairly obvious that we are booting up in emergency mode using the backup root partition. I also edit root's .bash_profile to set PS1.
Okay, that was a lot of setup work just to prepare for implementing iSCSITarget (or any other kernel rebuild), but it's always worth it.
Final notes:
- When I test booted the emergency root partition, things didn't work as planned. So while my concept is sound, I may have screwed something up. I think it's an error with /etc/fstab in the emergency partition, so I'll troubleshoot that later.
- It's also possible that you'll need to do a GRUB install on all (3) of the primary mirror disks.
Step 2 - Downloading and compiling the iSCSITarget software
So far, I've found (2) links to be useful here. One is Moving on.... and the other is iSCSI Enterprise Target を CentOS5 にインストールする (japanese). While the 2nd link is in Japanese, it shows the commands in english.
Head over to the The iSCSI Enterprise Target page and download the latest tarball containing the source code. The current version is 0.4.15. If you're using Firefox in CentOS's Gnome shell, it will probably prompt you to open the file with the archive manager. I created a subfolder under /root/iscsitarget-0.4.15 and extracted the contents there.
You will also need to go into Applications -> Add/Remove Software and add the development tools and libraries to your system. (Mostly you just need gcc.)
As noted on jackshck's page, you will also need to install the following packages:
openssl-devel (I installed the x86_64 version)
kernel-devel (again, I'm using the x86_64 version)
Open up a terminal window and go to where you extracted the iscsitarget tarball (I put mine in /root/iscsitarget-0.4.15).
# ls -l /usr/src/kernels
(make note of the kernel folder)
# make KSRC=/usr/src/kernels/2.6.18-8.1.4.el5-x86_64/
# make KSRC=/usr/src/kernels/2.6.18-8.1.4.el5-x86_64/ install
Now we can start up the ietd daemon:
# /etc/init.d/iscsi-target start
And add it to our default runlevel (this is similar to the rc-update command in Gentoo Linux):
# chkconfig iscsi-target on
Step 3 - Creating a target
This is where we get into the nitty-gritty and where I need to take a break and do some research. The /etc/ietd.conf file already exists at this point, but only contains a commented out sample configuration.
Notes:
Dec 21 2007 - The comment about iSCSITarget software for Microsoft Windows really isn't on-topic. But I'll go ahead and list the link to it, but not as an HTML link. Pricing for the real version is currently $395 (Server) or $995 (Professional). And personally, there's no way that I'd recommend running a SAN on top of Microsoft Windows (even Server 2003, which is a nice product).
Posted by Thomas at 07:09 3 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2007, Backups, CentOS5, iSCSITarget, SoftwareRAID
Monday, May 07, 2007
Brute force disaster recovery for CentOS5
Today's trick is moving a CentOS5 system from an old set of disks over to a new set of disks. Along the way, I'll create an image of the system to allow me to restore it later on.
The CentOS5 system is a fresh install running RAID-1 across (3) disks using Linux Software RAID (mdadm). There are (4) primary partitions (boot, root, swap, LVM) with no data on the LVM partition.
(Why 3 active disks? The normal setup for this server was RAID-1 across 2 disks with a hot-spare. Rather then have a risky window of time where one disk has failed and the hot-spare is synchronizing with the remaining good disk, I prefer to have all 3 disks running. That way, when a disk dies, we still have 2 disks in action. The mdadm / Software RAID doesn't seem to care and it doesn't seem to affect performance at all.)
Because this is RAID-1, capturing the disk layout and migrating over to the new disks will be very easy. It's also a very fresh install, so I'm just going to grab the disk contents using "dd" (most of the partition's sectors are still zeroed out from the original install). Once I've backed up the (3) partitions on the first drive, I'm going to pull the (3) drives and replace them with the new ones.
I'll get the machine up and running with the first replacement drive, then configure the blank 2nd and 3rd drives and add them to the RAID set. That is, if mdadm doesn't beat me to the punch and start the sync on the 2nd/3rd disks automatically.
If things go bad, I can always drop the original disks back in the unit and power it back up. I plan on keeping them around for a few days, just in case. I'll have to recreate the LVM volumes, but there aren't any yet (just a PV and a VG).
One advantage of pulling the old drives out completely and rebuilding using fresh drives - I'll end up with a tested disaster recovery process.
Now for the nitty gritty. I'm using a USB pocket drive formatted with ext3 for the rescue work. Make sure that you plug this in before booting the rescue CD.
Login to the system and power it down.
Boot the CentOS5 install DVD
At the "boot:" enter "linux rescue"
Work your way through the startup dialogs
When prompted whether to mount your linux install, choose "Skip"
This should give you a command shell with useful tools. So let's poke around and check on our system.
Looking at "cat /proc/mdstat" shows that while the mdadm software is running, it has not assembled any RAID arrays.
The "fdisk -l" command shows us that the (3) existing disks are named sda, sdb, sdc. Each has (4) partitions (boot, root, swap, LVM).
My USB drive showed up as "/dev/sdd" so I'll create a "/backup" folder and mount it using "mkdir /backup ; mount /dev/sdd1 /backup ; df -h"
Naturally, we should create a sub-folder under /backup for each machine and possibly create another folder underneath it using today's date. We should grab information about the current disk layout and store it in a text file (fdisk.txt).
# cd /backup ; mkdir machinename ; cd machinename
# mkdir todaysdate ; cd todaysdate
# fdisk -l > fdisk.txt
Now to grab the boot loader and image the two critical partitions (boot and root). We'll grab the boot loader off of all (3) drives because it's so small (and it may not be properly synchronized).
dd if=/dev/sda bs=512 count=1 of=machinename-date-sda.mbr
dd if=/dev/sdb bs=512 count=1 of=machinename-date-sdb.mbr
dd if=/dev/sdb bs=512 count=1 of=machinename-date-sdc.mbr
dd if=/dev/sda1 | gzip > machinename-date-ddcopy-sda1.img.gz
dd if=/dev/sda2 | gzip > machinename-date-ddcopy-sda2.img.gz
Total disk space for my system was around 1.75GB worth of compressed files (8GB root, 250MB boot). You could also use bzip2 if you need more compression. Unfortunately, the CentOS5 DVD does not include the "split" command, which could cause issues if you're trying to write to a filesystem that can't handle files over 2GB in size.
Now you should shut the box back down, burn those files to DVD-R, install the new (blank) disks, and boot from the install DVD again. Again, mount the drive that holds the rescue image files to a suitable path.
dd of=/dev/sda bs=512 count=1 if=machinename-date-sda.mbr
fdisk /dev/sda (fix the last partition)
dd if=/dev/sda bs=512 count=1 of=/dev/sdb
dd if=/dev/sda bs=512 count=1 of=/dev/sdc
That will restore the MBR and partition table from the old drive to the new one. If your new drive has a different size, then the last partition will be incorrectly sized for the disk. Fire up "fdisk" and delete / recreate the last partition on the disk.
Restore the two partition images:
# gzip -dc machinename-date-ddcopy-sda1.img.gz | dd of=/dev/sda1
# gzip -dc machinename-date-ddcopy-sda2.img.gz | dd of=/dev/sda2
At this point, we should be able to boot the system on the primary drive and have Software RAID come up in degraded mode for the arrays. Things that will need to be done once the unit boots:
Tell mdadm about the 2nd (and 3rd) disks and tell it to bring those partitions into the arrays and synchronize them.
Create a new swap area
Recreate the LVM physical volume (PV) and volume group (VG)
Restore any data from the LVM area (we had none in this example)
Getting the Software RAID back up and happy is the trickiest of the steps.
Login as root, open up a terminal window
# cat /proc/mdstat
# fdisk -l
Notice that the swap area on our system is sda3, sdb3, sdc3 and will need to be loaded as /dev/md1.
# mdadm --create /dev/md1 -v --level=raid1 --raid-devices=3 /dev/sda3 /dev/sdb3 /dev/sdc3
# mkswap /dev/md1 ; swapon /dev/md1
Now we're ready to recreate the LVM area
# mknod /dev/md3 b 9 3
# mdadm --create /dev/md3 -v --level=raid1 --raid-devices=3 /dev/sda4 /dev/sdb4 /dev/sdc4
# pvcreate /dev/md3 ; vgcreate vg /dev/md3
Finally, we should add the 2nd and 3rd drive to md0 and md2.
mdadm --add /dev/md0 /dev/sdc1
mdadm --add /dev/md0 /dev/sdb1
Note: If your triple mirror RAID array puts the additional disks in as spares, make sure that you have (a) grown the number of raid devices to 3 for the RAID1 set and (b) make sure that there are no other arrays synchronizing as the same time. It's also best to add the elements one at a time, rather then adding both at the same time. I'm not sure if it's a bug in mdadm or just the way it works, but it took me two tries to get my triple mirror back up with all disks marked as "active" instead of (2) active and (1) hot-spare.
Posted by Thomas at 10:52 0 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2007, Backups, CentOS5, DisasterRecovery, SoftwareRAID
Friday, July 28, 2006
Result code 0x57 when scheduling a backup
This was a slightly odd one that did not show up on Google at all. We had a bunch of backup jobs on our main Windows 2003 file server and had recently promoted the Win2003 server to a domain controller using DCPROMO.EXE. In the process of doing that, some tasks refused to run and we also had to delete and re-add a lot of tasks.
In the Scheduled Tasks window, we saw a result code of "0x57" in the Last Result column. In the schedule log (Scheduled Tasks, Advanced, View Log):
"Backup-FileServer-DailyAppend-Week2.job" (ntbackup.exe)
Finished 7/28/2006 8:30:48 PM
Result: The task completed with an exit code of (57).
We checked a few things and finally took a very close look at the "Run" field in the task. Turns out that we were missing a double-quote in the middle of the NTBACKUP command line.
Posted by Thomas at 20:44 0 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2006, Backups, Win2003
Sunday, May 14, 2006
Microsoft Outlook Backup method
This is how I backup my PST files whenever I login to my laptop. Because I always have MSOutlook open, I find it difficult to get a good backup of the PST files.
1) You will want a copy of Info-Zip's ZIP and UNZIP executables.
2) Create a folder on your hard drive to hold these executables. I recommend creating a folder called "C:\bin" and adding that folder to your system's PATH statement.
Right-click on My Computer, Properties, Advanced, Environment Variables... Under System Variables, highlight "PATH" and click "Edit"... place C:\BIN at the start. It should look similar to "C:\bin;%SystemRoot%\system32;%SystemRoot%...". Make sure you leave the existing directories in the list and just add the "C:\bin;" at the start of the listing.
Place the zip.exe and unzip.exe files in the C:\BIN folder.
3) Locate your PST files. Create the following as a text file in the same directory. Name it as "backupemail.cmd".
@echo off
rem Backup my Outlook PST files
IF NOT EXIST "X:\EMailBackup\*" GOTO quit
IF NOT EXIST "C:\Data\EMail\*.pst" GOTO quit
C:
CD "\Data\EMail"
ZIP -u1 X:\EMailBackup\MyPSTs.zip *.pst
X:
CD \EMailBackup
IF EXIST MyPSTs-3.zip DEL MyPSTs-3.zip
IF EXIST MyPSTs-2.zip REN MyPSTs-2.zip MyPSTs-3.zip
IF EXIST MyPSTs-1.zip REN MyPSTs-1.zip MyPSTs-2.zip
IF EXIST MyPSTs.zip REN MyPSTs.zip MyPSTs-1.zip
:quit
Notes:
a) My PST files are stored in C:\Data\EMail. Anywhere that you see "C:\Data\EMail" you should replace with the folder name where your PST files are stored.
b) I backup my PST files to X:\EMailBackup. You will need to replace this path with the location where you keep your ZIP file backups.
c) This script keeps the past 3 backups by renaming them out of the way.
d) You must have at least one file in the X:\EMailBackup folder in order for the backup to run the first time. After that, the script will work properly.
e) If you find that ZIP is too slow, you may wish to change the "-u9" to "-u1" in order to get faster compression (but less compression).
f) I'm pretty sure the above script is foolproof and correct, but as always you should have a good backup before you attempt things like this.
4) Create a shortcut link to "backupemail.cmd" and place it in your Programs -> Startup folder. That way it will run as
We process the folders on a account-by-account basis, which reduces the amount of files that rdiff-backup has to keep track of and lowers the memory requirements. It uses a brute-force method of looking for a "subscriptions" file which appears in the root of the MailDir folder for each user. If your installation doesn't have that file, you may need to search for files like "dovecot.index" or ".Junk".
The 3rd line in the "DIRS=" statement is a little one-line perl that will randomize the list of directories. For a backup that runs each day, processing the directories in a random order gives a better chance that all directories will eventually be backed up - even if there is a directory that sometimes causes the script to break. If the script always went in alphabetical order, and the script always breaks at the mid-point, then the directories towards the end of the alphabet will never be backed up. If you don't want that optimization, you can simply replace that section with $SORT and the directories will be processed in alphabetical order.
BKPHOST & BKPBASE control where the files get backed up to. The BASE argument tells the script where to find the IMAP folders on the current server.
Note 1: In order to backup to a remote system like this, it works best if you setup SSH keys and a non-admin / non-root account on the destination server with limited access.
Note 2: "rdiff-backup" works best over LAN networks. If the transfer is aborted due to a link going down, then it will back out completely from the transaction and the folder will not actually be backed up. If the link is too unreliable, this means that rdiff-backup might never accomplish anything at all. One workaround is to rdiff-backup to a local directory and then rsync (with --partial) to a remote host over the unreliable link.
#!/bin/bash
FIND=/usr/bin/find
GREP=/bin/grep
RM=/bin/rm
SED=/bin/sed
SORT=/bin/sort
# source directory
BASE="/var/vmail/"
# destination
BKPHOST=backup-host.example.com
BKPBASE="/backup/mail/vmail/"
echo ""
echo "Backup $BASE to $BKPHOST"
echo "Started at: " `date`
echo ""
# since RHEL5/CentOS5 don't have "sort -R" option to randomize,
# use the following example
# echo -e "2\n1\n3\n5\n4" | perl -MList::Util -e 'print List::Util::shuffle <>'
DIRS=`$FIND $BASE -maxdepth 3 -name subscriptions | \
$GREP '/var/vmail' | $SED 's:^/var/vmail/::' | $SED 's:subscriptions$::' | \
perl -MList::Util -e 'print List::Util::shuffle <>'`
# keep track of directories processed so far (debug purposes)
DCNT=0
for DIR in ${DIRS}
do
echo ""
echo "`date` - Backup: $DIR"
rdiff-backup -v3 --print-statistics --create-full-path \
/var/vmail/$DIR ${BKPHOST}::${BKPBASE}${DIR}
rdiff-backup -v3 --force --remove-older-than 27W \
${BKPHOST}::${BKPBASE}${DIR}
# the following is debug code, to stop the script after N directories
DCNT=$(($DCNT+1))
#echo "DCNT: $DCNT"
#if [[ $DCNT -ge 10 ]]; then exit 0; fi
done
echo ""
echo "Backup finished at:" `date`
echo ""
Peformance for rdiff-backup is limited by the speed of the disks, then the CPU, and possibly the SSH overhead. On older 2GHz Opteron servers, I see throughput of 4-8Mbps over a gigabit LAN. Not that great for an initial backup, but MailDir folders have thousands of individual files. Since most files are mail messages (text) they compress well with the SSH compression. Things go much faster on later runs as only the changes get transferred from the host to the backup server.
A directory of about 810MB and 105,000 files took just under 11 minutes for an initial backup. The SSH compression sped things up quite a bit because the actual transfer speed was around 4.4 megabytes per second, while the network interface never went much above 5-6 megabits per second. That puts the net throughput at around 15-60 gigabytes per hour.
Posted by Thomas at 14:20 0 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2011, Backups, IMAP
Saturday, January 10, 2009
Subversion Backups - Finding SVN repositories
The first trick when backing up SVN repositories is finding them so that you can run the svnadmin hotcopy command. Well, you *could* just setup a list of export DIRS in your backup script - but as you add new SVN repositories, you have to constantly edit that script.
Caveat #1: This setup probably only works for FSFS repositories. I don't use BerkleyDB repositories (a.k.a. BDB) so I can't guarantee that it correctly locates them. I've chosen to look for folders that contain the "db/uuid" file as our "marker" file. Which should result in zero false-positives or mis-identified repositories.
Caveat #2: I'm making the assumption that all of your repositories are stored in a central location (/var/svn), but they do *not* all have to be at the same depth.
Step #1 - Find the uuid files
This is pretty simple, we're just going to use find and grep.
# find /var/svn -name uuid | grep "db/uuid"
/var/svn/tgh-photo/db/uuid
/var/svn/tgh-dev/db/uuid
/var/svn/tgh-web/db/uuid
Step #2 - Clean up the pathnames
Even better, we can tack a sed command onto the end to trim off the "/db/uuid" portion, which gets us exactly what we need for passing to the "svnadmin hotcopy" command.
# find /var/svn -name uuid | grep "db/uuid" | sed 's/\/db\/uuid//'
/var/svn/tgh-photo
/var/svn/tgh-dev
/var/svn/tgh-web
(Make sure that you get all of the "\" and "/" in the right places.)
Step #3 - Strip off the base directory
Since I'm going to create a script variable called "BASE" that equals "/var/svn/", I'm also going to strip that off of the front of the paths.
However, we'll need to convert the BASE variable into something that sed can properly deal with. Otherwise the slashes won't be escaped properly for the sed replacement.
# echo "/var/svn/" | sed 's/\//\\\//g'
\/var\/svn\/
# find /var/svn -name uuid | grep "db/uuid" | sed 's/\/db\/uuid//' | sed 's/^\/var\/svn\///'
tgh-photo
tgh-dev
tgh-web
Or, even better, we can use a different delimiter for sed. That gives us a search line of:
DIRS=`find ${BASE} -name uuid | grep 'db/uuid$' | sed 's:/db/uuid$::' | sed 's:^/var/svn/::'`
Which puts a list of directory names into the DIRS varaible in our bash script.
Step #4 - Putting it all together
Here's our basic script.
#!/bin/bash
BASE="/var/svn/"
HOTCOPY="/var/svn-hotcopy/"
DIRS=`find ${BASE} -name uuid | grep 'db/uuid$' | sed 's:/db/uuid$::' | sed 's:^/var/svn/::'`
for DIR in ${DIRS}
do
echo "svnadmin hotcopy ${BASE}${DIR} to ${HOTCOPY}${DIR}"
rm -r ${HOTCOPY}${DIR}
svnadmin hotcopy ${BASE}${DIR} ${HOTCOPY}${DIR}
done
However, we're not quite done yet because the svn hotcopy command doesn't like it when the destination folder does not exist. So let's add the following scrap of code into the loop.
if ! test -d ${HOTCOPY}${DIR}
then
mkdir -p ${HOTCOPY}${DIR}
fi
The final script
#!/bin/bash
BASE="/var/svn/"
HOTCOPY="/var/svn-hotcopy/"
FIND=/usr/bin/find
GREP=/bin/grep
RM=/bin/rm
SED=/bin/sed
SVNADMIN=/usr/bin/svnadmin
DIRS=`find ${BASE} -name uuid | $GREP 'db/uuid$' | $SED 's:/db/uuid$::' | $SED 's:^/var/svn/::'`
for DIR in ${DIRS}
do
echo "svnadmin hotcopy ${BASE}${DIR} to ${HOTCOPY}${DIR}"
if ! test -d ${HOTCOPY}${DIR}
then
mkdir -p ${HOTCOPY}${DIR}
fi
$RM -r ${HOTCOPY}${DIR}
$SVNADMIN hotcopy ${BASE}${DIR} ${HOTCOPY}${DIR}
done
# insert rdiff-backup line here
Hopefully that works out. Note the use of "rm -r", which could cause data loss if there are errors in the script. You will want to be very careful while working on the script.
Posted by Thomas at 07:59 0 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2009, Backups, bash, SubVersion
PostgreSQL - Backup Speed Tests
Our backup script for pgsql dumps the databases out in plain text SQL format (my preferred method for a variety of reasons). The question was, do we leave it as plain text and/or which compressor do we use?
...
Here are sample times for a plain-text SQL backup.
real 3m30.523s
user 0m14.053s
sys 0m5.132s
The raw database cluster is 22GB (22150820 KB), but that includes indexes. The resulting size of the backups is 794MB (812436 KB). The specific command line used is:
pg_dump -a -b -O -t $s.$t -x $d -f $DIR/$d/$s.$t.sql
($s, $t, $d and $DIR are variables denoting the schema, table, database, and base backup directory)
...
gzip (default compression level of "6")
pg_dump -a -b -O -t $s.$t -x $d | gzip -c > $DIR/$d/$s.$t.sql
CPU usage is pegged at 100% on one of the four CPUs in the box during this operation (due to gzip compressing the streams). So we are bottlenecked by the somewhat slow CPUs in the server.
real 3m0.337s
user 2m17.289s
sys 0m6.740s
So we burned up a lot more CPU time (user 2m 17s) compared to the plain text dump (user 14s). But the overall operation still completed fairly quickly. So how much space did we save? The resulting backups are only 368MB (376820 KB), which is a good bit smaller.
(It would be better, but a large portion of our current database is comprised of the various large "specialized" tables, which are extremely random and difficult to compress. I can't talk about the contents of those tables, but the data in them is generated by a PRNG.)
...
gzip (compression level of "9")
pg_dump -a -b -O -t $s.$t -x $d | gzip -c9 > $DIR/$d/$s.$t.sql.gz
We're likely to be even more CPU-limited here due to telling gzip to "try harder". The resulting backups are 369MB (376944 KB), which is basically the same size.
real 9m39.513s
user 7m28.784s
sys 0m12.585s
So we burn up 3.2x more CPU time, but we don't really change the backup size. Probably not worth it.
...
bzip2
pg_dump -a -b -O -t $s.$t -x $d | bzip2 -c9 > $DIR/$d/$s.$t.sql.bz2
real 19m45.280s
user 3m52.559s
sys 0m11.709s
Interesting, while bzip2 is about twice as slow as gzip (the default compression level), it didn't perform as badly as the maximum compression option of gzip. The resulting backup files are only 330MB (337296 KB), which is a decent improvement over the gzip compression level.
Now, the other interesting thing is that bzip2 took a lot longer to run then gzip, but the server is pretty busy at the moment.
...
Ultimately, we ended up going with bzip2 for a variety of reasons.
- Better compression
- The additional CPU usage was not an issue
- We could change to a smaller block size (-c2) to be more friendly to rsync
Posted by Thomas at 07:45 0 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2009, Backups, PostgreSQL
Thursday, January 08, 2009
PostgreSQL - Basic backup scheme
Here's a basic backup scheme. We're using pg_dump in plain-text mode, compressing the output with bzip2, and writing the results out to files named after the database, schema and table name. It's not the most efficient method, but allows us to go back to:
- any of the past 7 days
- any Sunday within the past month
- the last week of each month in the past quarter
- the last week of each quarter within the past year
- the last week of each year
Which is about 24-25 copies of the data, stored on the hard drive. So you'll need to make sure that you have enough space on the drive to handle all of these copies.
Most of the grunt work is handled by the include script, the daily / weekly / monthly backup scripts simply setup a few variables and then call the main include script.
backup_daily.sh
#!/bin/bash
# DAILY BACKUPS (writes to a daily folder each day)
DAYNR=`date +%w`
echo $DAYNR
DIR=/backup/pgsql/daily/$DAYNR/
echo $DIR
source ~/bin/include_backup_compressed.sh
backup_weekly.sh
#!/bin/bash
# WEEKLY BACKUPS
# Backups go to a five directories based on the day of the month
# converted into 1-5 based on modulus arithmetic. The fifth week
# will sometimes be left over for a few months depending on how
# many weeks there are in the year.
WEEKNR=`date +%d`
echo $WEEKNR
let "WEEKNR = (WEEKNR+6) / 7"
echo $WEEKNR
DIR=/backup/pgsql/weekly/$WEEKNR/
echo $DIR
source ~/bin/include_backup_compressed.sh
backup_monthly.sh
#!/bin/bash
# MONTHLY BACKUPS
# Backups go to three directories based on the month of year
# converted into 1-3 based on modulus arithmetic.
MONTHNR=`date +%m`
echo $MONTHNR
let "MONTHNR = ((MONTHNR -1) % 3) + 1"
echo $MONTHNR
DIR=/backup/pgsql/monthly/$MONTHNR/
echo $DIR
source ~/bin/include_backup_compressed.sh
backup_quarterly.sh
#!/bin/bash
# QUARTERLY BACKUPS
# Backups go to a four directories based on the quarter of the year
# converted into 1-4 based on modulus arithmetic.
QTRNR=`date +%m`
echo $QTRNR
let "QTRNR = (QTRNR+2) / 3"
echo $QTRNR
DIR=/backup/pgsql/quarterly/$QTRNR/
echo $DIR
source ~/bin/include_backup_compressed.sh
backup_yearly.sh
#!/bin/bash
# ANNUAL BACKUPS
YEARNR=`date +%Y`
echo $YEARNR
DIR=/backup/pgsql/yearly/$YEARNR/
echo $DIR
source ~/bin/include_backup_compressed.sh
include_backup_compressed.sh
#!/bin/bash
# Compressed backups to $DIR
echo $DIR
DBS=$(psql -l | grep '|' | awk '{ print $1}' | grep -vE '^-|^Name|template[0|1]')
for d in $DBS
do
echo $d
DBDIR=$DIR/$d
if ! test -d $DBDIR
then
mkdir -p $DBDIR
fi
SCHEMAS=$(psql -d $d -c '\dn' | grep '|' | awk '{ print $1}' \
| grep -vE '^-|^Name|^pg_|^information_schema')
for s in $SCHEMAS
do
echo $d.$s
TABLES=$(psql -d $d -c "SELECT schemaname, tablename FROM pg_catalog.pg_tables WHERE schemaname = '$s';" \
| grep '|' | awk '{ print $3}' | grep -vE '^-|^tablename')
for t in $TABLES
do
echo $d.$s.$t
if [ $s = 'public' ]
then
pg_dump -a -b -O -t $t -x $d | bzip2 -c2 > $DIR/$d/$s.$t.sql.bz2
else
pg_dump -a -b -O -t $s.$t -x $d | bzip2 -c2 > $DIR/$d/$s.$t.sql.bz2
fi
done
done
done
We tried using gzip instead of bzip2, but found that bzip2 worked a little better even though it uses up more CPU. We use a block size of only 200k for bzip2 in order to be more friendly to an rsync push to an external server.
Posted by Thomas at 08:24 0 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2009, Backups, PostgreSQL
Wednesday, January 07, 2009
Very basic rsync / cp backup rotation with hardlinks
Here's a very basic script that I use with RSync that makes use of hard links to reduce the overall size of the backup folder. The limitations are:
- Every morning, a server copies the current version of all files across SSH (using scp) into a "current" folder. There are two folders on the source server that get backed up daily (/home and /local).
- Later on that day, we run the following script to rsync any new files into a daily folder (daily.0 through daily.6).
- In order to bootstrap those daily.# folders, you have to use "cp -al current/* daily.2/" on each, which fills out the seven daily backup folders with hardlinks. Change the number in "daily.2" to 0-6 and run the command once for each of the seven days. Do this after the "current" folder has been populated with data pushed by the source server.
- Ideally, the source server should be pushing changes to the "current" folder using rsync. But in our case, the current server is an old Solaris 9 server without rsync. Which means that our backups are likely to be about 2x to 3x larger then they should be.
- RDiff-Backup may have been a better solution for this particular problem (and we may switch).
- This shows a good example of how to calculate the current day of week number (0-6) as well as calculating what the previous day number was (using modulus arithmetic).
- I make no guarantees that permissions or ownership will be preserved. But since the source server strips all of that information in the process of sending the files over the wire with scp, it's a moot point for our current situation. (rdiff-backup is probably a better choice for that.)
#!/bin/bash
# DAILY BACKUPS (writes to a daily folder each day)
DAYNR=`date +%w`
echo DAYNR=${DAYNR}
let "PREVDAYNR = ((DAYNR + 6) % 7)"
echo PREVDAYNR=${PREVDAYNR}
DIRS="home local"
for DIR in ${DIRS}
do
echo "----- ----- ----- -----"
echo "Backup:" ${DIR}
SRCDIR=/backup/cfmc1/$DIR/current/
DESTDIR=/backup/cfmc1/$DIR/daily.${DAYNR}/
PREVDIR=/backup/cfmc1/$DIR/daily.${PREVDAYNR}/
echo SRCDIR=${SRCDIR}
echo DESTDIR=${DESTDIR}
echo PREVDIR=${PREVDIR}
cp -al ${PREVDIR}* ${DESTDIR}
rsync -a --delete-after ${SRCDIR} ${DESTDIR}
echo "Done."
done
It's not pretty, but it will work better once the source server starts pushing the daily changes via rsync instead of completely overwriting the "current" directory every day.
The code should be pretty self explanatory but I'll explain the two key lines.
cp -al ${PREVDIR}* ${DESTDIR}
This overwrites all files in ${DESTDIR}, which is today, with the files from yesterday, but does it by creating hard links of all files. Old files which were deleted since last week will be left behind until the rsync step.
rsync -a --delete-after ${SRCDIR} ${DESTDIR}
This then brings today's folder up to date with any changes as compared to the source directory (a.k.a. "current"). It also deletes any file in today's folder that don't exist in the source directory.
References:
Easy Automated Snapshot-Style Backups with Linux and Rsync
Local incremental snap shots with rsync
Posted by Thomas at 05:12 0 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2009, Backups, RSync
Friday, May 25, 2007
iSCSITarget on CentOS5
Setting up our test iSCSI SAN box this week. The original plans were to run this on top of Gentoo (which is very powerful and flexible) but after 3 years, I'm not very pleased with Gentoo as a server OS. Which is a whole different topic. So we've migrated over to using CentOS5, which is derived from Red Hat Enterprise Linux 5, a distro that is more suited for corporate use.
There's not much to talk about in terms of the base system. It's a pretty vanilla 64bit CentOS5 install (from DVD) running on top of a dual-CPU dual-core pair of Socket F Opterons. The primary packages that I've installed so far are "Yum Extender" (from stock repositories) and "rdiff-backup" (downloaded as an RPM). The OS runs on top of a 3-disk RAID1 (mirror, all drives active) Software RAID for safety.
I use a semi-customized partition layout on the (3) operating system disks. I have:
a) /boot
b) / (root, the primary OS install area)
c) swap
d) a backup root partition (which is basically a clone of the primary, except for a small change in /etc/fstab) designed for quick recovery from a situation that would hose the primary root partition
e) /var/log (broken out to its own area)
f) /backup/system (a place to store system backups)
g) LVM area (no allocated areas yet)
I mention all that because the first step before installing iscsitarget is to make sure I can recover if things go awry. Since installing iscsitarget involves mucking with the running kernel, I want a good backup of /boot along with making sure GRUB offers me options to boot an older kernel. I'll also freshen my root backup partition.
Step 1 - Backing up /, /boot, and the existing kernel
Simplicity is often best when dealing with the base OS. My methods are crude, but designed to get me back up and running without needing much in the way of software. The primary requirement is a bootable USB pen drive or bootable LiveCD (such as RIPLinuX) with the necessary tools. You could also use the CentOS5 boot DVD.
I'll run with the CentOS install DVD since that's what I have sitting in the optical drive at the moment. When CentOS boots up, enter "linux rescue" at the boot prompt. Note, if you have multiple NICs installed, it's probably better to not start networking (because the CentOS rescue mode takes forever to initialize unconnected NICs).
Select "Skip" when asked about mounting the existing install at /mnt/sysimage. We'll be doing things our own way instead.
Start up Software RAID on the key partitions (/boot, /, the backup /, and the backup partition). The following commands will (usually) startup your existing RAID devices automatically.
# mdadm --examine --scan >> /etc/mdadm.conf
# mdadm --assemble --scan
In my case "md0" is /boot, "md2" is my base CentOS install, "md3" is the backup root partition, and "md5" is where I can store image files. So let's double-check that.
# mkdir /mnt/root ; mount /dev/md3 /mnt/root
# mkdir /mnt/backuproot ; mount /dev/md3 /mnt/backuproot
If we then examine the output of "df -h" or by using "ls" on the mounted volumes we can verify that we know which is which. Let's mount our backup area and create image files. I prefer to kick off the "dd" commands in the background so that I can monitor progress and keep multiple CPUs busy.
# mkdir /mnt/backup ; mount /dev/md5 /mnt/backup
# cd /mnt/backup ; mkdir images ; cd images
# dd if=/dev/md0 | gzip > dd-md0-boot-20070525.img.gz &
# dd if=/dev/md2 | gzip > dd-md2-root-20070525.img.gz &
# dd if=/dev/md3 | gzip > dd-md3-bkproot-20070525.img.gz &
We should also backup the master boot records on each of the hard drives in the unit.
# for i in a b c; do dd if=/dev/sd$i count=1 bs=512 of=dd-sd$i-mbr-20070525.img; done
Unfortunately, the CentOS5 DVD doesn't include tools like "G4L" (Ghost for Linux) or I'd make a second set of backup files using that. I may boot my RIPLinuX CD and see what tools are there. (Because you can never have too many backups.)
Now I can dump the contents of "md2" (the original root) to "md3" (our backup root).
# dd if=/dev/md2 of=/dev/md3
Now for some cleanup stuff...
# mount /dev/md3 /mnt/backuproot
# vi /mnt/backuproot/etc/fstab
We'll need to change any references of "md2" to "md3". Basically flip them around so that "md3" is the official root when /etc/fstab gets processed. I also like to change the prompt and system name to remind myself that I'm using the emergency system. Again, our primary goal is to be able to get a box back up and operational in the case where the primary root partition is hosed. Get it up quickly, then schedule some downtime to deal with it properly.
Now would also be a good time to tune the ext3 file system on your partitions.
The last thing we need to do is edit GRUB's configuration so that we can select our backup root OS from the selection menu.
# mkdir /mnt/boot
# mount /dev/md0 /mnt/boot
# vi /mnt/boot/grub/grub.conf
Things that we'll want to do here (you could also accomplish this by booting the server in normal mode and editing grub.conf there using a more comfortable text editor):
a) Change the timeout=5 value to timeout=15 (or 30 or 60). By default, CentOS doesn't give you very long to pick an alternate boot. I find 5 seconds to be too short of a window, especially on a unit where the storage controller takes a minute or two to scan and setup the drives.
b) Copy the latest "title" section and change "root=/dev/md2" to "root=/dev/md3". I always make the "EMERGENCY" boot option the 2nd one in the list.
# mkdir /mnt/backuproot
# mount /dev/md3 /mnt/backuproot
# vi /mnt/backuproot/etc/sysconfig/network
I like to change the hostname to have "-emergency" tacked onto the end. Which should make it fairly obvious that we are booting up in emergency mode using the backup root partition. I also edit root's .bash_profile to set PS1.
Okay, that was a lot of setup work just to prepare for implementing iSCSITarget (or any other kernel rebuild), but it's always worth it.
Final notes:
- When I test booted the emergency root partition, things didn't work as planned. So while my concept is sound, I may have screwed something up. I think it's an error with /etc/fstab in the emergency partition, so I'll troubleshoot that later.
- It's also possible that you'll need to do a GRUB install on all (3) of the primary mirror disks.
Step 2 - Downloading and compiling the iSCSITarget software
So far, I've found (2) links to be useful here. One is Moving on.... and the other is iSCSI Enterprise Target を CentOS5 にインストールする (japanese). While the 2nd link is in Japanese, it shows the commands in english.
Head over to the The iSCSI Enterprise Target page and download the latest tarball containing the source code. The current version is 0.4.15. If you're using Firefox in CentOS's Gnome shell, it will probably prompt you to open the file with the archive manager. I created a subfolder under /root/iscsitarget-0.4.15 and extracted the contents there.
You will also need to go into Applications -> Add/Remove Software and add the development tools and libraries to your system. (Mostly you just need gcc.)
As noted on jackshck's page, you will also need to install the following packages:
openssl-devel (I installed the x86_64 version)
kernel-devel (again, I'm using the x86_64 version)
Open up a terminal window and go to where you extracted the iscsitarget tarball (I put mine in /root/iscsitarget-0.4.15).
# ls -l /usr/src/kernels
(make note of the kernel folder)
# make KSRC=/usr/src/kernels/2.6.18-8.1.4.el5-x86_64/
# make KSRC=/usr/src/kernels/2.6.18-8.1.4.el5-x86_64/ install
Now we can start up the ietd daemon:
# /etc/init.d/iscsi-target start
And add it to our default runlevel (this is similar to the rc-update command in Gentoo Linux):
# chkconfig iscsi-target on
Step 3 - Creating a target
This is where we get into the nitty-gritty and where I need to take a break and do some research. The /etc/ietd.conf file already exists at this point, but only contains a commented out sample configuration.
Notes:
Dec 21 2007 - The comment about iSCSITarget software for Microsoft Windows really isn't on-topic. But I'll go ahead and list the link to it, but not as an HTML link. Pricing for the real version is currently $395 (Server) or $995 (Professional). And personally, there's no way that I'd recommend running a SAN on top of Microsoft Windows (even Server 2003, which is a nice product).
Posted by Thomas at 07:09 3 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2007, Backups, CentOS5, iSCSITarget, SoftwareRAID
Monday, May 07, 2007
Brute force disaster recovery for CentOS5
Today's trick is moving a CentOS5 system from an old set of disks over to a new set of disks. Along the way, I'll create an image of the system to allow me to restore it later on.
The CentOS5 system is a fresh install running RAID-1 across (3) disks using Linux Software RAID (mdadm). There are (4) primary partitions (boot, root, swap, LVM) with no data on the LVM partition.
(Why 3 active disks? The normal setup for this server was RAID-1 across 2 disks with a hot-spare. Rather then have a risky window of time where one disk has failed and the hot-spare is synchronizing with the remaining good disk, I prefer to have all 3 disks running. That way, when a disk dies, we still have 2 disks in action. The mdadm / Software RAID doesn't seem to care and it doesn't seem to affect performance at all.)
Because this is RAID-1, capturing the disk layout and migrating over to the new disks will be very easy. It's also a very fresh install, so I'm just going to grab the disk contents using "dd" (most of the partition's sectors are still zeroed out from the original install). Once I've backed up the (3) partitions on the first drive, I'm going to pull the (3) drives and replace them with the new ones.
I'll get the machine up and running with the first replacement drive, then configure the blank 2nd and 3rd drives and add them to the RAID set. That is, if mdadm doesn't beat me to the punch and start the sync on the 2nd/3rd disks automatically.
If things go bad, I can always drop the original disks back in the unit and power it back up. I plan on keeping them around for a few days, just in case. I'll have to recreate the LVM volumes, but there aren't any yet (just a PV and a VG).
One advantage of pulling the old drives out completely and rebuilding using fresh drives - I'll end up with a tested disaster recovery process.
Now for the nitty gritty. I'm using a USB pocket drive formatted with ext3 for the rescue work. Make sure that you plug this in before booting the rescue CD.
Login to the system and power it down.
Boot the CentOS5 install DVD
At the "boot:" enter "linux rescue"
Work your way through the startup dialogs
When prompted whether to mount your linux install, choose "Skip"
This should give you a command shell with useful tools. So let's poke around and check on our system.
Looking at "cat /proc/mdstat" shows that while the mdadm software is running, it has not assembled any RAID arrays.
The "fdisk -l" command shows us that the (3) existing disks are named sda, sdb, sdc. Each has (4) partitions (boot, root, swap, LVM).
My USB drive showed up as "/dev/sdd" so I'll create a "/backup" folder and mount it using "mkdir /backup ; mount /dev/sdd1 /backup ; df -h"
Naturally, we should create a sub-folder under /backup for each machine and possibly create another folder underneath it using today's date. We should grab information about the current disk layout and store it in a text file (fdisk.txt).
# cd /backup ; mkdir machinename ; cd machinename
# mkdir todaysdate ; cd todaysdate
# fdisk -l > fdisk.txt
Now to grab the boot loader and image the two critical partitions (boot and root). We'll grab the boot loader off of all (3) drives because it's so small (and it may not be properly synchronized).
dd if=/dev/sda bs=512 count=1 of=machinename-date-sda.mbr
dd if=/dev/sdb bs=512 count=1 of=machinename-date-sdb.mbr
dd if=/dev/sdb bs=512 count=1 of=machinename-date-sdc.mbr
dd if=/dev/sda1 | gzip > machinename-date-ddcopy-sda1.img.gz
dd if=/dev/sda2 | gzip > machinename-date-ddcopy-sda2.img.gz
Total disk space for my system was around 1.75GB worth of compressed files (8GB root, 250MB boot). You could also use bzip2 if you need more compression. Unfortunately, the CentOS5 DVD does not include the "split" command, which could cause issues if you're trying to write to a filesystem that can't handle files over 2GB in size.
Now you should shut the box back down, burn those files to DVD-R, install the new (blank) disks, and boot from the install DVD again. Again, mount the drive that holds the rescue image files to a suitable path.
dd of=/dev/sda bs=512 count=1 if=machinename-date-sda.mbr
fdisk /dev/sda (fix the last partition)
dd if=/dev/sda bs=512 count=1 of=/dev/sdb
dd if=/dev/sda bs=512 count=1 of=/dev/sdc
That will restore the MBR and partition table from the old drive to the new one. If your new drive has a different size, then the last partition will be incorrectly sized for the disk. Fire up "fdisk" and delete / recreate the last partition on the disk.
Restore the two partition images:
# gzip -dc machinename-date-ddcopy-sda1.img.gz | dd of=/dev/sda1
# gzip -dc machinename-date-ddcopy-sda2.img.gz | dd of=/dev/sda2
At this point, we should be able to boot the system on the primary drive and have Software RAID come up in degraded mode for the arrays. Things that will need to be done once the unit boots:
Tell mdadm about the 2nd (and 3rd) disks and tell it to bring those partitions into the arrays and synchronize them.
Create a new swap area
Recreate the LVM physical volume (PV) and volume group (VG)
Restore any data from the LVM area (we had none in this example)
Getting the Software RAID back up and happy is the trickiest of the steps.
Login as root, open up a terminal window
# cat /proc/mdstat
# fdisk -l
Notice that the swap area on our system is sda3, sdb3, sdc3 and will need to be loaded as /dev/md1.
# mdadm --create /dev/md1 -v --level=raid1 --raid-devices=3 /dev/sda3 /dev/sdb3 /dev/sdc3
# mkswap /dev/md1 ; swapon /dev/md1
Now we're ready to recreate the LVM area
# mknod /dev/md3 b 9 3
# mdadm --create /dev/md3 -v --level=raid1 --raid-devices=3 /dev/sda4 /dev/sdb4 /dev/sdc4
# pvcreate /dev/md3 ; vgcreate vg /dev/md3
Finally, we should add the 2nd and 3rd drive to md0 and md2.
mdadm --add /dev/md0 /dev/sdc1
mdadm --add /dev/md0 /dev/sdb1
Note: If your triple mirror RAID array puts the additional disks in as spares, make sure that you have (a) grown the number of raid devices to 3 for the RAID1 set and (b) make sure that there are no other arrays synchronizing as the same time. It's also best to add the elements one at a time, rather then adding both at the same time. I'm not sure if it's a bug in mdadm or just the way it works, but it took me two tries to get my triple mirror back up with all disks marked as "active" instead of (2) active and (1) hot-spare.
Posted by Thomas at 10:52 0 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2007, Backups, CentOS5, DisasterRecovery, SoftwareRAID
Friday, July 28, 2006
Result code 0x57 when scheduling a backup
This was a slightly odd one that did not show up on Google at all. We had a bunch of backup jobs on our main Windows 2003 file server and had recently promoted the Win2003 server to a domain controller using DCPROMO.EXE. In the process of doing that, some tasks refused to run and we also had to delete and re-add a lot of tasks.
In the Scheduled Tasks window, we saw a result code of "0x57" in the Last Result column. In the schedule log (Scheduled Tasks, Advanced, View Log):
"Backup-FileServer-DailyAppend-Week2.job" (ntbackup.exe)
Finished 7/28/2006 8:30:48 PM
Result: The task completed with an exit code of (57).
We checked a few things and finally took a very close look at the "Run" field in the task. Turns out that we were missing a double-quote in the middle of the NTBACKUP command line.
Posted by Thomas at 20:44 0 comments
Email This
BlogThis!
Share to Twitter
Share to Facebook
Labels: 2006, Backups, Win2003
Sunday, May 14, 2006
Microsoft Outlook Backup method
This is how I backup my PST files whenever I login to my laptop. Because I always have MSOutlook open, I find it difficult to get a good backup of the PST files.
1) You will want a copy of Info-Zip's ZIP and UNZIP executables.
2) Create a folder on your hard drive to hold these executables. I recommend creating a folder called "C:\bin" and adding that folder to your system's PATH statement.
Right-click on My Computer, Properties, Advanced, Environment Variables... Under System Variables, highlight "PATH" and click "Edit"... place C:\BIN at the start. It should look similar to "C:\bin;%SystemRoot%\system32;%SystemRoot%...". Make sure you leave the existing directories in the list and just add the "C:\bin;" at the start of the listing.
Place the zip.exe and unzip.exe files in the C:\BIN folder.
3) Locate your PST files. Create the following as a text file in the same directory. Name it as "backupemail.cmd".
@echo off
rem Backup my Outlook PST files
IF NOT EXIST "X:\EMailBackup\*" GOTO quit
IF NOT EXIST "C:\Data\EMail\*.pst" GOTO quit
C:
CD "\Data\EMail"
ZIP -u1 X:\EMailBackup\MyPSTs.zip *.pst
X:
CD \EMailBackup
IF EXIST MyPSTs-3.zip DEL MyPSTs-3.zip
IF EXIST MyPSTs-2.zip REN MyPSTs-2.zip MyPSTs-3.zip
IF EXIST MyPSTs-1.zip REN MyPSTs-1.zip MyPSTs-2.zip
IF EXIST MyPSTs.zip REN MyPSTs.zip MyPSTs-1.zip
:quit
Notes:
a) My PST files are stored in C:\Data\EMail. Anywhere that you see "C:\Data\EMail" you should replace with the folder name where your PST files are stored.
b) I backup my PST files to X:\EMailBackup. You will need to replace this path with the location where you keep your ZIP file backups.
c) This script keeps the past 3 backups by renaming them out of the way.
d) You must have at least one file in the X:\EMailBackup folder in order for the backup to run the first time. After that, the script will work properly.
e) If you find that ZIP is too slow, you may wish to change the "-u9" to "-u1" in order to get faster compression (but less compression).
f) I'm pretty sure the above script is foolproof and correct, but as always you should have a good backup before you attempt things like this.
4) Create a shortcut link to "backupemail.cmd" and place it in your Programs -> Startup folder. That way it will run as
Hi Bhaskar i want to install Centos On the server which has 4 TB hard disk and 8GB ram ,Which version should i install and how?
ReplyDelete