Simple Backups with rsnapshot — 3 Step Guide

Introduction

Since I recently set up a simple backup scheme for my laptop, workstation and server that I’m quite happy with, I thought I should share. The thing about backups is that everyone talks about how one should have them, but who really has an adequate backup scheme?

My laptop and workstation runs Arch Linux. The server is an old FreeBSD 7.0 install. I know the FreeBSD version is ancient, but the installed ports are kept up to date, and the machine has been running fine for years. It hosts this blog, along with the blogs and websites of some friends, a DNS server and a private SILC server me and some friends use for chatting.

So in short, here’s what I did for backups on the server. For the laptop and workstation, the instructions are pretty much identical, except you can leave out the parts about MySQL backups.

Step 1 — Install rsnapshot

portinstall rsnapshot (pacman -S rsnapshot on Arch)

Step 2 — Configure rsnapshot

I use the following configuration to tell rsnapshot to keep seven days of daily backups along with one montly backup in /usr/.rsnapshot.

config_version  1.2
snapshot_root   /usr/.snapshots/
cmd_rm          /bin/rm
cmd_rsync       /usr/local/bin/rsync
cmd_logger      /usr/bin/logger

cmd_postexec    /usr/local/bin/backup-strongspace.sh

interval        daily   7
interval        monthly 1

verbose         2
loglevel        3
logfile /var/log/rsnapshot
lockfile        /var/run/rsnapshot.pid

rsync_long_args --delete --numeric-ids --relative --delete-excluded --filter="dir-merge,n- .backup-exclude"
link_dest       1

backup  /usr/home/              localhost/
backup  /etc/           localhost/
backup  /var/named/     localhost/
backup  /var/www/       localhost/
backup  /usr/local/etc/ localhost/
backup_script   /usr/local/bin/backup-mysql.sh  localhost/mysql/

The interval directives tell rsnapshot how many backups to keep. E.g. with the above configuration, if I execute rsnapshot daily ten times, the last seven of the backups will be kept. rsnapshot uses hard linking to save space, so the disk usage won’t be horrible.

By using the cmd_postexec directive, I give the path to a script to execute after each backup run. In my backup-strongspace.sh script I have:

#!/bin/sh

/usr/local/bin/rsync -az --delete --delete-excluded /usr/.snapshots/daily.0 estan@estan.strongspace.com:/strongspace/estan/dose

This will sync the latest backup to my Strongspace account (40 GB Starter account, $4.99/month).

The --filter="dir-merge,n- .backup-exclude" is obscure rsync syntax and means that I can put stuff to be excluded from backup in directory specific .backup-exclude files.

Next comes the backup directives, these simply specify the directories that should be backed up.

Using the backup-script directive, I specify the path to script to be run in an empty temporary directory before that directory is backed up. The backup-mysql.sh script I’ve specified contains the following:

#!/bin/sh

/usr/local/bin/mysqldump -u root -pmypassword --all-databases | gzip > all-databases.sql.gz

This will simply make a gzipped dump of all MySQL databases on the machine, which will then be backed up by rsnapshot.

An important note for Linux users is that you probably want to specify cmd_cp /bin/cp, as rsnapshot can take advantage of some features of GNU cp.

Step 3 — Configure cron job

I created two scripts that are run from cron as part of FreeBSD’s regular periodic maintenance scripts:

/etc/periodic/daily/001.backup:

#!/bin/sh

/usr/local/bin/rsnapshot daily > /tmp/rsnapshot.out 2>&1 || cat /tmp/rsnapshot.out | mail -s "daily backups failed on `hostname`" my@email.com

and

/etc/periodic/montly/001.backup:

#!/bin/sh

/usr/local/bin/rsnapshot monthly > /tmp/rsnapshot.out 2>&1 || cat /tmp/rsnapshot.out | mail -s "monthly backups failed on `hostname`" my@email.com

This means I’ll get an e-mail if the backups fail for some reason, and can then inspect the rsnapshot log in /var/log/rsnapshot. On the FreeBSD server I already had an e-mail server configured, but on my laptop and workstation I set up msmtp instead, which is a simple SMTP mailer, and the configuration of the cron jobs is a bit different from FreeBSD.

Result

If I ever mess something up, I’ll have backups from the past seven days to restore from, or from the last monthly backup. And if the HDD crashes, I’ll always have a copy of the latest stuff on my Strongspace account. I’m very happy and it feels good to finally have backups.

Feel free to share your own backup strategy in the comments.

Cheers,
Elvis

Bordering on the Insane — A Story of a Near Collapse

You’ll have to excuse the witty title, but I’ve been working on table borders. Specifically collapsed multi-line borders, properly joined at intersections. It is hard work I tell you. No, really, it’s downright ridiculous.

Some Background

Since a table cell may span multiple rows or columns, along each side of the cell, it may share its border on that side with N neighboring cells, or with the table border. Along each such shared border segment, the neighboring border along that segment must be identified and collapsed with the cell border according to certain rules. The most commonly used rules are those specified in the CSS collapsing border model, sometimes with slight modifications. This is also what I’m aiming for in my implementation.

Lines in Scribus can traditionally be represented by an arbitrary number of lines, each with its own color, width and style, drawn on top of each other, thin over thick. Like this:

A multi-line in Scribus
A multi-line in Scribus

In trying to keep in style, I’d of course like to support these types of lines in my implementation of table borders. This is also supported by competing products such as InDesign.

Borders from different cells, or from the table itself, meeting at an intersection in the table should optionally be joined. Joining is the process of adjusting the start and end points of the border, as well as adjusting the start and end points of the individual lines constituting the border, in order to make a “best effort” join with any other borders meeting at the intersection.

This is where the fun begins. I’ve identified at least these 41 possible cases of joins:

Table Border Join Cases
Table Border Join Cases

The Past ~Two Weeks

In the past two weeks most of my work has been trying to find a joining/painting algorithm that correctly identifies all the cases above and performs the necessary adjustments.

To paint an entire table, the painting algorithm must iterate over all cell edges in the table, and for each edge, iterate over all shared border segments. For each segment, the segment is collapsed with the correct neighboring border. Next, each of the, possibly six, other border segments meeting the segment at its start and end point must also be identified. This means identifying all the cells surrounding the segment and collapsing the appropriate shared border segments between them.

Let’s take a simple case as an example. In the example below we want to paint the top border of the green-tinted cell, which spans two columns. The thin red dotted line represents the underlying table grid.

Painting a Top Border
Painting a Top Border

In the first iteration above, in addition to collapsing the shared border segment between the cell itself and the cell above it, the five border segments coming in to meet it at the two intersections must be identified and collapsed correctly. After that, adjustments for joining can be made to the segment start and end, before the segment is finally painted.

Similarly, in the second iteration, there are four additional collapses that needs to be done before joining adjustments and finally painting can be done.

Needless to say, it’s been quite a chore trying to get this to work. Especially the joining algorithm has been a tough nut to crack. I’ve used up numerous sketch pads trying to figure it out. When working on something like this, pen and paper is invaluable. But, although there are some cases it can’t quite handle in a pleasing way, I think I finally have an approach that will work. I’ve intentionally made the code for collapsing and joining strictly separated from the rest of the code, to ease unit testing.

To not get too complicated the algorithm I’ve settled on imposes a strict painting order — horizontal borders must be painted on top of vertical ones. This means two iteration across the table. Iteration is quite fast though, and besides, I’d rather spend my time optimizing cell accesses on the table than convoluting the joining algorithm with added complexity.

So without further ado, here’s a screenshot of some collapsed joined and non-joined borders on a table in Scribus:

Joined Borders in Scribus
Joined Borders in Scribus

Although there are some bugs in there, I have other fish to fry at the moment, so I’m going to leave painting for a while. And if there’s anyone out there who, after looking at that picture with the 41 join cases I’ve identified, get a brilliant idea for an algorithm that covers them all with a minimal amount of code, then contact me! Please!

That’s all for now. Bye ’til next time!

Scribus Tables GSoC — First Steps

Hi all,
This is the first in a series of blog posts detailing my work with bringing proper support for tables to Scribus as part of Summer of Code 2011. I’ll try to keep the posts in roughly the same format as the reports I’m sending off to my mentor Craig each week. This will make it easier for me to reuse some of the material, saving hacking time :) Each post will be divided into the following sections:

  • Work Report, a report on what I’ve been up to since my last report.
  • Project Status, how I’m standing with regard to the schedule in my project proposal.
  • Upcoming Work, a rough outline of my next steps.

As this is the very first blog post about my project, I’ll start with a little introduction with general information about my project.

Introduction

Tables provide a compact way of representing tabular information in a page layout, and is a common design element in books, magazines and advertisements. Scribus currently has a shortcut for creating an automatically grouped table-like array of text frames. These are not proper tables. To stay competitive, Scribus needs better support for tables. Support that enables the user to create and edit tables as distinct items with their own set of properties as commonly found in other table implementations. This is the rationale for my Summer of Code project, in which I’ll add a new table page item to Scribus.

First glimpse of new tables
First glimpse of new tables

The user should be able to perform common table operations such as inserting rows and columns, merging and splitting cells, formatting borders and backgrounds as well as editing the text of the table. The table will represent the text of the cells using standard Scribus text frames.

More information can be found in my project proposal.

So without further ado, here follows my first report.

Work Report

  • Bootstrapping: Drafted an initial design document including some mockups on the Scribus wiki and discussed the ideas in detail on the scribus-dev mailing list. Also set up a development environment (using Qt Creator this time even though I’m a die-hard Vim junkie).
  • Added a skeleton of a table style. This was way back in April, just to see what work is involved in creating a new type of style. The style is the most basic one imaginable. At the moment I think it has a single property for background color. Instances of the new table style can be managed in the Style Manager. It doesn’t do style inheritance yet, though that should be relatively easy to fix.
  • Added class PageItem_Table, a new page item for tables. The class represents a grid of table cells by keeping a few lists with row / column geometries.
  • Added ability to insert a new PageItem_Table from toolbar or Insert menu followed by mouse dragging. Just like the current “tables”.
  • Added API to PageItem_Table for the basic table operations
    • insert and remove rows and columns,
    • resize rows and column.
  • Added support for merging cells to PageItem_Table as well as an accompanying helper class CellArea used when keeping track of areas of merged cells. The CellArea class also comes with what I think is the very first unit tests in Scribus. The tests are in tests/ and can be runned using make test.
  • Added some rudimentary painting to PageItem_Table, just to be able to see the cells. Still a long way to go for proper table / cell painting, which is kind of non-trivial.
  • As scriptability allows me to experiment with tables before any UI is done, I’ve added the following new scripting methods. These have been added to the old scripter, but should be easy enough to port to ScripterNG later on.
    • createTable(x, y, width, height, numRows, numColumns, ["name"])
    • getTableRows(["name"])
    • getTableColumns(["name"])
    • insertTableRows(index, numRows, ["name"])
    • removeTableRows(index, numRows, ["name"])
    • getTableRowHeight(row, ["name"])
    • setTableRowHeight(row, height, ["name"])
    • insertTableColumns(index, numColumns, ["name"])
    • removeTableColumns(index, numColumns, ["name"])
    • getTableColumnWidth(column, ["name"])
    • setTableColumnWidth(column, width, ["name"])
    • mergeTableCells(row, column, numRows, numColumns, ["name"])

Project Status

The goal in the schedule for this first week was simply Data structures for tables. Intentionally a quite moderate goal, as I didn’t know how long it would take to get up to speed. So unsurprisingly, I think I’m a little ahead of schedule. Worth noting though is that the tables are currently only skeletal in their nature; they hold no content and are simply a grid of empty cells. This is all according to plan though, and integration with text frames is scheduled for later.

Upcoming Work

The goals for next week according to the schedule is

  • Basic table layout with fixed column widths and mock content in cells.
  • Insertion/removal of rows/columns.
  • Basic drawing of table.

but as at least some of this has already been done, I think I’ll start looking at other things as well. Right now I’m looking at proper table / cell border painting, as well as fixing a couple of bugs in the code I have so far.

I have an installation of Adobe InDesign CS5.5 running in VirtualBox and I’ve been looking at what border model they’re using and how they’re doing border conflict resolution. It seems to be a variant of the CSS2 collapsing border model, which is probably what I’ll try to go for initially in Scribus as well.

That’s all for now folks! I’ll conclude with a little screencast showing me playing around with a table using the new scripting methods. I highly recommend you download the source OGV instead of watching it here, as the quality turned out quite bad in the blip.tv conversion.

So it seems my…

summer of 2011 will come in a distinct 2009 flavor. Can’t wait to get started ;)

I’ll probably blog my progress here. What do you guys think; OK for me to have it aggregated on p.k.o? Scribus KDE-ish enough for you?


PS. For those of you who didn’t understand or can’t be bothered to click the links; I’ve been accepted into GSoC this year to improve the tables support of Scribus. You might remember me doing the same for KWord as part of GSoC back in 2009. DS.