Serving Mercurial over SSH

So…

I’ve been deploying our big Django application to all of our servers using a pretty slick setup with Fabric and rsync.

This worked fine when I was the only developer, working on my local machine and pushing to a Mercurial repository on one of our internal Ubuntu boxes. Since it was all local, I just used the server’s Apache setup and mod_wsgi and didn’t worry about security too much. The Linux box is completely firewalled off from the Intertubes.

However, as we get more people working on the code, and as we deploy to more servers, having the ability to update to and from our Mercurial repository is becoming more important. Just ask Jeff(rey), whose template changes I clobbered this afternoon.

Since the shared hosting on which I wanted to set up my Mercurial repositories doesn’t have mod_wsgi, nor is it really safe to put a ‘foreign’ module like that into a cPanel setup, I had to find another way to serve mercurial securely.

Also, since we’re finding that Apache and its threading model are consuming waaaaaaaayyyy too much memory under load, we’re moving toward lighter weight, single purpose servers for everything anyway.

So…I found mercurial-server.

It gives secure, tightly controlled Mercurial access over a simple SSH connection.

After I found the `.deb` file and was able to use `dpkg` to actually install it, things went pretty smoothly.

Setup instructions are pretty straightforward with the only part that confused me a little was extracting the key from SSH Agent with `ssh-add -L`. I wasn’t using SSH Agent so the directions didn’t work but once I figured out that al I needed in-hand was the public key, I was on my way.

The repositories are kept squirreled away in kind of an odd location that’s not mentioned anywhere, but that doesn’t seem to be much of an issue as long as that directory tree (`/var/lib/mercurial-server/repos/`) is getting backed up, I’m fine with wherever it wants to put it. The reason for that location, far as I can figure, is that the /var/ tree is supposed to be for things:

/var/ Variable files—files whose content is expected to continually change during normal operation of the system—such as logs, spool files, and temporary e-mail files. Sometimes a separate partition.

At any rate, it took a couple of hours to get set up right and, while speeds don’t seem to be quite as good as they were running under Apigche, it’ll work just fine for my automated setup.

Now, to figure out how to get hooks to do all the necessary pushing and pulling for me…

To see how to get this running on CentOS, see my next blog post. What a PITA!

New project: Denormalizer

So…

I was working on South integration for a project the other day and realized that I wanted to denormalize some things.

Like, keep the number of pages in the site update in the update record, the number of errors, title in the site update page record rather than sucking it from the saved HTML, that kind of thing.

And, I realized that, when things go wrong, there should be an automated way of fixing/updating the denormalized data.

Hence, the Denormalizer.

The basic idea is to specify the denormalized pieces of data in your database in a format resembling a South migration. You provide rules that say “run this query on that database, and stick the results over here.” Every denormalized piece of data has a rule for creating it.

Then, whenever things get hosed, or you’re just feeling insecure about the state of the data, put it into “maintenance mode” and have the Denormalizer go and count things up, extract other things (like the title, above) and fix up the denormalized fields.

Any time you want to make a new denormalized data chunk, just specify the rules for it, and off you go…it’ll be just like it was always there being dutifully updated along with the “normal” fields.

Pretty cool idea; wonder if I’ll ever get to implement it…