Serving Mercurial over SSH

So…

I’ve been deploying our big Django application to all of our servers using a pretty slick setup with Fabric and rsync.

This worked fine when I was the only developer, working on my local machine and pushing to a Mercurial repository on one of our internal Ubuntu boxes. Since it was all local, I just used the server’s Apache setup and mod_wsgi and didn’t worry about security too much. The Linux box is completely firewalled off from the Intertubes.

However, as we get more people working on the code, and as we deploy to more servers, having the ability to update to and from our Mercurial repository is becoming more important. Just ask Jeff(rey), whose template changes I clobbered this afternoon.

Since the shared hosting on which I wanted to set up my Mercurial repositories doesn’t have mod_wsgi, nor is it really safe to put a ‘foreign’ module like that into a cPanel setup, I had to find another way to serve mercurial securely.

Also, since we’re finding that Apache and its threading model are consuming waaaaaaaayyyy too much memory under load, we’re moving toward lighter weight, single purpose servers for everything anyway.

So…I found mercurial-server.

It gives secure, tightly controlled Mercurial access over a simple SSH connection.

After I found the `.deb` file and was able to use `dpkg` to actually install it, things went pretty smoothly.

Setup instructions are pretty straightforward with the only part that confused me a little was extracting the key from SSH Agent with `ssh-add -L`. I wasn’t using SSH Agent so the directions didn’t work but once I figured out that al I needed in-hand was the public key, I was on my way.

The repositories are kept squirreled away in kind of an odd location that’s not mentioned anywhere, but that doesn’t seem to be much of an issue as long as that directory tree (`/var/lib/mercurial-server/repos/`) is getting backed up, I’m fine with wherever it wants to put it. The reason for that location, far as I can figure, is that the /var/ tree is supposed to be for things:

/var/ Variable files—files whose content is expected to continually change during normal operation of the system—such as logs, spool files, and temporary e-mail files. Sometimes a separate partition.

At any rate, it took a couple of hours to get set up right and, while speeds don’t seem to be quite as good as they were running under Apigche, it’ll work just fine for my automated setup.

Now, to figure out how to get hooks to do all the necessary pushing and pulling for me…

To see how to get this running on CentOS, see my next blog post. What a PITA!

JavaScript Command Line Shell OS X

So…

I was writing a whole bunch of JavaScript recently and was getting incredibly tired of having to refresh the browser to see if things were working.

Yes, I have FireBug and such but what I really want is a command-line JavaScript interpreter like my iPython.

To make a long story short, none of the instructions I found for building SpiderMonkey worked on OS X and I couldn’t find a binary so I went to plan B: Rhino.

Download and unzip. What you’ll get is a directory named `rhino1_7R2`.

So, once you have that, just go get jLine. It’s a single .jar file, stick that into your `rhino1_7R2` directory. Mine is named `jline-0_9_5.jar`.

Make sure you’re in your `rhino1_7R2` directory where you should see something like this if you do an `ls`:

(~/rhino1_7R2)# ls *.jar
-rw-r--r--@ 1 ssteiner  staff    46424 Dec  6 09:00 jline-0_9_5.jar
-rw-r--r--  1 ssteiner  staff  1164702 Mar 22  2009 js-14.jar
-rw-r--r--  1 ssteiner  staff   871260 Mar 22  2009 js.jar

then type the oh-so-completely-obvious command:

(~/rhino1_7R2)# java -classpath js.jar:jline-0_9_5.jar jline.ConsoleRunner
 org.mozilla.javascript.tools.shell.Main

You’ll see the `js>` prompt and there you are; a full JavaScript command line to try things out in.

I got some great help from this article when I was first starting this exploration and he goes into mixing Java and JavaScript code a little bit which is of no interest to me, at this point.

Sure is nice to have a command line to whack around to try out regular expressions and such though…

Installing Parallels Tools on Ubuntu

I’ve just downloaded the latest Parallels for OS X and am firing up a couple of Ubuntus for local development.

After an absolute nightmare with OS X Python support (trying to get 2.6.2 to build virtual machines and having everything go completely, unusably out to lunch), I’m switching to doing all my development on an Ubuntu virtual machine.

Since I can map an Ubuntu drive through Expandrive to look just like a local drive, and use iTerm to connect to it to run things and a local terminal to connect to have all my OS X tools available, this works very, very well.

Unfortunately, the instructions for setting up Parallels Tools are very vague and in a dialog that has to be dismissed before you can actually do what it says, I’m writing this post.

  • Start by starting the VM, and choosing Virtual Machine -> Install Parallels Tools from the menu
  • Go to a terminal prompt on the machine and run:
    	# sudo mount /dev/cdrom /media/cdrom
  • In the same terminal, run:
    	# sudo /media/cdrom/install
  • Restart the system with:
    	# sudo shutdown -r now

Follow the instructions, restart the virtual machine, and Parallels Tools will be running.

Unfortunately, far as I can tell, it still doesn’t support cut & paste or automatic mouse capture/uncapture so I’m not sure what, exactly, it is supposed to do…

Snow Leopard vs. virtualenv – easy_install virtualenv==dev != latest

So there I was, merrily plooking along with my various Python projects and had occasion to make a new virtual environment using `mkvirtualenv` from Doug Hellmann’s excellent virtualenvwrapper.

And it hung.

I ctrl-C’d out after a few minutes and tried again. Hung.

Figured it might be a Snow Leopard thing, so I did a quick:

	# easy_install virtualenv==dev 

Figuring that’d get me the latest version.

Same thing.

Poked around in the source looking for a clue for a minute, then did the obvious; Googled for the error message.

Which lead me to this post.

Turns out that easy_install grabs from a subversion repository that’s not quite up to date with the new code up on bitbucket.

To quote that post:

Turns out that triggers an install from the Subversion repository at colorstudy.com which *doesn’t* have the Snow Leopard fix, but is also labeled as version 1.3.4dev. So I guess I was chasing my tail a bit.
I should have done this:

> easy_install http://bitbucket.org/ianb/virtualenv/get/tip.zip

That gets the virtualenv with fix I was after, and indeed does work.

So, the lesson is: in this time of projects moving off of their own little subversion repositories and onto bitbucket and github, and easy_install, out of the box, supporting only subversion and CVS (which I won’t dignify with a link), check your assumptions about which version of what you’ve got installed; sometimes things LIE!

Hopefully, this will save someone else some time and trouble.

P.S.
Speaking of subversion, I’m hoping to get to use the setuptools Mercurial plugin working sometime soon since most of my new projects are on Mercurial, but I’ll probably wait until I convert over to Distribute which may get it built in sooner rather than later.

Create tar.gz of current directory and all subdirectories

Just another quick note to myself since I always forget to remember this.

To compress the whole of the entire current directory into a compressed tar/gz file:

	# tar cvz [--remove-files] -f mytarfile.tgz .

c == create
v == verbose
f == output file
z == compress
–remove-files == optionally remove files after adding them to the archive
. == current directory contents

To see what you’ve done:

	# tar --list -f mytarfile.tgz

I always forget to use the -f switch so it just sits there waiting for some magic input to arrive. What, am I supposed to type the contents of an archive for it to show me?

I always use tar vs. zip since tar remembers file permissions whereas zips do not (that may have changed, don’t care).

Inconsolata Font, Panic Sans Font

So, I’ve been using whatever the system provided for fonts forever. The other day, I was searching for something else and came upon this post.

If you spend a lot of time in a text editor, do yourself a favor and try Inconsolata.

Looks great, prints well, good all around.

Reading through the comments, someone mentioned Panic Sans.

I have Coda, and I could use the font in Coda, but the font didn’t show up in the font dialog in any other program.

Turns out it’s buried inside the Coda application bundle.

Go to Coda.app in Finder, Right-Click->Show Package Contents, then Spotlight “Panic Sans” by file name.

Double the click and install it when it comes up in Font Book and it’ll be available everywhere.

I haven’t decided between Inconsolata and Panic Sans but I’m liking them both better than Monaco which I’ve been using for years.

The WSSW Stack

Choosing The Stack

Ok, so I’ve been plooking around with various web frameworks, even languages, for a couple of years now.

Now, while starting WebSauce Software for real, it’s time to choose a standard toolset. This is what we are going to use to produce our software until further notice.

Unless there’s a compelling reason to change, this is what we’re using.

If something great comes along to replace a component then fine, but it’ll have to be pretty damn good for us to switch.

If it’s great, we’ll switch.

Adapt or die!

First a little history.

We got into the web business about 7 years ago after 25 years of general purpose contract programming which overlapped with about 10 years of software publishing.

I started consulting in about 1982, started publishing software in about 1986, stopped publishing software in 1994, and retired from the software business, sort of, in 1995, had my first son in 2002, and went back to work in 2004-ish.

I did some consulting between 1995 and 2004, but only a handful of really complex, challenging jobs. I was not making a living, I was just taking on work I liked and wanted to do.

When I went back to work, I didn’t know exactly what type of work would be coming up and I wasn’t too worried about it. I’ve always managed to keep busy.

Unfortunately, I had been out of the loop for almost 10 years so most of my old consulting contract clients were gone, companies changed hands, engineers at those companies moved around to parts unknown. In short, I didn’t really have any contacts any more.

So, I rented an office and hung my shingle out to see what would happen.

People kept asking me if we did websites.

So, I said we did.

Now, it’s not that we hadn’t done websites before that for ourselves or for customers, but we weren’t in the business of making websites for other people.

So, now we were, and we did.

Lots of them.

We grew, hired people, had clients, had a stream of new clients, a few big clients, I wrote some nice tools for in-house use that made us more efficient than other companies, we learned the web development business and everything was hunky-dunky.

Except…

I hate making new websites for people who don’t already have them.

They have unrealistic expectations of what the site can do for them and especially, how much it should cost. At least people with existing sites have an idea what things cost, and know what the site is doing or not doing for them.

Improving an existing site is way better, for us. Less friction, better
results all’round.

What I do like…

Fixing existing sites

Fixing up an existing site is a blast. We get to leverage all of our cool tools and, because of those tools, we’re very efficient at it. Because of our efficiency, clients get a better deal that they did from their prior company which makes us look good and, since almost everything is automated, we make good profit margins.

Best part? I get paid to spend time ploinking on the tools we use to do customer jobs more efficiently which is the most fun for me.

Doing SEO

Getting sites to rank well in the Search Engines, making sure that their customers can do useful things with their website, and generally helping our customers serve their customers better.

My software engineering background has allowed me to write some tools that do things in this area that nobody else has. We’ll be publishing some of them soon. We’ll let you know ;-).

Writing web applications

Things that are kind of like desktop applications but run in a browser and do things that are appropriately web based. We’ve done SalesForce.com integration, custom database editing applications for real estate brokers, inventory control and management against existing, legacy databases that just need a new view to be more useful than they already are, all kinds of stuff. Love it.

How I’ve Written All This Stuff

I’ve written utilities for doing the repetitive parts of SEO and also written web applications for various purposes for clients and for in-house needs.

I was always hunting for the best development toolset both for client applications and for our own internal tools.

I’ve gone through a lot of tools.

So I tried…in no particular order

and God knows how many other frameworks, version control systems, WSGI components, templating languages, and chunks and parts of various solutions.

So…I’ve finally settled

So, after all that trial and error, here’s my toolset.

This is what I’m using from now on unless there’s a compelling reason to use something else. Most of the bigger tools (Django, for example) have or are developing plug-in parts for things like the templating system so these choices are not as rigid as having this list might imply.

Linux Distribution: Ubuntu

I’ve used just about every Linux distribution at one time or another, we host lots of sites on the Centos series, I think one of our in-house boxes is Suse. Then I started using Ubuntu since it seemed to be the one most of the documentation for the tools I was using was written for. I figured there must be some reason for that since it was just too pervasive to be a coincidence. Not a coincidence. It just works better. All of our cloud servers are now fired up with Ubuntu 9.04 server configuration and I run Kubuntu (I absolutely hate Gnome, love KDE). I’m envious of the MacOS-X Aqua theme, only for Gnome so far, but it’s not enough of a reason to switch to Gnome.

Ubuntu has been rock solid, and apt-get blows away any other system package tool I’ve used (yum, nasty RPMs, etc.).

Language: Python

The language I always come back to. I’ve tried other languages. Seems like I’ve tried every other language at one time or another. Last time I counted it was, like, 40 or something including dialects of Basic, Pascal, C, C++, Delphi, various Assembly languages, Perl, Ruby, Awk, SmallTalk, Lisp, Sed, Haskell, and many, many others I can’t even remember. I don’t remember who said it but Python really is executable pseudo code

VCS: Mercurial (hg)

Up until a few months ago, we were Subversion users. I feel dirty even saying it, now. We used Perforce for one job but I hated it the whole time. I always found Subversion annoying; especially trying to merge branches.

The centralized repository always gave me an uncomfortable feeling I never identified until I started using Git on an Open Source project I was working on.

The first time I did a merge, I was hooked. It was painless and it wasn’t a trivial merge either. I had to manually resolve one conflict out of 30 or so changes. It took five minutes. It would have taken all day in Subversion and I would have been swearing the whole time. I was leaning toward Git, not having used any of the other likely suspects much until this announcement.

Then, there’s Google’s support which double sealed the deal.

Since the main Python repository is going to be Mercurial, and since that will likely drive adoption on other projects that have yet to move out of Subversion, and since Mercurial is written in Python, it would be silly to use anything else since there’s really little obviously superior about any other DVCS.

Mercurial is also sure to get lots of loving attention and will pass Git in short order in any area where it’s currently lagging. Fortunately, Git, Mercurial, and Bazaar are similar enough that it’ll be easy enough to switch around when needed.

WebSauce’s projects will all be DVCS’d in Mercurial and I’ll document the setup as soon as I get around to it. The setup, that is…

Desktop App Development: Cocoa/Objective-C

I tried writing my first OS X Application for publication using Python and PyObjc. I had a working prototype but, even with expert help, couldn’t get it to run anywhere but my development system. Next app is pure Objective-C and, if I need Python for something, I’ll run it as an external process and work on getting the results back some way other than being running inside the main application space.

Web Framework: Django

I may not like some of the parts of the Django stack so much but it all hangs together well and, if I get sufficiently dissatisfied with any particular part, I’m sure there will be a way to “fix” it on my own checkout and submit a patch. I’m pretty sure most of the Django pieces are pluggable to some extent and, where they’re not, it would be good of me to help make them so. That’s what Open Source is all about, right?

Web ToolKit: Twisted

Twisted does so many things, and our applications need so many of them, that it’d be silly not to use the grandfather of all things Python and Web.

Sure, it’s a little hard to wrap your head around in the beginning, and there are parts that are dark, deep, and mysterious, but I’ve been hanging around on the mailing list and IRC channel and I’m confident that if I run into a problem, and do my research before asking for help, that I’ll be able to get any problem solved in relatively short order.

Because so much of the rest of our apps require Twisted services, we’re going to run our Django app using Twisted’s WSGI unless we run into problems, Then we’ll fall back to eiter CherryPy’s WSGI, Apache’s mod_python, Apache’s mod_wsgi. Whatever, not a big deal.

Documentation Language: Restructured Text

The documentation format of Python that can be easily converted to everything else.

It’s human-readable in source form, intuitive, and is everywhere in all the tools I use.

No brainer.

Other Tools

The stack really isn’t worth anything unless you can deploy it.

For that, I’m relying on several other Python based tools:

Paste

I’m only using the directory template creation of Paste. Paste is for the most part, overgrown and under-focused but the directory templating works well enough for now.

virtualenv, virtualenvwrapper

These allow me to set up an isolated Python environment in which to run my applications. Keeps all the cruft out of the system and gives me an attainable target to deploy.

zc.buildout

Allows creation of a completely self-contained app. Virtualenv’s great for development, but this wraps it all up in a one-stop-shopping bundle.

fabric

Makes deployment as simple as writing a Python script that does what you want to distribute an application to wherever you want to deploy it.

Sphinx

The documentation tool used on the Python project itself. You can set up a documentation structure in one command, write your docs in reStructuredText, and have it in html, latex, and several other formats in a flash.

github/BitBucket/LaunchPad

Not really part of the deployment stack but from having worked on several open source projects on github with git, I think it’s about the best there is right now. I’m still interested in looking at BitBucket and I’m contributing to a few projects there as well but Github seems to be more mature and has a much more informative and useful interface. LaunchPad is very ambitious, and seems well thought out and pretty all-encompasing. Unfortunately, the only backend it supports is Bazaar. Yuck.

Basecamp

We’ve been using Basecamp for a while now for project management. It’s not perfect but it is the best shared system we’ve found. We’ve tried Google Docs and got addicted to shared documents but the rest of the system doesn’t provide any project management functionality so things tended to get lost in there since there was no way to indicate what was to be done next. Basecamp also has shared documents (Writeboards) and also ToDo Lists and Milestones which make it possible to keep a project moving.

FogBugz

We’re currently using FogBugz to track our bugs in the OS X product that we’re untangling the Python code from and it really is a great bug tracking system.

We’ve been focused mostly in BaseCamp so it will be interesting to see how well they integrate or whether we move to another system for this functionality. An obvious choice would be Trac and, with the buildout script, maybe it won’t be so abominable to install.

For now, that’s it.

I’ll be updating this as I update the toolset but this is it, for now…

Trial/demo Software

I download tons of software. I’m always looking for something to help with my workflow, make me more productive, help me keep on top of various types of works in progress, anything; I’m a productivity software junkie.

Many of the softwares I download are useless. And now I also sound French.

I’m using two different trials, with two different approaches:

Relaunch

Relaunch saves you time by taking Snapshots of which applications you are using, and starts them back up for you. Think of it as a launcher on steroids that lets you switch between work contexts with one click.

And:

MercuryMover

MercuryMover reduces the friction you feel when you use your Mac. With MercuryMover, you can move and resize windows on your Mac from the keyboard, positioning them precisely where you want. By shunning the slow and imprecise mouse, MercuryMover empowers you to work faster and play more.

The two softwares have different approaches to the “trial period.”

Relaunch is strictly time limited.

MercuryMover is “number of actions” limited.

I installed them both at about the same time.

Relaunch’s trial period has expired.

I have 67 more moves left in Relaunch.

MercuryMover, once I figured out how it worked, and what it’s good for, has become more and more integral to my daily work flow. It lets me instantly “bring the current window to my main monitor, and to the middle of the screen, filling up the screen completely.” It took a couple of weeks to remember to use it and, once I did, it got more and more indispensable.

Relaunch, according to the marketing blurb on their site, will do even more; it’ll restore all of my apps when working on a project to the same state as they were when I closed the project.

Problem is, I’ve not had time to set it up on a project to see how awesome it is.

And, my trial period has expired.

So…MecuryMover is going to have me as a customer; the “number of uses” worked — the more I used it, the more I liked it and I’m sold.

Relaunch never got used and, for however good it might have been, my trial period has expired and I may never know how good it might have been.

Of course, I’m going to send this to both companies; hopefully it will help Relaunch make its policy more clear and let users know when their trial is coming up.

Update: Richy at Wired Up Fired Up (makers of Relaunch) sent me a license key and we had a good e-mail chat about the trial policy. I’ve been using it a bit today and it seems to work pretty well. More if it makes it into my daily workflow.

Sorry to say, I haven’t heard from MercuryMover’s author(s). I filed a crash report with a detailed step-by-step but haven’t heard back on either issue.

Using reStructuredText with TextMate’s Browser Preview

I’m working on several open and not so open source projects in Python these days and am using Sphinx for documentation.

TextMate has a Browser Preview window with the ability to pre-process the editor contents through a script before rendering.

The Sphinx standard input format is reStructuredText as handled by Docutils.

You can install Sphinx with:

	# sudo easy_install docutils sphinx

To get TextMate to pre-process the reStructuredText properly before display, check the “Pipe text through” checkbox in the Web Preview window and put the rst2html.py into the textbox for the name of the script.

Picture 2.png

There will be some issues with “Unknown interpreted text role ‘data’” type things but you’ll pretty well be able to see what it’ll look like.

I may explore making this more accurate and flexible but, for now, this’ll do.

htop — prettier top command for *nix

htop is a prettier, more functional version of the *nix top command.

It adds cool things like a scrollable process display so you can see all processes, not just the screen’s worth you can see with old top.

# apt-get install htop

will do it on Ubuntu.