How to Clone a VirtualBox Virtual Machine on OS X

So…

I’m working on my automatic deployment scripts and decided to do some work using local VirtualBox VM’s rather than doing everything in the cloud. The cloud part all works just fine, so there’s no point wasting bandwidth (and money) proving it out a hundred times.
[Continue Reading...]

eyeball scripts

June 26, 2010 — eyeball scripts
On the #linode channel at irc.oftc.net; you heard it here first…
[Continue Reading...]

SSH works, SFTP doesn’t on Ubuntu 10.04

So…

I’m setting up one of my new VPSes (Linode Rocks!) and I was having a weird thing where I could ssh in with root, but not sftp. Other users could do both just fine.
[Continue Reading...]

Serving Mercurial over SSH

So…

I’ve been deploying our big Django application to all of our servers using a pretty slick setup with Fabric and rsync.

This worked fine when I was the only developer, working on my local machine and pushing to a Mercurial repository on one of our internal Ubuntu boxes. Since it was all local, I just used the server’s Apache setup and mod_wsgi and didn’t worry about security too much. The Linux box is completely firewalled off from the Intertubes.

However, as we get more people working on the code, and as we deploy to more servers, having the ability to update to and from our Mercurial repository is becoming more important. Just ask Jeff(rey), whose template changes I clobbered this afternoon.

Since the shared hosting on which I wanted to set up my Mercurial repositories doesn’t have mod_wsgi, nor is it really safe to put a ‘foreign’ module like that into a cPanel setup, I had to find another way to serve mercurial securely.

Also, since we’re finding that Apache and its threading model are consuming waaaaaaaayyyy too much memory under load, we’re moving toward lighter weight, single purpose servers for everything anyway.

So…I found mercurial-server.

It gives secure, tightly controlled Mercurial access over a simple SSH connection.

After I found the `.deb` file and was able to use `dpkg` to actually install it, things went pretty smoothly.

Setup instructions are pretty straightforward with the only part that confused me a little was extracting the key from SSH Agent with `ssh-add -L`. I wasn’t using SSH Agent so the directions didn’t work but once I figured out that al I needed in-hand was the public key, I was on my way.

The repositories are kept squirreled away in kind of an odd location that’s not mentioned anywhere, but that doesn’t seem to be much of an issue as long as that directory tree (`/var/lib/mercurial-server/repos/`) is getting backed up, I’m fine with wherever it wants to put it. The reason for that location, far as I can figure, is that the /var/ tree is supposed to be for things:

/var/ Variable files—files whose content is expected to continually change during normal operation of the system—such as logs, spool files, and temporary e-mail files. Sometimes a separate partition.

At any rate, it took a couple of hours to get set up right and, while speeds don’t seem to be quite as good as they were running under Apigche, it’ll work just fine for my automated setup.

Now, to figure out how to get hooks to do all the necessary pushing and pulling for me…

To see how to get this running on CentOS, see my next blog post. What a PITA!

Enable Remote Access to MySQL

I haven’t set MySQL up in years; it’s always on the server, and I guess we never try to access it from outside.

So… I had completely forgotten about allowing access. Fortunately, it only took a few quick Googles before I had the answer and, even though it’s in a million other places, I’m going to add it here since I want to be able to use this blog as my reference.

Edit, on Ubuntu, with:

	$ sudo vim /etc/mysql/my.cnf

And make it look like this:

[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
language = /usr/share/mysql/English
bind-address = 127.0.0.1
bind-address = you.re.ip.add
# skip-networking

In 5.1.x it seems that `skip-networking` has been dropped as the default and `bind-address` is 127.0.0.1 by default. I just left that in place and added my IP address. I’m not sure if that’s right, but it works.

I found the full reference here. I really like that ‘nixCraft’ site; I seem to end up there a lot.

VirtualBox on OS X 10.6 With Ubuntu 9.10

So…I wanted to run VirtualBox with Ubuntu 9.10 as my testbed for some new stuff. I had been mostly happy using Parallels version 4.0, mostly.

To make a long story short, Ubuntu 9.10 crashed Parallels 4.x and I wasn’t really interested in upgrading Parallels again since their Ubuntu support has always given me troubles.

So I downloaded Sun’s VirtualBox.

To make a long story short, forget all the dhcpserver stuff related to Host-Only ethernet ports. I got it to work; once. Then, for hours, tried every combination I could find on the InterWebs and got nowhere.

How to Actually Make It Work

Make the second port (from the GUI) a Host-Only adapter, take defaults for everything else.

Go to your Mac command line and type:

# VBoxManage list hostonlyifs

Note the IP address of the 192.168.XXX.1 adapter — that’s your gateway and gives the 3 octet for the addresses below. Mine was 56 so I had 192.168.56.1 for my gateway.

Then go sudo vim /etc/network/interfaces and add this to the end:

auto eth1
iface eth1 inet static
address 192.168.XXX.10
netmask 255.255.255.0
gateway 192.168.XXX.1

Restart the networking with:

# sudo /etc/init.d/networking restart
# ifconfig

You should see your second port all set up, should be able to ping it from your Mac, etc.

What a pain! Hopefully this saved you some; it took hours to figure this out.

Let me know if this helped; I’d love to hear.

S

Ubuntu, new dhcp lease, please

So…I just fired up a new virtual server, and wanted to have my router assign a fixed IP address to it.

Got it all setup in my router and tried:

	# ifconfig eth0 dhcp status

	dhcp: unknown host
	ifconfig: `--help' gives usage information.

I tried all variations and eventually just ended up doing a:

	# sudo /etc/init.d/networking restart

And all was well..

This definitely comes under the “things I’m just going to forget” category of blogging reasons.

Installing Parallels Tools on Ubuntu

I’ve just downloaded the latest Parallels for OS X and am firing up a couple of Ubuntus for local development.

After an absolute nightmare with OS X Python support (trying to get 2.6.2 to build virtual machines and having everything go completely, unusably out to lunch), I’m switching to doing all my development on an Ubuntu virtual machine.

Since I can map an Ubuntu drive through Expandrive to look just like a local drive, and use iTerm to connect to it to run things and a local terminal to connect to have all my OS X tools available, this works very, very well.

Unfortunately, the instructions for setting up Parallels Tools are very vague and in a dialog that has to be dismissed before you can actually do what it says, I’m writing this post.

  • Start by starting the VM, and choosing Virtual Machine -> Install Parallels Tools from the menu
  • Go to a terminal prompt on the machine and run:
    	# sudo mount /dev/cdrom /media/cdrom
  • In the same terminal, run:
    	# sudo /media/cdrom/install
  • Restart the system with:
    	# sudo shutdown -r now

Follow the instructions, restart the virtual machine, and Parallels Tools will be running.

Unfortunately, far as I can tell, it still doesn’t support cut & paste or automatic mouse capture/uncapture so I’m not sure what, exactly, it is supposed to do…

The WSSW Stack

Choosing The Stack

Ok, so I’ve been plooking around with various web frameworks, even languages, for a couple of years now.

Now, while starting WebSauce Software for real, it’s time to choose a standard toolset. This is what we are going to use to produce our software until further notice.

Unless there’s a compelling reason to change, this is what we’re using.

If something great comes along to replace a component then fine, but it’ll have to be pretty damn good for us to switch.

If it’s great, we’ll switch.

Adapt or die!

First a little history.

We got into the web business about 7 years ago after 25 years of general purpose contract programming which overlapped with about 10 years of software publishing.

I started consulting in about 1982, started publishing software in about 1986, stopped publishing software in 1994, and retired from the software business, sort of, in 1995, had my first son in 2002, and went back to work in 2004-ish.

I did some consulting between 1995 and 2004, but only a handful of really complex, challenging jobs. I was not making a living, I was just taking on work I liked and wanted to do.

When I went back to work, I didn’t know exactly what type of work would be coming up and I wasn’t too worried about it. I’ve always managed to keep busy.

Unfortunately, I had been out of the loop for almost 10 years so most of my old consulting contract clients were gone, companies changed hands, engineers at those companies moved around to parts unknown. In short, I didn’t really have any contacts any more.

So, I rented an office and hung my shingle out to see what would happen.

People kept asking me if we did websites.

So, I said we did.

Now, it’s not that we hadn’t done websites before that for ourselves or for customers, but we weren’t in the business of making websites for other people.

So, now we were, and we did.

Lots of them.

We grew, hired people, had clients, had a stream of new clients, a few big clients, I wrote some nice tools for in-house use that made us more efficient than other companies, we learned the web development business and everything was hunky-dunky.

Except…

I hate making new websites for people who don’t already have them.

They have unrealistic expectations of what the site can do for them and especially, how much it should cost. At least people with existing sites have an idea what things cost, and know what the site is doing or not doing for them.

Improving an existing site is way better, for us. Less friction, better
results all’round.

What I do like…

Fixing existing sites

Fixing up an existing site is a blast. We get to leverage all of our cool tools and, because of those tools, we’re very efficient at it. Because of our efficiency, clients get a better deal that they did from their prior company which makes us look good and, since almost everything is automated, we make good profit margins.

Best part? I get paid to spend time ploinking on the tools we use to do customer jobs more efficiently which is the most fun for me.

Doing SEO

Getting sites to rank well in the Search Engines, making sure that their customers can do useful things with their website, and generally helping our customers serve their customers better.

My software engineering background has allowed me to write some tools that do things in this area that nobody else has. We’ll be publishing some of them soon. We’ll let you know ;-).

Writing web applications

Things that are kind of like desktop applications but run in a browser and do things that are appropriately web based. We’ve done SalesForce.com integration, custom database editing applications for real estate brokers, inventory control and management against existing, legacy databases that just need a new view to be more useful than they already are, all kinds of stuff. Love it.

How I’ve Written All This Stuff

I’ve written utilities for doing the repetitive parts of SEO and also written web applications for various purposes for clients and for in-house needs.

I was always hunting for the best development toolset both for client applications and for our own internal tools.

I’ve gone through a lot of tools.

So I tried…in no particular order

and God knows how many other frameworks, version control systems, WSGI components, templating languages, and chunks and parts of various solutions.

So…I’ve finally settled

So, after all that trial and error, here’s my toolset.

This is what I’m using from now on unless there’s a compelling reason to use something else. Most of the bigger tools (Django, for example) have or are developing plug-in parts for things like the templating system so these choices are not as rigid as having this list might imply.

Linux Distribution: Ubuntu

I’ve used just about every Linux distribution at one time or another, we host lots of sites on the Centos series, I think one of our in-house boxes is Suse. Then I started using Ubuntu since it seemed to be the one most of the documentation for the tools I was using was written for. I figured there must be some reason for that since it was just too pervasive to be a coincidence. Not a coincidence. It just works better. All of our cloud servers are now fired up with Ubuntu 9.04 server configuration and I run Kubuntu (I absolutely hate Gnome, love KDE). I’m envious of the MacOS-X Aqua theme, only for Gnome so far, but it’s not enough of a reason to switch to Gnome.

Ubuntu has been rock solid, and apt-get blows away any other system package tool I’ve used (yum, nasty RPMs, etc.).

Language: Python

The language I always come back to. I’ve tried other languages. Seems like I’ve tried every other language at one time or another. Last time I counted it was, like, 40 or something including dialects of Basic, Pascal, C, C++, Delphi, various Assembly languages, Perl, Ruby, Awk, SmallTalk, Lisp, Sed, Haskell, and many, many others I can’t even remember. I don’t remember who said it but Python really is executable pseudo code

VCS: Mercurial (hg)

Up until a few months ago, we were Subversion users. I feel dirty even saying it, now. We used Perforce for one job but I hated it the whole time. I always found Subversion annoying; especially trying to merge branches.

The centralized repository always gave me an uncomfortable feeling I never identified until I started using Git on an Open Source project I was working on.

The first time I did a merge, I was hooked. It was painless and it wasn’t a trivial merge either. I had to manually resolve one conflict out of 30 or so changes. It took five minutes. It would have taken all day in Subversion and I would have been swearing the whole time. I was leaning toward Git, not having used any of the other likely suspects much until this announcement.

Then, there’s Google’s support which double sealed the deal.

Since the main Python repository is going to be Mercurial, and since that will likely drive adoption on other projects that have yet to move out of Subversion, and since Mercurial is written in Python, it would be silly to use anything else since there’s really little obviously superior about any other DVCS.

Mercurial is also sure to get lots of loving attention and will pass Git in short order in any area where it’s currently lagging. Fortunately, Git, Mercurial, and Bazaar are similar enough that it’ll be easy enough to switch around when needed.

WebSauce’s projects will all be DVCS’d in Mercurial and I’ll document the setup as soon as I get around to it. The setup, that is…

Desktop App Development: Cocoa/Objective-C

I tried writing my first OS X Application for publication using Python and PyObjc. I had a working prototype but, even with expert help, couldn’t get it to run anywhere but my development system. Next app is pure Objective-C and, if I need Python for something, I’ll run it as an external process and work on getting the results back some way other than being running inside the main application space.

Web Framework: Django

I may not like some of the parts of the Django stack so much but it all hangs together well and, if I get sufficiently dissatisfied with any particular part, I’m sure there will be a way to “fix” it on my own checkout and submit a patch. I’m pretty sure most of the Django pieces are pluggable to some extent and, where they’re not, it would be good of me to help make them so. That’s what Open Source is all about, right?

Web ToolKit: Twisted

Twisted does so many things, and our applications need so many of them, that it’d be silly not to use the grandfather of all things Python and Web.

Sure, it’s a little hard to wrap your head around in the beginning, and there are parts that are dark, deep, and mysterious, but I’ve been hanging around on the mailing list and IRC channel and I’m confident that if I run into a problem, and do my research before asking for help, that I’ll be able to get any problem solved in relatively short order.

Because so much of the rest of our apps require Twisted services, we’re going to run our Django app using Twisted’s WSGI unless we run into problems, Then we’ll fall back to eiter CherryPy’s WSGI, Apache’s mod_python, Apache’s mod_wsgi. Whatever, not a big deal.

Documentation Language: Restructured Text

The documentation format of Python that can be easily converted to everything else.

It’s human-readable in source form, intuitive, and is everywhere in all the tools I use.

No brainer.

Other Tools

The stack really isn’t worth anything unless you can deploy it.

For that, I’m relying on several other Python based tools:

Paste

I’m only using the directory template creation of Paste. Paste is for the most part, overgrown and under-focused but the directory templating works well enough for now.

virtualenv, virtualenvwrapper

These allow me to set up an isolated Python environment in which to run my applications. Keeps all the cruft out of the system and gives me an attainable target to deploy.

zc.buildout

Allows creation of a completely self-contained app. Virtualenv’s great for development, but this wraps it all up in a one-stop-shopping bundle.

fabric

Makes deployment as simple as writing a Python script that does what you want to distribute an application to wherever you want to deploy it.

Sphinx

The documentation tool used on the Python project itself. You can set up a documentation structure in one command, write your docs in reStructuredText, and have it in html, latex, and several other formats in a flash.

github/BitBucket/LaunchPad

Not really part of the deployment stack but from having worked on several open source projects on github with git, I think it’s about the best there is right now. I’m still interested in looking at BitBucket and I’m contributing to a few projects there as well but Github seems to be more mature and has a much more informative and useful interface. LaunchPad is very ambitious, and seems well thought out and pretty all-encompasing. Unfortunately, the only backend it supports is Bazaar. Yuck.

Basecamp

We’ve been using Basecamp for a while now for project management. It’s not perfect but it is the best shared system we’ve found. We’ve tried Google Docs and got addicted to shared documents but the rest of the system doesn’t provide any project management functionality so things tended to get lost in there since there was no way to indicate what was to be done next. Basecamp also has shared documents (Writeboards) and also ToDo Lists and Milestones which make it possible to keep a project moving.

FogBugz

We’re currently using FogBugz to track our bugs in the OS X product that we’re untangling the Python code from and it really is a great bug tracking system.

We’ve been focused mostly in BaseCamp so it will be interesting to see how well they integrate or whether we move to another system for this functionality. An obvious choice would be Trac and, with the buildout script, maybe it won’t be so abominable to install.

For now, that’s it.

I’ll be updating this as I update the toolset but this is it, for now…

Cloning in Parallels Gives Bad ETH0

So, I finally got to the point where I was pretty happy with my Ubuntu Server 8.1 image up to the point where I was going to start firing up Python virtenvs and add stuff to them.

The idea is to have a canonical ref, set up to just before the point of doing the actual Python stuff.

First step, obviously, is to clone off the reference VM to a new one and start mucking about with that.

So I cloned, and checked ifconfig and there was no ‘eth0′ interface. Which meant I couldn’t actually do anything with the clone.

Restarted, checked Parallels configuration etc. ad naseum, searched Google a hundred times, etc. etc.

Finally, I found the magic search term:

parallels cloned vm no eth0

First article mentioned exactly the problem I was experiencing. So I went and looked at my /etc/udev/rules.d/70-persistent-net.rules and sure as shit, the entry for eth0 on both VMs were identical. IOW, they were using the same MAC address and everything.

It’s a Debian thing. Here are the relevant articles:

http://forum.parallels.com/showthread.php?t=22204
http://communities.vmware.com/thread/83063

I edited the persistent-net.rules file a couple of times following the instructions but couldn’t get it to work. Since it’s a generated file anyway, I just deleted it, restarted the VM, and it picked up a new MAC address etc. and all was well.

Since I can’t do this easily in a startup file because it only has to be done the first time a VM is booted, it’ll just have to go into the checklist of things to do when cloning off a new VM until I either file a detailed bug report with Parallels and they fix it (never) or I script around it (equally unlikely).