Mail Server Down, All Apple Mail Rules Unselected, Fix with TextMate

So…

There I was, working on our mail server and it was off for a couple of minutes.

Restarted, went back to Apple Mail, and all my messages were piling up in my Inbox.

But I have rules for all these messages!

Open the mail rules and they’re all unchecked!?

WTF

Seems that if Mail tries to run a move rule, and the target mailbox isn’t available, it helpfully (?!?) unchecks it.

Thing is, there’s no easy way to deal with a group of rules; the rules interface in Apple Mail sucks, pure and simple.

So, I went looking for the .plist and found it at:

	mate ~/Library/Mail/MessageRules.plist

Finding a regular expression that would allow me to replace the following was a little trickier:

<dict>
	<key>rules</key>
	<array>
		<dict>
			<key>Active</key>
			<string>1</string>
and so on...

So, basically, each rule has a dict, with various pieces of stuff in them, one of which is the flag that tells whether the rule is active. Unfortunately, it’s not all stored in one tag, so the search has to include the <key>Active</key> plus the newline, plus the following <string> tag.

A quick regular expression handles that just fine:

(<key>Active</key>\n.*<string>)0(</string>)

Two captures (inside the parenthesis) allow you to use the matched text in the replacement. The problem comes in when you try to use the obvious replacement expression:

$11$0

Unfortunately, TextMate (and all normal regexp engines) try to find the match $11, when what you mean is “match $1 followed by a 1 character.”

Rather than bore you with the details, the solution is this:

$1(?0:)1$2

That fixes it right up by using a dummy placeholder (?0:) that always evaluates to nothing, but keeps things straight that we mean $1.

Just for reference, I found this at: A cached google page from ‘Old Nabble’ Since it might be long gone, I’ve duplicated the text here:


Search & replace regex question
by Christian Bogen-2 Sep 02, 2008; 11:35am :: Rate this Message:
Reply | Print | View Threaded | Show Only this Message
Hi everybody!

Is there (in TextMate’s built-in search) a solution for forming a replace string for a variable directly followed by a number, ie. $15 (read: $1 and 5, without a space in between)?

Can this somehow be done or is reformulating the search string the only way in such a case?

Thanks!

Chris

_______________________________________________
Re: Search & replace regex question
by Hans-Jörg Bibiko Sep 02, 2008; 11:53am :: Rate this Message:
Reply | Print | View Threaded | Show Only this Message

On 02.09.2008, at 17:35, Christian Bogen wrote:

> Hi everybody!
>
> Is there (in TextMate’s built-in search) a solution for forming a
> replace string for a variable directly followed by a number, ie. $15
> (read: $1 and 5, without a space in between)?
>
> Can this somehow be done or is reformulating the search string the
> only way in such a case?

Maybe you could try to use Conditional Insertions mentioned in the TM
manual chapter 20.4.3

http://manual.macromates.com/en/regular_expressions

Naïve example:

I have the string

123456

and I want to get

123457

by using this regexp:

123(.)(.)7

this won’t work:

123$1$27 because $27 isn’t set

but this should work

123$1$2(?0:)7

(?0:) is a kind of dummy and inserts nothing

Cheers,

–Hans

_______________________________________________

Re: Search & replace regex question
by Christian Bogen-2 Sep 02, 2008; 12:01pm :: Rate this Message:
Reply | Print | View Threaded | Show Only this Message
2008/9/2 Hans-Jörg Bibiko
Maybe you could try to use Conditional Insertions mentioned in the TM
manual chapter 20.4.3
[...]
(?0:) is a kind of dummy and inserts nothing

Good idea, thanks! I didn’t know that …

Chris

How to Clone a VirtualBox Virtual Machine on OS X

So…

I’m working on my automatic deployment scripts and decided to do some work using local VirtualBox VM’s rather than doing everything in the cloud. The cloud part all works just fine, so there’s no point wasting bandwidth (and money) proving it out a hundred times.

Thing is, it took almost an hour to get a stupid VM set up with stupid Ubuntu 10.04 and updated to latest stupid versions of stupid everything.

So I wanted to just clone my perfectly prepared VirtualBox machine.

Conflicting info on the web led me to the following very simple solution:

Go to ~/Library/VirtualBox/HardDisks. This is on OS X, so if that’s not where they are for you, go where they are.

Run:

# VBoxManage clonehd old.vdi new.vdi

This updates the uuid and such as it makes the copy, so it avoids a whole bunch of problems half of the posts I found were trying to work around.

Create a new VM from the VirtualBox menu, set the drive to the new clone, and fire it up.

So far, it boots, but networking is all broken and I remembered having EXACTLY this same problem under Parallels.

Same, problem, same solution, done: Fixing Cached Stupid Network Info.

eyeball scripts

June 26, 2010 — eyeball scripts
On the #linode channel at irc.oftc.net; you heard it here first…


[09:01am] Tiven: i mean, it happened few days ago with like just 2 users browsing gallery3
[09:01am] gmcharlt left the chat room. (Quit: This computer has gone to sleep)
[09:01am] ssteinerX: does php-cgi limit process ram via php.ini as it does in apache?  should...
[09:01am] Tiven: i have no idea
[09:01am] ssteinerX: sounds like something's getting confused and not being squelched by the ram collar
[09:02am] ssteinerX: was it one process, or a whole bunch
[09:02am] Tiven: its always all of the processes
[09:02am] Tiven: like, 4 children so 4 processes or something
[09:02am] Tiven: each one of them taking a big chunk of memory leading to swap in no time
[09:02am] ssteinerX: that sounds like a flat-out bug to me, but PHP's memory management has never been "stellar"
[09:03am] ssteinerX: are you watching it and restarting php-cgi, or just praying 'till it goes back down, or what?
[09:03am] Tiven: lol
[09:03am] mikegrb: lolz
[09:03am] Tiven: yeah i keep watching it now and then and restarting it
[09:04am] ssteinerX: script handy?
[09:04am] Tiven: otherwise if it goes few days and its going "good", i will later notice it when small stuff will be really slow on the server

[09:04am] ssteinerX: or is this an eyeball script ;-)

[09:04am] Tiven: like, even ssh responsiveness, even mumble lags, etc
[09:04am] Tiven: nah eyeball ^^
[09:04am] ssteinerX: i'll share if I run into the same thing -- too much stuff running for eyeball scripts
[09:04am] Tiven: hehe thanks
[09:05am] cmayo joined the chat room.
[09:05am] ssteinerX: too much eyeball scripts give me a headache, but I haven't tried setting the process limit higher...
[09:06am] snubby: heh @ ssteinerX
[09:06am] snubby: eyeball scripts eh
[09:06am] ssteinerX: snubby: proud to say I just made that up
[09:06am] ssteinerX: i should tweet it or something so i get credit

SSH works, SFTP doesn’t on Ubuntu 10.04

So…

I’m setting up one of my new VPSes (Linode Rocks!) and I was having a weird thing where I could ssh in with root, but not sftp. Other users could do both just fine.

The connection would be made, I tried the usual ssh -vvv type things to get the debugging info, but it looked like the connection was just fine.

Finally, after digging around for a couple of hours and trying every configuration change I (and all the nice people on #linode) could think of, I found this:

OpenSSH FAQ (Frequently asked questions)

2.9 – sftp/scp fails at connection, but ssh is OK.

sftp and/or scp may fail at connection time if you have shell initialization (.profile, .bashrc, .cshrc, etc) which produces output for non-interactive sessions. This output confuses the sftp/scp client. You can verify if your shell is doing this by executing:

ssh yourhost /usr/bin/true

If the above command produces any output, then you need to modify your shell initialization.

I had just added some things to my .bashrc, like, for example:

	bind '"\e[A":history-search-backward'
	bind '"\e[B":history-search-forward'

A quick call to:

	# ssh root@myhost /usr/bin/true

confirmed that those two commands were failing in an SFTP session like so:

	# ssh root@myhost /usr/bin/true
	/root/.bashrc: line 18: bind: warning: line editing not enabled
	/root/.bashrc: line 19: bind: warning: line editing not enabled
	bash: /usr/bin/true: No such file or directory

I couldn’t remember which was which (.bash vs. .bash_profile) so a quick Google search turned up this post.

I moved those two statements to .bash_profile, and sftp now works just fine…

Well, that was fun…

S3FS on CentOS

So, we’re using CentOS 5 for some of our servers; the ones we need cPanel for.  These are our shared setups with people running blogs, Joomla, Drupal, and such.

I’ve never liked FTP for anything due to its insecurity, slowness, and its inability to recover from even the simplest  of errors.

So I finally got our server provider to build a kernel with FUSE support built in so that I could use s3fs to mount an Amazon S3 bucket as a normal mount.

It was a little annoying to set up the first time but, when I had do it a second time, and had to go all the way back to the beginning, I figured I’d better write it down this time.

Install Subversion

First step is to get s3fs from its site at: http://code.google.com/p/s3fs/wiki/FuseOverAmazon.

I prefer to check out from Subversion but Subversion was not installed on my server.

A simple:

	# yum install subversion

 

gave me an error about a missing dependency:

Error: Missing Dependency: perl(URI) >= 1.17 is needed by package subversion-1.4.2-4.el5_3.1.x86_64 (base)

 

To make a long story short, I ended up downloading and installing the RPM directly with:

# wget http://mirror.centos.org/centos/5/os/i386/CentOS/perl-URI-1.35-3.noarch.rpm
# rpm install perl-URI*

 

Download and Install s3fs

Once I had subversion installed, I checked out and built s3fs:

# svn checkout http://s3fs.googlecode.com/svn/trunk/ s3fs-read-only
# cd s3fs-read-only/s3fs
# make install

 

There are a handful of warnings from the compiler, but I ignored them since I wasn’t particularly interested in working on the code.

Setting up Keys

You can invoke s3fs with your Amazon credentials on the command line, in the environment, or in a configuration file. Since command lines and environments are too easy for bad guys to find, I opted for the configuration file approach.

Create a file /etc/passwd-s3fs with a line containing a accessKeyId:secretAccessKey pair.

You can have more than one set of credentials (i.e., credentials for more than one amazon s3 account) in /etc/passwd-s3fs in which case you’ll have to specify -o accessKeyId=aaa on the command line.

Once that’s all set up, you can mount the S3 bucket mybucket at the mountpoint /mnt/mybucket, the command line is:

	# /usr/bin/s3fs mybucket /mnt/mybucket

Now, you can treat /mnt/mybucket as a regular copy destination including using it for rsync!

If you ever want to get rid of the mount, the normal unix umount command does the trick:

	# umount /mnt/mybucket

Enjoy!

mercurial-server on CentOS or RedHat 5.x

So, as per my previous blog post, I got mercurial-server set up on one of our Ubuntu servers just fine.

Then I tried on one of our cPanel based CentOS servers.

What a pain in the ass!

There has been an effort to get this working, somewhat, by adding a new target to the Makefile: `setup-useradd`.

This uses the RedHat/CentOS `useradd` command in place of Ubuntu’s `adduser`.

Unfortunately, because of the way the Makefile is set up, if you don’t have all the prerequisites for building the docs, the product only gets half-setup.

Rather than get all the prerequisites, I found it simpler to just whack the Makefile around to remove the documentation build from the Makefile.

Find the line that says:

`installfiles: installetc installdoc pythoninstall`

and replace it with:

`installfiles: installetc pythoninstall`

You’ll also have to remove the –system parameter to the `useradd` call on line 62 of the Makefile since it’s not supported and is not necessary.

Do those things, then use:

	# make sudo setup-useradd

and everything else will go just fine. If you need to build the docs, you’re on your own, I have no interest in getting CentOS all set up for this unnecessary step.

Serving Mercurial over SSH

So…

I’ve been deploying our big Django application to all of our servers using a pretty slick setup with Fabric and rsync.

This worked fine when I was the only developer, working on my local machine and pushing to a Mercurial repository on one of our internal Ubuntu boxes. Since it was all local, I just used the server’s Apache setup and mod_wsgi and didn’t worry about security too much. The Linux box is completely firewalled off from the Intertubes.

However, as we get more people working on the code, and as we deploy to more servers, having the ability to update to and from our Mercurial repository is becoming more important. Just ask Jeff(rey), whose template changes I clobbered this afternoon.

Since the shared hosting on which I wanted to set up my Mercurial repositories doesn’t have mod_wsgi, nor is it really safe to put a ‘foreign’ module like that into a cPanel setup, I had to find another way to serve mercurial securely.

Also, since we’re finding that Apache and its threading model are consuming waaaaaaaayyyy too much memory under load, we’re moving toward lighter weight, single purpose servers for everything anyway.

So…I found mercurial-server.

It gives secure, tightly controlled Mercurial access over a simple SSH connection.

After I found the `.deb` file and was able to use `dpkg` to actually install it, things went pretty smoothly.

Setup instructions are pretty straightforward with the only part that confused me a little was extracting the key from SSH Agent with `ssh-add -L`. I wasn’t using SSH Agent so the directions didn’t work but once I figured out that al I needed in-hand was the public key, I was on my way.

The repositories are kept squirreled away in kind of an odd location that’s not mentioned anywhere, but that doesn’t seem to be much of an issue as long as that directory tree (`/var/lib/mercurial-server/repos/`) is getting backed up, I’m fine with wherever it wants to put it. The reason for that location, far as I can figure, is that the /var/ tree is supposed to be for things:

/var/ Variable files—files whose content is expected to continually change during normal operation of the system—such as logs, spool files, and temporary e-mail files. Sometimes a separate partition.

At any rate, it took a couple of hours to get set up right and, while speeds don’t seem to be quite as good as they were running under Apigche, it’ll work just fine for my automated setup.

Now, to figure out how to get hooks to do all the necessary pushing and pulling for me…

To see how to get this running on CentOS, see my next blog post. What a PITA!

New project: Denormalizer

So…

I was working on South integration for a project the other day and realized that I wanted to denormalize some things.

Like, keep the number of pages in the site update in the update record, the number of errors, title in the site update page record rather than sucking it from the saved HTML, that kind of thing.

And, I realized that, when things go wrong, there should be an automated way of fixing/updating the denormalized data.

Hence, the Denormalizer.

The basic idea is to specify the denormalized pieces of data in your database in a format resembling a South migration. You provide rules that say “run this query on that database, and stick the results over here.” Every denormalized piece of data has a rule for creating it.

Then, whenever things get hosed, or you’re just feeling insecure about the state of the data, put it into “maintenance mode” and have the Denormalizer go and count things up, extract other things (like the title, above) and fix up the denormalized fields.

Any time you want to make a new denormalized data chunk, just specify the rules for it, and off you go…it’ll be just like it was always there being dutifully updated along with the “normal” fields.

Pretty cool idea; wonder if I’ll ever get to implement it…

WineBottler, WineHQ, Holy Crap!

For Our Few Windows Apps

I’ve been running my last remaining Windows apps, or any new ones I have to run for whatever reason, either on my one real Windows machine, or under Parallels or, more recently, VirtualBox.

We keep the battered old EMachines ET6212 with a whopping 2GB of memory and Windows XP around mostly to run QuickBooks since the Mac version of QuickBooks sucks, can’t convert our old data, etc. I do have one SEO program that I’m still using on that Windows machine, mostly because it keeps the Windows annoyance off my main Mac and it mostly runs scheduled SEO check jobs anyway.

Any time we do have to run a Windows program, I go back to my trusty old Windows 2000 disks and fire up a new Virtual Machine under Parallels or VirtualBox. Fortunately, there have been few programs that won’t run on that now 10 year old (!) OS.

Wine, WineBottler

Today, I was looking for an SQL query building GUI and came across a post that said that SQLyog was still the best MySQL application out there and gave instructions for running it under Wine.

The link from that page was broken but eventually lead me to WineBottler which is apparently the latest greatest effort to make a seamless bundle of Windows apps to run on OS X without a full emulator like Parallels or VirtualBox.

I downloaded WineBottler, opened the DMG, and dragged Wine.app and WineBottler.app to my Applications folder. Go Mac installation.

SQLyog Community Edition

Downloaded the SQLyog Community Edition (what the hell is up with that name?) from this page, then Opened With Wine. This started the automatic bundling process and eventually, after opening and closing a bunch of windows that I didn’t keep much track of, I told it to save SQLyog821CommunityEdition.app to my Desktop, which it did. In the course of all this, it also opened the Windows program!

So there it was, a Mac Application bundle that contained a Windows program, and a Windows program I’d never seen before running on my desktop with no big Windows VM hogging up CPU and memory.

After clearing the nag screens that are part of the SQLyog Community Edition Experience, I poked around, and damn if it didn’t actually seem to work.

Now, to see if it’s actually a better tool than the two Mac ones I’ve been exploring: Sequel Pro, and Querious.

My First Tomcat Experience

I’ve decided, for various reasons, to use Jasper Reports to do the reporting for our app.

I hired someone to help with the initial Tomcat setup; boy am I glad I did…

The project that I contracted was to create a script to set up Tomcat, with Jasper Server, to access MySQL, with example Python scripts to trigger and pull down pre-cooked reports.

So far we’ve got Tomcat running on one port, Apache 2 running on another, the Jasper web UI pages being served by Tomcat, phpMyAdmin set up and working and MySQL accessible from our local network by IP and locally on the server.

At least until this morning.

As per a previous post, I had enabled remote access to MySQL by modifying the `my.cnf` file and, at that time, I had questioned whether having two `bind-address` statements in the config would work.

This morning, I couldn’t connect to the Jasper server either locally on the server itself or remotely from within my networkand, in the sea of Java exception messages, I pulled out `AccessControlException` and went hunting.

I found this post:

The Problem:

When our webapp trys to connect to the database on localhost, an exception

‘java.security.AccessControlException: access denied
(java.net.SocketPermission localhost resolve)’

Well, that kind of looks like what I’m getting.

So, it would seem that I can’t access the local MySQL server via localhost’s IP address any more which would make sense if it’s a “whoever gets there last” scheme for IP addresses in the configuration.

But phpMyAdmin Can Do It!

Thing is…phpMyAdmin is still working, so it is able to get at the local MySQL so perhaps it’s just a difference in how they specify local access; maybe Tomcat’s not using the local port?

I figured I’d work from what was working (phpMyAdmin) backwards so I started looking for the configuration files.

Holy crap.

Check this out:

$ sudo find / -name phpmyadmin
/usr/share/dbconfig-common/data/phpmyadmin
/usr/share/doc-base/phpmyadmin
/usr/share/doc/phpmyadmin
/usr/share/phpmyadmin
/etc/phpmyadmin
/var/lib/doc-base/omf/phpmyadmin
/var/lib/doc-base/documents/phpmyadmin
/var/lib/phpmyadmin

WTF?! Nine (9) different places where pieces of phpMyAdmin had ended up? I understand that documentation might be separate, but is this really necessary?

Note that this is how it is set up by using the Ubuntu `aptitude` utility which is very damn handy but this is just silly.

Turns out that the actual PHP files for the server are in `/usr/share/phpmyadmin` and the actual database access configuration seems to be in ` /etc/dbconfig-common/phpmyadmin.conf`, which actually just fills in the `pma_table_info` table with default access information.

In `phpmyadmin.conf`, where it fills in the `pma_table_info` record, the only thing actually specified is `mysql` as the database type. The host and port are left blank as shown below:

# dbc_dbserver: database host.
#       leave unset to use localhost (or a more efficient local method
#       if it exists).
dbc_dbserver=''

# dbc_dbport: remote database port
#       leave unset to use the default.  only applicable if you are
#       using a remote database.
dbc_dbport=''

So, I went and did the same thing to Jasper’s configuration file at ` /var/lib/tomcat6/conf/Catalina/localhost/jasperserver.xml`

I changed the line:

       url="jdbc:mysql://localhost:3306/jasperserver?useUnicode=true&characterEncoding=UTF-8"

to read:

        url="jdbc:mysql:///jasperserver?useUnicode=true&characterEncoding=UTF-8"

Restarted the server with:

	#  sudo /etc/init.d/tomcat6 restart

No joy.

Bind to One or All

Well, I still don’t have a satisfactory reason for why phpMyAdmin can connect to the server whether localhost is specified or not but I suspect it’s something Java-ish that I’ll just have to ask my guy about after New Years.

For now…

You can either have MySQL bind to one address, or all. After a bit more poking around, I finally found the doc and now it says that ‘whoever gets there last wins’ where previous versions didn’t address the issue.

So…if I want to restrict access, I can do it through a combination of firewall settings and permissions in the database.