WARNING: This is rather a technical post. If you’re not comfortable doing neckbeardy things like typing stuff at the Unix command line and adding stuff to your bash profile, you probably want to give it a miss.
All I wanted to be able to do was write posts in Markdown in a Dropbox folder and have them be automatically published as a static blog hosted on Amazon S3. On a Mac. Not too much to ask, right?
Well it turned out to be quite difficult and time consuming, with lots of wrong turns into blind alleys. So if you’re thinking about setting up something similar, here’s what worked for me.
This setup only really has two pieces:
A static blog generator—this takes files that you write (typically in Markdown) and converts them into a set of HTML pages, including a main page, individual post pages, archives, etc.
Some way to sync these HTML files to S3.
And this is what I actually ended up using:
I also used Lingon to automate it all (using launchd).
But first a little bit on how I arrived at this point.
Some Right Turns, Some Wrong Turns
There are many static blog generators out there. The most popular seems to be Jekyll, so I decided to have a go at installing that. It uses Ruby. Macs come with Ruby, so it should be easy. It went OK, but eventually I reached the point where I was required to upgrade Xcode. Xcode is a hefty download, and I didn’t want to wait, so I started looking at alternatives.
Pelican seems to offer similar functionality to Jekyll, but uses Python instead of Ruby. And Python is also pre-installed on OS X, so why not give it a shot? So I installed Pelican and the things that it requires: pip, virtualenv, and virtualenvwrapper. (Ignore the fact that the pip docs say you can get pip by installing virtualenv—the easiest way is to install pip and then use it to install virtualenv.) Pip also requires another package called distribute, so install that too. And if you want to write in Markdown, you’ll need to install the Markdown module, as described in the Pelican docs.
So now that I’ve got all these installed, it should “just work”, right? You wish. To get Pelican to generate the HTML, you need to run the
make command with the makefile that comes with Pelican. But to get
make (if you don’t already have it), you need to install the latest version of Xcode. (Déjà-vu much?)
So, much later that day, I have the latest Xcode, and have installed the optional (but required here) command-line tools. Now it works, right? Well, almost. Now I get an “unknown locale” error. Say what now? A little more Googling reveals that I need to set the
LC_CTYPE environment variable in my
.bashrc file to something like
en_US.UTF-8. (An aside: I tried all this with tcsh, which I like more than bash, but ran into even more problems, so I switched back to bash.)
Anyway, now, finally, it works. On to the next piece of the puzzle—syncing the files to S3.
There were a couple of things that I tried that did not work for me. The first was trying to mount my S3 bucket as a remote drive using MacFusion. I thought I could mount the bucket right inside a folder in my Dropbox. But it didn’t really work.
The second was trying to upload using SFTP via an FTP-to-S3 gateway (like Cloud Gates). The gateway part works great (and I use it regularly for other stuff). And Pelican has an FTP
make target that you can modify to use SFTP. But again, it didn’t really work.
s3cmd, on the other hand, works great. At least, once I figured out that I needed to install the latest version from Github instead of the one on the download page. s3cmd has a sync command that lets you sync a local directory with one in your S3 bucket.
Automating It All
The only thing left to do was to automate it so that as soon as I add or change a file in Pelican’s
content directory, it will automatically regenerate the HTML and sync the output directory with S3.
On OS X, launchd is the thing to use. But manually creating the necessary plist files is a bit of a pain. Thankfully, there are tools that can make it easier. One of these is Lingon. I used it to add two items:
One to start Pelican’s
make regenerate, which, once started, detects any changes in files in the
content directory and updates the HTML
One to run s3cmd sync whenever the
index.html file in the output directory changes
This is what they look like in Lingon:
I couldn’t get these to work by typing the commands directly into Lingon—I had to wrap each one in a shell script. This is what they look like. (Of course, you’ll have to change the directory paths and the S3 bucket name.)
To sync with S3:
/usr/local/bin/s3cmd sync --delete-removed \
Don’t forget to
chmod +x these so they can be executed.
Some Other Gotchas
When you’re writing posts, you must give them a title and a date, like this:
Title: My Awesome Post
If you don’t, Pelican won’t process them. You can also add a status of “draft”:
Pelican will put the generated draft post in a folder called
drafts. The post will not appear on your blog until you remove this line.
This was a total pain in the ass to set up. But the result is so worth it. When I hit save in Textmate, and twenty or so seconds later, the post appears on my blog, it feels like magic. And because the files are in Dropbox, I can do the same in iA Writer on my iPhone or iPad (or move from one device to the other if the kids are playing Subway Surf on the iPad).
You can see it in action on the Livetyping blog.
UPDATE: After finding that my whole blog had completely disappeared (twice), I stopped using