Jonathan Hitchcock - Vhata Vas Hyah

Recently, a friend of mine told me that he and a friend wanted "something nice" to do on a sunny Saturday afternoon in Cape Town. My response was (transcribed):

Beer in the sun? Or, wait, there's an awesome italian ice-cream shop in Hout Bay, on your right, just before you round the corner that takes you to Chapman's Peak. Drive the long way round though - along the coast as much as possible. I'm not sure that Chapman's Peak drive is open now, but if it is, that would be a win, and it would take you down past Cape Point, and then you can stop in Simonstown and visit Just Nuisance. Then you could go to the jetty in Kalk Bay, then take the Muizenberg Coastal Road, along, ending up in Stellenbosch for supper. You should've started this two hours ago. I think Chapman's Peak drive is closed, actually, but you could still take Ou Kaapse Weg, which cuts across the lower cape, and bypasses going all the way round on Chapman's Peak. You could also go up northish, to Blaauwberg, and walk along the beach there. Go to Blue Peter and have a cocktail while sitting on the grass out front, although that'll probably be a bit crowded. Or go straight out to Stellenbosch, pick up some olives and cheese from the Spar, and go sit on the university lawns and eat them in the sun. Or go up to Rhodes Memorial and have hot chocolate at the cafe. You live in the most beautiful city in Africa, and you come online to ask what to do on a day like this?!

My point was that there are endless things to do in this amazing city. I've said it before, and I'll say it again.

I recently discovered Do Stuff in Cape Town - an awesome site that lets people add activities, tag them, rate them, add photographs and descriptions, and allows you to search for things by a number of criteria. My friend Megan tried to find a similar site for Johannesburg, and did in fact find a site entitled "Things to do in Johannesburg", which basically boiled down to:

• Shopping
• Sun City
• Leaving Johannesburg for a holiday

Now, I don't want to say anything bad about Johannesburg, but... Well, you know... As Simon and Adrianna say, "Dry, flat and perpetually on fire.". But I digress. I come not to bury Johannesburg, but to praise Cape Town.

A little over a year ago, I wrote about wandering around the cape with Megan. Well, she recently returned, and once again, Cape Town didn't let us down.

Starting on Friday night, we had an awesome Mexican dinner at Panchos in Obs, and then went to Banana Jam for cocktails and drinks with some friends. On Saturday morning, after a bit of shopping at Canal Walk, we drove through to Stellenbosch for the Wine Festival. It was a little packed with students, but we had a lovely afternoon of tasting wines and a few other delicacies (oysters, mmm), then picked up some nice wine, and headed back to Cape Town, where we were having some friends over for dinner.

I have to digress here. Oh, Woolworths, how I love thee. I give you money, and you really impress my friends. For dinner, we made:

• Starter: Philadelphia cheese with almonds and strawberries, drizzled with honey, on crackers.
• Starter: Avo wrapped in salmon on a bed of rocket, with a lemon wedge.
• Mains: Fettucine with pesto, sun-dried tomatoes and feta, with a greek salad and garlic bread.
• Dessert: Pecan-nut pie with ice-cream

And while I am a little proud of that, I'll come right out and say that I couldn't have done it without the love and support of Woolworths, who always believed in me.

On Sunday, we went to Pastis - a lovely french restaurant in Groot Constantia - for breakfast with Paul and Kerry-Anne Gilowey, and Deon and tink, and thence to Cavendish for a bit more shopping (I sat in Mugg and Bean). When we were done there, it was still an awesome afternoon, so we drove down to the Radisson Hotel, where we had cocktails in the sun, and walked along the pier as the sun set. Finally, to top off the weekend, we went to Beluga for sushi and more cocktails. (Unfortunately, due to a misunderstanding I had with an airport luggage attendant (he must have thought that the camera in my bag belonged to him?), there was no photographic evidence like last time.)

Now, you'd think that I'd just about said all I could about Cape Town. But you're wrong. A week ago was Heidi's birthday, and a bunch of us went to Grabouw for the weekend. To say that it was beautiful will not suffice. We stayed in charming cottages nestled in a clearing in a pine forest, next to a lake in the mountains just outside Grabouw, and it was lovely. Jacques took this incredible panorama:

I don't think I can say anything to top that, so I'll tip my hat, once again, to the Cape, and stop.

Senator Barack Obama has won the race to become the nominee of the Democratic Party in the United States of America. Today, Hilary Clinton threw her weight behind him:

"Today as I suspend my campaign, I congratulate him on the victory he has won and the extraordinary campaign he has won," Mrs Clinton said. "I endorse him and throw my full support behind him and I ask of you to join me in working as hard for Barack Obama as you have for me."

I can't really tell you why I'm so hopeful for Obama. Maybe it's because people I really respect have come out and endorsed him - several times - breaking their usual non-political stances in several instances. But I've read about him, and I've watched his speeches (transcript here), and I'm optimistic about where America is going.

If you haven't seen the video before, watch it now: yes, we can.

America, so unwell, but still able to thrill us when you get it right...

If you have any spare clothes (and I know you do), or any spare cash (and I know you do), please, PLEASE, help out. There are thousands of people left with no homes, and nothing possessions but what they're wearing. Most supermarkets should have donation boxes, but for more ideas of how to help, check out the Treatment Action Campaign site, and there's even a facebook group for you.

Thanks.

A champagne breakfast out of town in Stanford on the Garden RouteOverberg. Nothing more relaxing.

Pictured: Danni Davies, and our bottle of Poingracz. Not pictured: Matth Gair, Emily Davies, Shannon Morreira, James Reeler, and yours truly.

You can't really make him out, but that is totally my man MC Hawking up there on stage. He's presenting a talk entitled Universe. It's awesome.

As I said on the GeekDinner announcement list:

Since the last GeekDinner was held at the end of March, and since we hold the GeekDinners bimestrially, it seems we are due another one at the end of May. This is the eightthhth GeekDinner, and we're calling it "Happy Habanero". What with the habanero being the national vegetable of Azerbaijan, we're going to hold the dinner on Azerbaijan's Republic Day, which, according to wikipedia, is Wednesday, May 28th.

The venue for this dinner is Mel's Kitchen, in Rondebosch Village, just off Klipfontein Road.

As usual, you can sign up, and check on the other details, on the wiki page.

We're now in our second year of GeekDinners, and they seem to be going strong. We have a good model, mostly sustainable, although it is slightly dependent on the core group of organisers to get things moving. We have a solid set of regular attendees that should provide the dinners with enough momentum to continue, though, should anything happen, and I'm very positive about the future of the dinners. We're also managing, for the most part, to keep talks short and interesting to all comers - I know that newcomers always worry that everything's going to be "too hardcore techie", but honestly, it's not about microchips and "ones and zeroes". My favourite talks have been about hippies and the buttons on car radios. So, please, if you haven't been before, why not come along, meet some new people, share some free wine, and enjoy some excellent food.

The slideshow karaoke has become a regular feature of our dinners, and is always one of the most entertaining parts. The way it works is, somebody prepares a set of slides on any topic they want (we've had "Etiquette when dealing with British Royalty", "Common problems with cement tiles", and "A primer on lesser known Norse gods"). Somebody else then presents a talk based on these slides without any prior knowledge of the topic, or of the content of the slides - always to amusing effect. This time, Darb is preparing the slides, and we have yet to find a volunteer to present them. If you're keen, do volunteer. If not, maybe you have something interesting you'd like to talk about anyway - we have no volunteers for speakers yet.

If I've sold you, sign up on the wiki, and we'll see you there!

An apology

The sharp-eyed among you will have noticed that my blog has suddenly become one of those sites. I apologise for flooding your feed readers with pictures of seals, but let me explain.

A justification

The whole "let me post random photos from my life on my blog" thing was more an exercise in "how easy would it be to make a phoblog?" than a desire to share what my shoes look like. I will admit that when I took the sunset photo, I thought "this would be a really good thing to share with the world", because, let's admit it, Cape Town is one ridiculously beautiful city, and people need to hear that. But that got me thinking how easy it would be to make a photo shareable, and here is what I came up with.

An explanation

There is actually a function on my phone labelled "blog this", but I think it sends the image (or whatever) to a Sony-sponsored blogging site, and I'm frankly not interested in that. I wanted to solve the problem academically, for the general case, and as a side effect solve it for my specific case - I run this blog in a drupal instance on my own server hosted with Layered Tech.

A discussion

So, the various ways to get information from my phone to my server were MMS, email, some form of push to a web-page, or bluetooth/cable upload to a laptop/desktop which will send it on. The last option defeated the point - I wanted to be able to blog a photo from anywhere, using nothing but my phone. Using the web-page push is what the "blog this" function does, but for my specific case, I'd have to write a custom application for the phone, which was way more effort than I wanted to expend. Sending an MMS would require me to have a GSM modem listening somewhere to receive it, and had the added disadvantage of requiring that the images got resized down. So, it seems, the best way to get the information from my phone to my server was to simply send an email (with images attached).

A technical discussion

The rest of this post describes the technical details of what happens to the email when it arrives at my server.

As an overview: I catch mail meant for the phoblog using a procmail recipe, and pipe the mail to a python script, which parses the message and pulls out the relevant parts, constructing the body text, creating thumbnails of the images and saving them in the right place. Having deconstructed the message and constructed the blog post, it passes the bits (title, body, and publication date, which it extracts from the EXIF information in the photos) to a PHP script, which hooks into the Drupal API and actually creates the blog post.

The PHP script is necessary, since there's no other way to hook into the Drupal API. I could do something like faking a bunch of HTTP GETs and POSTs, and passing the information in as if I was actually blogging it from the web interface, but that's even more klunky than simply piping it into a PHP script. The question then arises why I couldn't write the whole thing in PHP, and save myself the expense of running two scripts requiring two different interpreters, but frankly, trying to get PHP to do what is necessary would end in such an inelegant, ugly, hackish result that it just wouldn't be worth it.

An added advantage to separating the Python parser and the PHP script is that you can replace the PHP script with one that injects an entry into a different blogging platform, and it'll still work fine. So, somebody could write a script that talks to Wordpress, and simply drop it into place.

---

The injector (the PHP script)

The PHP script needs to hook into the Drupal API, so we first need to bootstrap into the Drupal environment. First we fake some HTTP headers in the $_SERVER array so that Drupal knows which site is being "requested" (Drupal does some clever multi-site stuff based on which URL is being requested). Then we change to the Drupal base directory (defined as a constant at the top), include the bootstrap code (also defined at the top), and then simply run the drupal_bootstrap() function:

<?php 
// Defined as a constant, could/should be passed as an option or loaded from a config file:
define('PHOBLOG_DRUPAL_URI', 'http://vhata.net/');
// Fairly standard for Drupal installations, but as above:
define('PHOBLOG_DRUPAL_ROOT', '/usr/share/drupal');
define('PHOBLOG_DRUPAL_BOOTSTRAP', 'includes/bootstrap.inc');

// Fake the necessary HTTP headers that Drupal needs:
$drupal_base_url = parse_url(PHOBLOG_DRUPAL_URI);
$_SERVER['HTTP_HOST'] = $drupal_base_url['host'];
$_SERVER['PHP_SELF'] = $drupal_base_url['path'].'/index.php';
$_SERVER['REQUEST_URI'] = $_SERVER['SCRIPT_NAME'] = $_SERVER['PHP_SELF'];
$_SERVER['REMOTE_ADDR'] = NULL;
$_SERVER['REQUEST_METHOD'] = NULL;

// Change to Drupal root dir.
chdir(PHOBLOG_DRUPAL_ROOT);

require_once(PHOBLOG_DRUPAL_BOOTSTRAP);
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);

?>

Now we are running in a Drupal environment. The next step is to collect the information that we want to insert as a blog entry. We take the title and publish date from the arguments passed to the script, and then do a loop to read the body from standard input:

<?php 
$date = $_SERVER['argv'][1];
$subject = $_SERVER['argv'][2];

$fp = fopen('php://stdin', 'r');
$body = "";
while($line = fgets($fp, 4096)) {
        $body .= $line;
}

?>

I may be wrong, but I don't think there are any sanitization problems in the above code. Let me know if you can see any? I'm pretty sure I don't need to escape anything, since I pass all variables as-is to Drupal, which does full sanitization before using them. Anyway, the final step is to simply call the drupal node_save() function to save the blog post as a node (passing it some default values):

<?php 
node_save((object)(array('created' => $date,
        'title' => $subject,
        'body' => $body,
        'teaser' => $body,
        'format' => '3',
        'uid' => 1,
        'type' => 'blog',
        'status' => 1,
        'comment' => 2,
        'promote' => 0,
        'sticky' => 0)));

?>

My only worry there is that I specify that the format is type 3 (unfiltered HTML) - this might leave the phoblogger open to code-injection exploits. I should probably specify type 1, filtered HTML, to make sure that nobody can accidentally blog something nasty.

So, that's the PHP script that injects the entry into Drupal. The other part of the system is, of course, the Python script that parses the email in the first place.

The parser (the Python script)

I'm not posting the full script here, for a number of reasons, mostly to do with it "not being finished yet". It works, but it doesn't do everything it should (including a complete security check, since that's kinda hard to implement on emails, which can be faked). Suffice it to say, it uses the optparse, ConfigParser and logging modules to be nicely configurable, runnable, and debuggable, and all that. But, yeah, I'm still embarrassed about it, and won't post sourcecode until I think it's good-looking enough for public consumption. What I will post here is bits of python code that demonstrate the actual meat of the thing - how I deconstruct and process the email that I receive.

The basic steps I perform are:

1. Break up the email and extract the bits I need from it.
2. Process each attachment part:
   • Text attachments get HTML-ified
   • HTML attachments get inserted as-is
   • Image attachments get thumbnailed, and the thumbnails and originals get stored somewhere web-accessible, and a chunk of HTML that references them gets created.
3. Send the results of this processing to the injector script with the right subject and date.

Breaking up the email is trivial using the email module in python:

import email
msg = email.message_from_file(sys.stdin)
subject = u''.join(unicode(part, encoding or 'us-ascii') for part, encoding in email.header.decode_header(msg.get('subject')))
msgfrom = email.utils.getaddresses([msg.get('from')])[0][1]
msgid = msg.get('message-id')

for piece in msg.get_payload():
   processpiece(piece)


As you can see, no regular expressions needed to match headers, do MIME decoding, or break up an email address. You can even give it a list of all the different stupid formats for addresses that mail clients seem to use these days, and it will understand them:

>>> getaddresses(["jonathan@vhata.net", '"Jonathan III" <vhata@clug.org.za>', 'pope@vatican.org (Benedict)'])
[('', 'jonathan@vhata.net'),
 ('Jonathan III', 'vhata@clug.org.za'),
 ('Benedict', 'pope@vatican.org')]


I break each attachment up and send them to the processpiece() function one at a time.

Inside the processpiece() function, I can get at the content-type of the chunk I'm processing by using the get_content_type() method:

>>> piece.get_content_type()
'image/jpeg'
>>> piece.get_content_maintype()
'image'
>>> piece.get_content_subtype()
'jpeg'


and I can use this to work out what I want to do with the chunk. I can also get the chunk in its raw form (i.e. decoded from the MIME transport that email uses by simply calling get_payload() on it:

payload = piece.get_payload(decode=True)


If it's text, I simply replace all the newlines with HTML line breaks:

payload.replace("\n","<br />\n")


The difficult case is, of course, when it's an image. Here, I use the Python Imaging Library to process the image. I extract the EXIF timestamp and turn into a datetime structure, so that I can create a hierarchical directory tree to store the images. Then, I construct a thumbnail filename and create the thumbnail:

payload = piece.get_payload(decode=True)
image = Image.open(StringIO.StringIO(payload))

timestamp = datetime.datetime.strptime(image._getexif()[EXIF_DATETIME], "%Y:%m:%d %H:%M:%S")
self.entrystamp = timestamp

targetdir = "%04d/%02d/%02d" % (timestamp.year, timestamp.month, timestamp.day)
try:
   os.makedirs("%s/%s" % (TARGETDIR, targetdir), 0755)
except OSError:
   pass

fname = piece.get_filename()
(rootname, ext) = os.path.splitext(fname)
ext = ext.lower()
fname = "%s%s" % (rootname, ext)
thumbname = "%s-thumb%s" % (rootname, ext)

image.save("%s/%s/%s" % (TARGETDIR, targetdir, fname))
os.chmod("%s/%s/%s" % (TARGETDIR, targetdir, fname), 0644)
image2 = image.copy()
image2.thumbnail([THUMBSIZE,THUMBSIZE])
image2.save("%s/%s/%s" % (TARGETDIR, targetdir, thumbname))
os.chmod("%s/%s/%s" % (TARGETDIR, targetdir, thumbname), 0644)


Then I return a templated chunk of text to dump into the blog post. Easy as pie.

The last step is to pipe the individually formatted pieces to the injector script, passing it the date (extracted from the EXIF information above) and subject as parameters:

injector = subprocess.Popen([ADDCMD, entrystamp.strftime("%s"), "Phoblog: %s" % subject],stdin=subprocess.PIPE)
for piece in body:
   injector.stdin.write(piece)
injector.communicate()


And off it goes.

Some concerns

First and foremost, security is a problem. If I'm sending an email from my phone, anybody can send the same email from their own phone - there is no identification in the email. One way around this would be to require a keyword in the subject before accepting it. This is security by obscurity - anybody who gets hold of the keyword will be in. I can decrease this risk by forcing some sort of hash on the keyword. For example, if the keyword was "pilates", I could require that the number of consonants in the current day be appended to that: "pilates6" on a Sunday, "pilates7" on a Tuesday. This slightly decreases the risk, but not much. There are other, even cleverer variations on this theme, but they are all basically just security by obscurity. A better way would be to use authenticated SMTP, and only accept phoblog messages that were authenticated through my own SMTP server, and I think I might implement this, unless I can think of a flaw in the idea.

Another problem is that I might lay myself open to HTML/javascript/etc injections, but I think this will be allayed if I solve the problem above.

A conclusion

This has been a somewhat rambling, somewhat disjointed explication, but I hope it gives you the general gist of what I did. If I ever look at the script again, maybe I'll fix it up properly, and make it publicly available. I even registered phoblog.za.net but that's taking some time. Meantime, enjoy piccies.

My sister and I went to Camps Bay for the afternoon. It was extremely windy, and we both got thoroughly exfoliated, but at the end of it, we had this magnificent sunset as a reward. I don't think there's a prettier beach in the world than our west coast.

Siamese cats are born snobby. This is Bo, and he's been sick, and looks it, but he still manages to carry himself with an air of utter aloofness.

There's a little platform in the Waterfront harbour with steps up to it from the water, so the seals can climb up and sun themselves. This guy spent ten minutes grooming himself for us.