The grand Summer of Code Mentor Summit of 2009 concluded last week and I had the fantastic opportunity of being able to attend on behalf of Gentoo, Plan 9 and Mozilla. What follows is some indication of how awesome the summit was:

(Photo courtesy of warthog from Etherboot)

I met so many folks I’d only interacted with online so far (the classic nickname-to-face matching), but even better was the opportunity to meet folks powering open source projects from so many diverse backgrounds. I met many of my personal rockstars, and learned about a bunch of open source projects I’d never heard of

Also, one of the things that is only possible at an event like the summit was the ability to get a whole bunch of non-linux operating system groups in one room. We had a great discussion, and it resulted in the creation of the “rosetta-os” special interest group. Look for more activity on the common device drivers for non-linux operating systems front soon!

Other sessions worthy of special mention were Open Source Security, Recruiting and Retaining Awesome People, Advanced Trolling (yes, you read that right), and of course the always welcoming Casablanca where I spent most of my time. We discussed everything from our SoC experiences to the Afro Celt Sound System in that room, always full of creative energy and warmth.

After 4 years of participating in the Summer of Code, I am super happy to have finally met the faces behind the program. Every single person I met over the course of last weekend was friendly, intelligent and just generally awesome; that sort of thing doesn’t happen by chance. I feel warm and fuzzy inside to think that I’m actually a part of the revolution that is free and open source software, three cheers to everyone that made it possible!

My last day at Mozilla this summer was last Thursday. I didn’t take a lot of pictures this summer, because, you know, I took a lot last time around. Also, this strategy turned out pretty well because now there are more pictures of me floating around on the tubes! After a longish trans-atlantic flight, I’m back in Amsterdam now resuming work on my Master’s (because hacking on Minix is awesome).

No other internship has been ever so satisfying: over the summer, I worked on a wide range of mini-projects which allowed me to exercise skills ranging from systems to application level programming. I even did a bit of work in the mobile space (turns out programming in limited memory and processing speed is a *lot* different).

One such project that I’m especially excited about is support for video recording in the browser. Yes, there is even a canvas-based live preview of your webcam feed, in addition to Ogg/Theora encoding support! Combined with the audio recording support I wrote sometime ago, some really cool applications are now possible. Skype-like dialer in the browser? Why not?! (*hint* anyone is free to send in a patch for multiplexing the audio and video, they’re currently two separate Vorbis and Theora streams *hint*).

We also had 3 major releases for Weave during the summer: 0.4, 0.5 and 0.6. The last one was especially big, given the completely new, HTML based UI (big kudos to thunder for pulling it off!) and a bunch of other performance fixes. Also, the web UI I wrote last year underwent so many great changes by the wonderful folks at Glaxstar. Now we’re putting up a community design challenge to revamp the UI so we can ship the thing! (*hint* if you’re good at UI design you should participate in the challenge *hint*).

There’s so many more cool things I worked on that I’d like to talk about, but perhaps they deserve a separate blog post. Soon… (I keep promising myself that I should blog more often, it never works).

To add the already good times, my two students in the Summer of Code this year passed with flying colors. Yay!

It’s that time of the year. Google is, yet again, sponsoring students to write some awesome open source code this summer. If you’re a student, and you’d like to make some money contributing to some of the most well known and exciting open source software projects out there, you’d be missing out on a lot by not applying.

If you’re wondering about what the best way to get started is, check out this great advice page. All projects have also been tagged by programming language and field in this delicious profile. You can also search for ideas here.

I’m going to be mentoring for Mozilla, Glendix (under the Plan 9 Umbrella) and Gentoo this year. Get in touch if you’re interested in any of those ideas

The window for applications opens in a few hours. Good luck everyone!

My Goa trip was simply fantabulous. Apart from the fact that Goa is a great place for a vacation, I was accompanied by 7 of my college friends which made the trip one that I will cherish for a long time to come.We left Bangalore by bus on the 15th. The journey was pleasant and the view next morning was absolutely stunning:

The bus dropped us off at the Panaji bus terminus, and we took a shuttle from there to Vasco – where Ameya (our host) lived. After a nice lunch and a nap, we took off to the beach closest to base camp – Bogmalo. The beach was a quiet and clean with relatively few people around, which made it possible for us to play a game of beach football. We returned home after jumping around in a sea for a while.

We hired a couple of Activa’s the next day (this seems to be the norm for transportation in Goa) and reached Old Goa in an hour or so. We visited the really old church, which was really impressive – it also contained the remains of St. Francis Xavier. The archaeological museum next door was fun too, very informative about the history of Goa. We proceeded to the capital city of Panaji next, and after booking tickets for a river cruise aboard the Princess de Goa for the night, had lunch at the QuarterDeck.

After lunch, we visited Donapaula, a popular Jetty, which was unfortunately under renovation or something. The view was great though, and we enjoyed a nice little ride on the water scooter. After hanging out in Cafe Coffee Day for a while (These places are *everywhere*, I think they’re trying to become the Indian Starbucks :)) we reached the Panaji river coast to board the Princess de Goa. These river cruises seem to be a popular attraction, they basically consist of a few dance shows, a dance floor and an amazing view. We got to see the (in?)famous floating casino on the way too.

My Summer of Code mentor, Matt Lawless, happened to be in Goa too, so we scheduled lunch for the next day. We met at the Calangute post office (which was somewhat close to Matt’s home) and proceeded to the Calangute beach after lunch. My friends, meanwhile, reached the Baga beach, which was just next door to the Calangute beach (the two most famous beaches in Goa). We splashed around in the water for a while, joined by Matt, and then a second lunch

Matt decided to leave, and we went on to try some of the water sports at the beach. We went for a banana ride, a water scooter trip, but the one that took the cake was the parasailing. Nothing like a gentle ride in the sky to rejuvenate you. Flying over water with the beach behind you and the sunset in front is an experience I can’t put in words

The third and last day began with a long ride to south Goa, where we first visited the Benauli beach. This beach was beautiful, the sand was different than the others, and the most fun part was when I was buried by the others:

The rest of the day was spent at GoaKart, which was apparently a national Karting track. Parasailing was great, but karting was really the most exhilarating, especially because we raced and went for 4 rounds

Flickr didn’t let me upload more than 100MB of photos at once, so I moved to Picasa Web Albums instead. I wrote a small script: backr.py that uses James Clarke’s flickr.py to back up all my photos and uploaded them to Picasa Web, which allows me to create as many albums as I want (unlike Flickr). UPDATE: I finally got myself a Flickr Pro account, so pictures are back there again. This time I just used iPhoto to migrate

So, I guess that’s a few more items off my “list of things to do before I die”!

I feel I’m in the seventh heaven. After a few sleepless nights struggling with Mozilla’s XPCOM, I finally got the 9P Firefox plugin to work.The plugin is called Angled (an anagram of Glenda, the Plan 9 bunny) and is in a pretty simplistic state right now: you can read any files served by 9P right in your browser window. Let’s take a step by step look.First I startup Inferno to start a 9P server. ${INFERNO}/usr/anant/home is symlinked to my actual home directory, /Users/anant:

[theghost anant]$ emu
; runas nobody {listen -A tcp!localhost!1564 {export /usr/anant/home &}}

Let’s see what files are actually there:

[theghost web9]$ pwd/Users/anant/Plan9/web9
[theghost web9]$ lsREADME TODO js9p php9p

Alright, I open my browser window and type ‘ninep://localhost!1564/Plan9/web9/README’ into the address bar. I could also say ‘tcp!localhost!1564′, but TCP is the only protocol available for Angled, so it would be redundant. Now, for the goodies: Screenshots!

Cool! But wait, Angled also displays binary files right in the browser. There’s a catch though, it will only work for binary files that can be viewed directly in the browser window. Certain types of files (.doc for example) do trigger a download request, but then become corrupted for some reason.

[theghost content]$ pwd/Users/anant/Plan9/web9/js9p/angled/content
[theghost content]$ ls
angled.png firefoxOverlay.xul glenda-error.png overlay.js

Let’s say I want to view angled.png. Here’s what I get:

Okay, but what if you type in a URL that points to an invalid file? Check this out:

Sweet! I’m yet to figure out how to transmit the exact error message to that page, so you’ll have to make do with that generic image for now.

Okay, now onto the bad parts. Angled doesn’t support authentication yet (although the base JS implementation is capable of generating and parsing auth messages). Next, you won’t get directory listings (you’ll get a bunch of binary gibberish which is actually Rstat messages for the directory’s contents). Also, I’m doing the 9P connection and transactions in a blocking thread, so the UI freezes while all that is done. I couldn’t feel the difference since I was testing on my local 9P server, but connecting to remote 9P servers won’t be a pleasant experience. The solution to this is to create a custom nsIChannel implementation, which is a lot of work… I’ll do it when I get to it

Enjoy!

Creating a Firefox extension is nothing short of an adventure. I was able to get started pretty quickly, thanks to this web-based quick-start wizard, all the boilerplate code was generated in literally no time.

Now, onto the actual functionality of the extension. I have to implement a protocol handler for the 9P protocol, which essentially means you type in “ninep://sources.cs.bell-labs.com/” and start reading files right off the browser window. (ninep:// because a URL can’t start with a number)

This page provides some useful insights and code snippets on the subject of adding a new protocol handler. I was able to get as far as displaying a Glenda image whenever you type in a URL beginning with ‘ninep’.

The way this works is you create an XPCOM component that implements a standard interface. Specifically, the newChannel() method is where all the action is. It receives a URL and you do something and return an nsIChannel. Mozilla provides standard nsIChannel implementations for popular protocols such as http, ftp and even the ubiquitous file://.

The intuitive thing to do here would be to do all my 9P processing in the newChannel() implementation and return a stream in a standard channel. However, that’s not going to work, since newChannel() would then block and the UI would actually freeze until the 9P transaction completes. Sub-optimal.

The “proper” way to do this would be to create my own implementation of nsIChannel. That way I just create a new nsIChannel in newChannel() and be on my way. nsIChannel would then take care of firing callbacks as and when data arrives. There’s somewhere I can start with, and that’s the Mozilla implementation of the finger protocol. It’s written in C++, however, and I need to figure out how I can map the same to JavaScript (via XPConnect).

There’s one last hurdle before I finish my GSoC project. I’ve already written the JavaScript that produces binary messages for every possible 9P transaction, and all that needs to be done is to actually send those messages to the 9P server.

In a few of my previous posts, I mention that there were two apparent ways to do this:

a) Send the message wrapped in an XMLHttpRequest to a HTTP server that forwards the message to the actual 9P server.

b) Use Mozilla’s XPCOM components to access Sockets directly in JavaScript.

Well, it turns out that (a) is probably not a solution at all. HTTP is far from what one would call a protocol that supports streaming. A 9P server (correctly) doesn’t return an EOF until you have completed your whole session. So the first time I send an XMLHttpRequest to, say a PHP script, the script blocks forever, since PHP would never know when the first R-message has been actually sent through. I can always peek at the first 4 bytes and find out the length of the R-message, but what then? I can’t close the socket since that would terminate the 9P session, but I have to return from the script for the HTTP response to be actually sent.

PHP doesn’t support threading (proper) so I can’t do the select() mojo either. How about storing the socket FD in the session variable? Well, this is probably the closest to a good solution but that would limit every client to exactly one 9P session.

Although Mozilla’s XPCOM is one hell of a beast, I think it might be good to just build a firefox extension to access 9P resources. Not exactly sure of how I’m going to make it work, but tentatively, I’m thinking of parsing URIs beginning with 9p:// or something like that. Let’s see how this goes.

My mentor came up with two interesting ways of swapping endian-ness on JavaScript. The first one he proposed was based on what was “usually done in Plan 9″, something along the lines of:

b = "\1\1\1\2";
n = (b.charCodeAt(0) & 0xff) << 24;
n += (b.charCodeAt(1) & 0xff) << 16;
n += (b.charCodeAt(2) & 0xff) << 8;
n += (b.charCodeAt(3) & 0xff);


…which gives us n = 16843010.

Maht then had a look at the series of JavaScript lectures by Douglas Crockford at Yahoo!. The first of the series tells us that bit shifting is not faster than simple multiplication. Maht gave me this code snippet doing the same conversion, but using multiplication instead of bit shifts this time:

b = "\1\1\1\2";
n = b.charCodeAt(0) * 16777216;
n += b.charCodeAt(1) * 65536;
n += b.charCodeAt(2) * 256;
n += b.charCodeAt(3) * 1;


n, is of course 16843010; the real question is how much longer (or, shorter) did this take.

Venkman is probably one of the more mature “old-school” JavaScript debuggers out there. FireBug, the relatively modern sibling to Venkman certainly has a few nifty features, but profiling is not one of its strong points. After failing to profile the script in FireBug, I used the trusty old Venkman – and it came up with some interesting results:

Venkman Profile Report
Created .......... Thu Aug 09 2007 20:18:54 GMT+0530 (IST)
User Agent ....... Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6
Debugger Version . Venkman 0.9.87 [Mozilla rv:1.8.1.6/2]
..
Function Name: multi (Lines 1 - 6)
Total Calls: 1 (max recurse 0)
Total Time: 0.21 (min/max/avg 0.21/0.21/0.21)
Time (ex. calls): 0.21 (min/max/avg 0.21/0.21/0.21)
..
Function Name: shift (Lines 8 - 13)
Total Calls: 1 (max recurse 0)
Total Time: 0.02 (min/max/avg 0.02/0.02/0.02)
Time (ex. calls): 0.02 (min/max/avg 0.02/0.02/0.02)


As you can tell from the function names, multi uses simple multiplication, while shift uses bit-shifting. Turns out that bit-shifting does indeed take a lot lesser time. A Firefox quirk?

Now that the PHP bindings to libixp are somewhat usable, I’ve moved on to the JavaScript portion of my project. The first (and easier!) part of it is to map 9P to a RESTful scheme, so traditional web developers can use 9P without having to learn anything new.

The PHP bindings to libixp was an important pre-requisite to achieve this: presenting a REST interface to an existing 9P serve would require a “bridge” at the middle, to convert REST requests to 9P requests and vice-versa with responses. This “bridge” may be present at any location that is mutually accessible by the client wanting RESTful access and the server providing the 9P service. This bridge can be easily coded in PHP using the new libixp bindings.

To those not very familiar with REST, it is simply a way of accessing resources using plain-old HTTP. It’s become quite popular with web developers these days, as a much simpler alternative to SOAP. A lot of web services these days are RESTful, including those offered by Amazon and Yahoo.

From the client’s perspective, accessing a 9P resource boils down to 4 things: reading, creating, modifying and deleting. These operations map neatly onto the GET, PUT, POST and DELETE HTTP requests, respectively. And thus, we have our REST URI scheme. This scheme would be enough if my bridge exposes only a single 9P serve as a REST service. As an example, suppose I start a bridge at http://plan9.kix.in/rest/ that exposes only the tcp!sources.cs.bell-labs.com!564 9P serve, to read the file named ‘lsr’ I would perform a GET request at http://plan9.kix.in/rest/lsr.

However, the plot thickens when I want to design a bridge that allows access to any 9P service (which is definitely better). Now we need to encode the information represented in Plan 9 as: tcp!sources.cs.bell-labs.com!564 into a HTTP URI. The intuitive thing to do would be something like: [ROOT]/[PROTOCOL]/[9P-URI]/[PORT]/[FILE-PATH].

Hmm. That leads to really long URI’s like http://plan9.kix.in/rest/tcp/sources.cs.bell-labs.com/564/lsr. Besides that, there are several things that need to be worked out. What happens when you do a GET on a file that is actually a directory? What about parameters to GET that you usually pass to a read() function: Suppose you want to read the first 1024 bytes of a file only?

Comments and Suggestions welcome

P.S.: Thanks to some pointers by Kris, the PHP9P client example shown in the previous post now handles binary files correctly.

Hopefully this is the last of the series on Object Oriented PHP Extensions. We left the last post with a question of how to create objects of a certain class in a method of another. This is especially relevant to the libixp extension, since classes like IxpStat, IxpQid and IxpCFid don’t have any constructors of their own, but are rather returned by methods of the IxpClient class. For example, an open() on IxpClient returns a IxpCFid, and the IxpQid is a property of the IxpStat and IxpCFid classes.

How do we get this to work? The helpful folks at #php.pecl on EFNet pointed me to a few examples in the existing PHP codebase that do this. First, you need to initialize an object – which essentially involves:

• Calling ALLOC_ZVAL on the zval.
• Setting the type to OBJECT using Z_TYPE_P.
• Calling object_init_ex(), which in turn will call the corresponding create_object method for the class entry.
• Setting the object’s refcount to 1.

Once you’ve done this, most often you’d need to set up certain properties on the class object. In libixp’s case, we need to populate IxpStat’s, IxpCFid’s or IxpQid’s properties. You can create a method to populate each classes properties. I use a set of PROP_SET_* macros to update properties correctly. Check out the code to clear things up – object_instantiate() is the generic method for allocating memory for a object, and the PHP_*_initialize() set of methods set properties for particular classes. These functions are used in almost all of IxpClient’s methods.

Before signing off, another quick word about using properties in your classes. If your class doesn’t use any properties, you can get away without doing any memory allocation at all, see the FliteTTS extension for an example of this. However, if you do plan on using properties, you need a bunch of methods for allocating memory to the HashTable that will hold the properties, as well as the structures that your class will use. You can create a generic method create_object_ex() that does this, and then create two more methods create_object_new() and create_object_clone() that will act as handlers for new object creation as well cloning. These handlers are set during the class initialization (most probably located or called in PHP_MINIT) by setting the create_object member on the class entry struct and the clone_obj member on the classes’ standard object handlers.