UPDATE: ‘Fatalis’ has pointed out in the comments that the POST should be made to http://goo.gl/api/url with User-agent set to ‘toolbar’. The code now works, Yay!

Google just announced their own URL shortening service. Their service can only be used from the toolbar or FeedBurner, and I don’t particularly like adding extra toolbars to my browser. Maybe I can figure out a way to use their service from the command line?

I downloaded the toolbar XPI, unzipped it and peeked inside. Horribly indented JS awaited me. Nothing jsbeautifier couldn’t fix though. Few minutes later, I arrived at this readable JS function:

var getUrlShorteningRequestParams = function (b) {
    function c() {
        for (var l = 0, m = 0; m < arguments.length; m++)
          l = l + arguments[m] & 4294967295;
        return l
    }
    function d(l) {
        var m = String(l > 0 ? l : l + 4294967296);
        for (var o = 0, n = false, p = m.length - 1; p >= 0; --p) {
            var q = Number(m.charAt(p));
            if (n) {
                q *= 2;
                o += Math.floor(q / 10) + q % 10
            } else o += q;
            n = !n
        }
        m = m = o % 10;
        o = 0;
        if (m != 0) {
            o = 10 - m;
            if (l.length % 2 == 1) {
                if (o % 2 == 1) o += 9;
                o /= 2
            }
        }
        m = String(o);
        m += l;
        return l = m
    }
    function e(l) {
        for (var m = 5381, o = 0; o < l.length; o++) m = c(m << 5, m, l.charCodeAt(o));
        return m
    }
    function f(l) {
        for (var m = 0, o = 0; o < l.length; o++) m = c(l.charCodeAt(o), m << 6, m << 16, -m);
        return m
    }

    var i = e(b);
    i = i >> 2 & 1073741823;
    i = i >> 4 & 67108800 | i & 63;
    i = i >> 4 & 4193280 | i & 1023;
    i = i >> 4 & 245760 | i & 16383;

    var h = f(b);
    var k = (i >> 2 & 15) << 4 | h & 15;
    k |= (i >> 6 & 15) << 12 | (h >> 8 & 15) << 8;
    k |= (i >> 10 & 15) << 20 | (h >> 16 & 15) << 16;
    k |= (i >> 14 & 15) << 28 | (h >> 24 & 15) << 24;
    j = "7" + d(k);

    i = "user=toolbar@google.com&url=";
    i += encodeURIComponent(b);
    i += "&auth_token=";
    i += j;
    return i
};

So, I call getUrlShorteningRequestParams("http://www.kix.in/"); to get "user=toolbar@google.com&url=http%3A%2F%2Fwww.kix.in%2F&auth_token=78925814685". I see in their code that they do a POST request to the service to obtain a JSON return value that would contain the short URL. I punch it in using cURL:

$ curl -v -d "user=toolbar@google.com&url=http%3A%2F%2Fwww.kix.in%2F&auth_token=78925814685" http://goo.gl/
* About to connect() to goo.gl port 80 (#0)
*   Trying 74.125.19.102... connected
* Connected to goo.gl (74.125.19.102) port 80 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.19.7 (i386-apple-darwin10.2.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
> Host: goo.gl
> Accept: */*
> Content-Length: 77
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 405 HTTP method POST is not supported by this URL

Oops! Well, not really, the URL shortener from the toolbar doesn’t work either, I just get the full URL whenever I try to “share” something. Has anybody actually generated a real goo.gl short URL yet?

Their auth_token parameter seems completely superfluous to me as it is generated from the URL itself. Don’t we all know security by obscurity doesn’t work

Christmas came early this year.

Today, Google announced their new open source systems programming language: Go. I’m super excited about this, we all have been wondering what Rob Pike has been upto since he joined the big G, and now we know. Not just that, but Ken Thomson, Robert Griesemer, Ian Taylor and Russ Cox were all involved in the project, with Ken doing what he does best, writing compilers in lightning speed If that isn’t a list of heavyweight respectable computer scientists, I don’t know what is!

I think Go is poised to be the dominant systems programming language of the future. Go has nailed almost every aspect of a systems language, though some would say I’m biased. Go has been strongly influenced by Oberon, CSP languages like Limbo, and the standard libraries have tantalizing similarities to Plan 9. We’ve had Limbo and Plan 9 for a while now (more than a decade), but this is where my real love for Google begins to bubble, they took something awesome but unpopular and gave it a push to the masses. There are very few companies in the world who would attract the talent to do this, and even fewer who would open source the results. The attention Go has been getting is just mind blowing. Pike had been doing amazing work at Bell-Labs for quite a while, but none of it even got an inkling of the publicity Go is currently getting.

Google was what Pike needed to prove Utah2000 wrong.

I know one thing for sure, I’ll definitely be using my Plan 9 virtual machine a lot less; now that I can write clean concurrent programs that don’t make my head hurt, both in Linux and OS X. And GCC, I’m not shedding any tears while I bid you goodbye.

On another note, Google also announced today that they’ll be sponsoring free WiFi at a whole bunch of US airports this holiday season. For all its faults, Google definitely seems to be doing the right thing. For how long, it remains to be seen, but so far I’d say their track record has been better than excellent.

UPDATE: John Gruber points out that “judging from the copyright statements, [Go is] not an official Google project”. Could this be a result of the famous 20% time scheme?

The mere presence of systems like OpenID, Facebook Connect and a host of other identity services on the web today is attestation to the fact.

Authentication should be a feature of the protocol, not something that relies on hacks like cookies. 99% of websites today rely on cookies for authentication for their websites, besides offering custom registration and login pages. This means the browser, as the user’s agent, has no clue of what is going on. A user is forced to manually track myriads of accounts, remember passwords for each of them, and remember what personal information each of them holds. Sure, part of the problem is solved by using password managers (like the one in-built into Firefox, or external programs like 1Password), but even these programs rely on heuristic algorithms to determine if something looks like a login credential or not. There’s no explicit way for web pages to tell your browser: “This is a login form, please fill in details of the user’s identity here” or “These pages are privileged, please give me the user’s identity”. Why is that?

Actually, there is such a mechanism: HTTP based Authentication has been a feature present since HTTP/1.0, but only 1% of sites actually use it. The reason for that is purely cosmetic, most browsers display a very bland modal dialog when it encounters a page that requires HTTP Auth, and sites are unable to customize that interaction. So, the technically right way to do things sucks from a user experience perspective, and websites started adopting alternate means. Someone discovered they could use cookies to store session information on the client, and the whole situation exploded ever since. As a programmer, I feel very sad when I think about the fact that instead of fixing the problem in HTTP/1.1, web-based authentication took the route it did and led to the mess we are in today.

However, I must also state that HTTP authentication doesn’t solve the entire problem – there is still the issue of users having to create an account for every site they want to be part of. This is because there existed no protocols to federate and provide decentralized authentication. That is, until OpenID and OAuth came about. Now we’re at this exciting juncture, and the browser is in a unique position to use these tools together to provide the user with an experience that is secure and easy to use. Every architect will agree that it is indeed a fun challenge to use the state of identity on the web today and make it into something awesome.

This is precisely what the Mozilla Labs team has been thinking about for a while now. Sometime ago, we added support for automagic one-click OpenID logins to Weave. We plan to spin that “feature” out into it’s own extension and build on it, something we call “Weave Identity“, part of the broader “Open Identity” initiative by the Labs. “Weave Sync“, the original extension, will just focus on the synchronization parts so we can tackle these two different problems separately.

So, how exactly are we planning on doing this? Take a look at an initial version of a document describing an in-browser “Account Manager“. We’ve also put up a WEP (which expands to Weave Enhancement Proposal, by the way) describing the raw form of a specification for automatic actions on websites, such as user registration or password changes.

Keep in mind that all of this is in its very early stages (pre-alpha); but that also means it’s a great opportunity for the community to get involved! What are your thoughts on Open Identity? Use the discussion tab on any of those Wiki pages, start a thread on the Mozilla Labs group, or simply leave a comment on this blog entry, and chip in – we’d love to hear from you!

The grand Summer of Code Mentor Summit of 2009 concluded last week and I had the fantastic opportunity of being able to attend on behalf of Gentoo, Plan 9 and Mozilla. What follows is some indication of how awesome the summit was:

(Photo courtesy of warthog from Etherboot)

I met so many folks I’d only interacted with online so far (the classic nickname-to-face matching), but even better was the opportunity to meet folks powering open source projects from so many diverse backgrounds. I met many of my personal rockstars, and learned about a bunch of open source projects I’d never heard of

Also, one of the things that is only possible at an event like the summit was the ability to get a whole bunch of non-linux operating system groups in one room. We had a great discussion, and it resulted in the creation of the “rosetta-os” special interest group. Look for more activity on the common device drivers for non-linux operating systems front soon!

Other sessions worthy of special mention were Open Source Security, Recruiting and Retaining Awesome People, Advanced Trolling (yes, you read that right), and of course the always welcoming Casablanca where I spent most of my time. We discussed everything from our SoC experiences to the Afro Celt Sound System in that room, always full of creative energy and warmth.

After 4 years of participating in the Summer of Code, I am super happy to have finally met the faces behind the program. Every single person I met over the course of last weekend was friendly, intelligent and just generally awesome; that sort of thing doesn’t happen by chance. I feel warm and fuzzy inside to think that I’m actually a part of the revolution that is free and open source software, three cheers to everyone that made it possible!

Offered without comment.

I usually never pay attention to the auto complete suggestions offered by Google, until now, when I found they can be quite amusing. What’s even more fun though, is repeating a search across different country specific sites, to get a idea of what the people of that nation are most worried about. Here’s a sampling, let’s start with India:

#1 certainly explains India’s growing population. We’re also quite obsessive about learning proper English (Outsour Singh is desperately looking to land that call center job) and hacking Orkut accounts. Now, for the Netherlands:

I guess the one take-away from this is that the English speaking Dutch population (which is quite large, mind you) are mostly looking for more info on some romantic comedy from Hollywood. I was also curious about the results for the USA:

Hmm, why are there so many Americans wanting to learn to “tie a tie”? “How to solve a Rubix cube” is about the only intellectual entry to appear on the suggestion list among all three countries, until you realize that it’s actually spelled “Rubik’s”. I wouldn’t be surprised if “Rubix” makes the dictionary soon.

The common theme for all countries seems to be: learning to kiss. Indians are confused between “losing weight” and “reducing weight”, which also explains why everyone wants to get better at English. Some Indians also want to gain weight, a term which is most definitely absent from American searches. Our Dutch friends have no interest in either, I completely understand why; they maintain a very healthy lifestyle by cycling all over the place. The Americans have apparently mastered the art of downloading videos from Youtube, while the Indians and Dutch are still learning the ropes. American women first want to learn to get pregnant and then quickly want to get rid of the resulting stretch marks, while Indian ladies don’t bother with the latter.

The geeks out there will notice the UI improvements on the US version of Google over the other two. I think I’ll stop drawing inferences now

Try your own fun searches to see what auto-suggest has in store! Suggested starting point: “How to use”…

I’m back from the EU MozCamp in Prague and we all had a great time! Check out the slides from my talks: Labs Overview and Weave in Depth.

A few people at the MozCamp were interested in Weave’s use of cryptography to protect the user’s data and privacy. Although the specs for the Weave server are available, it may take someone new a while to wrap their head around the whole scheme. I’m going to attempt explaining what crypto operations we do and why we do it in this blog post.

First, let’s get some basic definitions out of the way. Symmetric cryptography means you have one key that can perform both encryption and decryption, and they are complementary operations. For Weave, we use AES with a 256 bit key, and we use it in a mode that requires an ‘initialization vector’ for every decryption. Asymmetric cryptography means there’s a pair of keys (usually called ‘public’ and ‘private’ keys). A piece of text “encrypted” by one key can only be “decrypted” by the other key. Here, we use RSA with a 2048 bit private key.

So, when a user first signs up for Weave using the wizard on their computer, we generate a (random) pair of public and private keys. Next, we use the user’s passphrase to create a symmetric key. This is done using a pretty standard algorithm known as PBKDF2 (short for “Password Key Derivation Function”). The PBKDF2 algorithm requires a ’salt’ value which is also stored on the server. Now that we have a symmetric key, we use it to encrypt the user’s private key and upload it along with the public key to the server. Note that the passphrase is never sent to the server, so if the user’s password ever gets compromised all the attacker can get is their encrypted private key, which really isn’t of much use (especially given that the key is 2048 bits long).

Whenever a particular “engine” is to be synchronized (an engine could be Tabs, Bookmarks, History etc.) we generate a random symmetric key for that engine. This key is then encrypted using the user’s public key (now, one can only retrieve the original symmetric key with the corresponding private key) and uploaded as being associated with a particular engine. All entries (the ‘ciphertext’ property in a “Weave Basic Object”) in that engine are encrypted with the symmetric key that was generated for it.

To make things clear, let’s enumerate the steps we would take to decrypt a single tab object for user ‘foo’:

1. Find the user’s cluster by making a GET request to https://services.mozilla.com/user/1/foo/node/weave. It returns https://sj-weave06.services.mozilla.com/.
2. Fetch the user’s encrypted private key and public key from https://sj-weave06.services.mozilla.com/0.5/foo/storage/keys/privkey and https://sj-weave06.services.mozilla.com/0.5/foo/storage/keys/pubkey respectively. The user’s password is required to access these JSON objects.
3. Ask the user for their passphrase and generate a 256 bit symmetric key from it using PBKDF2 and the ’salt’ found in the privkey object.
4. Use the generated symmetric key and the initialization vector found in the ‘iv’ property of the privkey object to decrypt the user’s private key.
5. Fetch the user’s encrypted tab objects from https://sj-weave06.services.mozilla.com/0.5/foo/storage/tabs/?full=1.
6. Fetch the corresponding symmetric key (the URL is also listed in the “encryption” property of every WBO), in this case https://sj-weave06.services.mozilla.com/0.5/foo/storage/crypto/tabs.
7. Decrypt the symmetric key with the user’s private key.
8. Use the decrypted symmetric key to decrypt any WBO from the tabs collection with the initialization vector found in the ‘bulkIV’ property of the tabs symmetric key WBO.
9. Profit.

A word about the formats in which the keys are actually stored in. All values are Base64. For symmetric keys, the key is stored as-is. For asymmetric keys, I wish we used a standard format like PKCS#12, but we don’t. It’s still ASN.1 though, in some format NSS exports private keys in. You need to do a bit of ASN.1 parsing to figure out the values you’re interested in.

Fortunately, I’ve already figured out most of the details for you – check out my Javascript or PHP implementations of the crypto elements required to decrypt Weave Basic Objects.

Finally, a quick note about why we do all this. Sharing is now reasonably easy, if you want to share your bookmarks with someone, you just need to encrypt the corresponding symmetric key with their public key and they’re good to go. Also, each WBO has it’s own ‘encryption’ property so this can be as granular as needed. Secondly, the passphrase is never stored anywhere (except possibly on the user’s computer) so the server never sees anything other than encrypted blobs of Base64′ed text. Along with making HTTPS mandatory, we think this is a pretty secure way of protecting the user’s data.

If you have other encryption schemes that might fit into Weave’s use cases please let us know! (We’ve already been looking at interesting developments in this area such as Tahoe). I’d also love to hear from you if you have any questions on our current cryptography scheme. We’re constantly trying to improve the security and efficiency of our system so these details are only valid until we change our scheme

Now, go write that third-party Weave client, you have no excuse not to!

I’m off to the beautiful city of Prague, or “Praha” as it is known locally, for the European MozCamp of 2009. Memories from the MozCamp last year are still fresh, and I’m definitely looking forward to this one!

On Friday, we’re going to be hosting a Labs Hackathon on Jetpack. This is your chance to get to know more about the framework that’s so easy to use that your mom could write an extension with it. Maybe not your Grandma though, you do need to know a bit of Javascript The hack session will last as long into the night as needed for you folks to come up with amazing ideas for Jetpacks and implement them. Drew Willcoxon from the Firefox team and I will be on hand all day to help you, so feel free to come and poke us. Oh, I almost forgot to mention that there’s Free Pizza involved.

On Saturday, I’ll be giving a talk on Weave. With 0.7 just released, we’ll be taking a look at our current state, what’s in store for the future, and maybe a few cool demos. We’re also especially interested in engaging with addon developers to see what Weave can do to make it easier for them to add sync functionality to their addons.

Be there!

Every Master’s student in a research university would have had to face this question at some point during their stay. For some, it’s been a no-brainer. Not for me, it’s easily the most difficult decision I’ve ever faced in my life.

On one hand, there’s the opportunity to work for a corporation, draw a handsome salary and help millions of people today by writing great software. On the other hand, there’s the opportunity to spend the next 5 years of my life with just enough money to survive on ramen, working on a really hard problem no one knows the solution to and derive satisfaction from the fact that my work might help millions of people tomorrow.

Computer science is also one of those fields where getting a PhD doesn’t mean you have to become a professor. Not that I don’t enjoy teaching, as a matter of fact I love teaching, but it’s comforting to know that you can always go back to what you sacrificed. The fruits of research in computer science typically reach mass consumption much faster than other fields, and several silicon valley companies specifically target doctorates for recruiting. Not to mention, you could always dropout — aren’t a lot of great institutions founded that way?

All of this tilts the scale a bit towards PhD, but this decision requires many more months of thinking! What are your thoughts on the matter? Have you had to make such a decision? What did you choose and why?

My last day at Mozilla this summer was last Thursday. I didn’t take a lot of pictures this summer, because, you know, I took a lot last time around. Also, this strategy turned out pretty well because now there are more pictures of me floating around on the tubes! After a longish trans-atlantic flight, I’m back in Amsterdam now resuming work on my Master’s (because hacking on Minix is awesome).

No other internship has been ever so satisfying: over the summer, I worked on a wide range of mini-projects which allowed me to exercise skills ranging from systems to application level programming. I even did a bit of work in the mobile space (turns out programming in limited memory and processing speed is a *lot* different).

One such project that I’m especially excited about is support for video recording in the browser. Yes, there is even a canvas-based live preview of your webcam feed, in addition to Ogg/Theora encoding support! Combined with the audio recording support I wrote sometime ago, some really cool applications are now possible. Skype-like dialer in the browser? Why not?! (*hint* anyone is free to send in a patch for multiplexing the audio and video, they’re currently two separate Vorbis and Theora streams *hint*).

We also had 3 major releases for Weave during the summer: 0.4, 0.5 and 0.6. The last one was especially big, given the completely new, HTML based UI (big kudos to thunder for pulling it off!) and a bunch of other performance fixes. Also, the web UI I wrote last year underwent so many great changes by the wonderful folks at Glaxstar. Now we’re putting up a community design challenge to revamp the UI so we can ship the thing! (*hint* if you’re good at UI design you should participate in the challenge *hint*).

There’s so many more cool things I worked on that I’d like to talk about, but perhaps they deserve a separate blog post. Soon… (I keep promising myself that I should blog more often, it never works).

To add the already good times, my two students in the Summer of Code this year passed with flying colors. Yay!