Christmas came early this year.

Today, Google announced their new open source systems programming language: Go. I’m super excited about this, we all have been wondering what Rob Pike has been upto since he joined the big G, and now we know. Not just that, but Ken Thomson, Robert Griesemer, Ian Taylor and Russ Cox were all involved in the project, with Ken doing what he does best, writing compilers in lightning speed If that isn’t a list of heavyweight respectable computer scientists, I don’t know what is!

I think Go is poised to be the dominant systems programming language of the future. Go has nailed almost every aspect of a systems language, though some would say I’m biased. Go has been strongly influenced by Oberon, CSP languages like Limbo, and the standard libraries have tantalizing similarities to Plan 9. We’ve had Limbo and Plan 9 for a while now (more than a decade), but this is where my real love for Google begins to bubble, they took something awesome but unpopular and gave it a push to the masses. There are very few companies in the world who would attract the talent to do this, and even fewer who would open source the results. The attention Go has been getting is just mind blowing. Pike had been doing amazing work at Bell-Labs for quite a while, but none of it even got an inkling of the publicity Go is currently getting.

Google was what Pike needed to prove Utah2000 wrong.

I know one thing for sure, I’ll definitely be using my Plan 9 virtual machine a lot less; now that I can write clean concurrent programs that don’t make my head hurt, both in Linux and OS X. And GCC, I’m not shedding any tears while I bid you goodbye.

On another note, Google also announced today that they’ll be sponsoring free WiFi at a whole bunch of US airports this holiday season. For all its faults, Google definitely seems to be doing the right thing. For how long, it remains to be seen, but so far I’d say their track record has been better than excellent.

UPDATE: John Gruber points out that “judging from the copyright statements, [Go is] not an official Google project”. Could this be a result of the famous 20% time scheme?

I’m back from the EU MozCamp in Prague and we all had a great time! Check out the slides from my talks: Labs Overview and Weave in Depth.

A few people at the MozCamp were interested in Weave’s use of cryptography to protect the user’s data and privacy. Although the specs for the Weave server are available, it may take someone new a while to wrap their head around the whole scheme. I’m going to attempt explaining what crypto operations we do and why we do it in this blog post.

First, let’s get some basic definitions out of the way. Symmetric cryptography means you have one key that can perform both encryption and decryption, and they are complementary operations. For Weave, we use AES with a 256 bit key, and we use it in a mode that requires an ‘initialization vector’ for every decryption. Asymmetric cryptography means there’s a pair of keys (usually called ‘public’ and ‘private’ keys). A piece of text “encrypted” by one key can only be “decrypted” by the other key. Here, we use RSA with a 2048 bit private key.

So, when a user first signs up for Weave using the wizard on their computer, we generate a (random) pair of public and private keys. Next, we use the user’s passphrase to create a symmetric key. This is done using a pretty standard algorithm known as PBKDF2 (short for “Password Key Derivation Function”). The PBKDF2 algorithm requires a ’salt’ value which is also stored on the server. Now that we have a symmetric key, we use it to encrypt the user’s private key and upload it along with the public key to the server. Note that the passphrase is never sent to the server, so if the user’s password ever gets compromised all the attacker can get is their encrypted private key, which really isn’t of much use (especially given that the key is 2048 bits long).

Whenever a particular “engine” is to be synchronized (an engine could be Tabs, Bookmarks, History etc.) we generate a random symmetric key for that engine. This key is then encrypted using the user’s public key (now, one can only retrieve the original symmetric key with the corresponding private key) and uploaded as being associated with a particular engine. All entries (the ‘ciphertext’ property in a “Weave Basic Object”) in that engine are encrypted with the symmetric key that was generated for it.

To make things clear, let’s enumerate the steps we would take to decrypt a single tab object for user ‘foo’:

1. Find the user’s cluster by making a GET request to https://services.mozilla.com/user/1/foo/node/weave. It returns https://sj-weave06.services.mozilla.com/.
2. Fetch the user’s encrypted private key and public key from https://sj-weave06.services.mozilla.com/0.5/foo/storage/keys/privkey and https://sj-weave06.services.mozilla.com/0.5/foo/storage/keys/pubkey respectively. The user’s password is required to access these JSON objects.
3. Ask the user for their passphrase and generate a 256 bit symmetric key from it using PBKDF2 and the ’salt’ found in the privkey object.
4. Use the generated symmetric key and the initialization vector found in the ‘iv’ property of the privkey object to decrypt the user’s private key.
5. Fetch the user’s encrypted tab objects from https://sj-weave06.services.mozilla.com/0.5/foo/storage/tabs/?full=1.
6. Fetch the corresponding symmetric key (the URL is also listed in the “encryption” property of every WBO), in this case https://sj-weave06.services.mozilla.com/0.5/foo/storage/crypto/tabs.
7. Decrypt the symmetric key with the user’s private key.
8. Use the decrypted symmetric key to decrypt any WBO from the tabs collection with the initialization vector found in the ‘bulkIV’ property of the tabs symmetric key WBO.
9. Profit.

A word about the formats in which the keys are actually stored in. All values are Base64. For symmetric keys, the key is stored as-is. For asymmetric keys, I wish we used a standard format like PKCS#12, but we don’t. It’s still ASN.1 though, in some format NSS exports private keys in. You need to do a bit of ASN.1 parsing to figure out the values you’re interested in.

Fortunately, I’ve already figured out most of the details for you – check out my Javascript or PHP implementations of the crypto elements required to decrypt Weave Basic Objects.

Finally, a quick note about why we do all this. Sharing is now reasonably easy, if you want to share your bookmarks with someone, you just need to encrypt the corresponding symmetric key with their public key and they’re good to go. Also, each WBO has it’s own ‘encryption’ property so this can be as granular as needed. Secondly, the passphrase is never stored anywhere (except possibly on the user’s computer) so the server never sees anything other than encrypted blobs of Base64′ed text. Along with making HTTPS mandatory, we think this is a pretty secure way of protecting the user’s data.

If you have other encryption schemes that might fit into Weave’s use cases please let us know! (We’ve already been looking at interesting developments in this area such as Tahoe). I’d also love to hear from you if you have any questions on our current cryptography scheme. We’re constantly trying to improve the security and efficiency of our system so these details are only valid until we change our scheme

Now, go write that third-party Weave client, you have no excuse not to!

It’s that time of the year. Google is, yet again, sponsoring students to write some awesome open source code this summer. If you’re a student, and you’d like to make some money contributing to some of the most well known and exciting open source software projects out there, you’d be missing out on a lot by not applying.

If you’re wondering about what the best way to get started is, check out this great advice page. All projects have also been tagged by programming language and field in this delicious profile. You can also search for ideas here.

I’m going to be mentoring for Mozilla, Glendix (under the Plan 9 Umbrella) and Gentoo this year. Get in touch if you’re interested in any of those ideas

The window for applications opens in a few hours. Good luck everyone!

One of my friends wanted to run a HTTP server on his office machine, but the network it is connected to blocks all ports except 22 (SSH). Sure, he could run Apache on port 22 but that would mean he could no longer login remotely.

I wrote a quick hack in python: muxer.py, that will multiplex incoming connections between an SSH and HTTP server. It is slow, and makes all incoming SSH connections wait for 5 seconds before responding, but it works! The 5 second timeout is required because the SSH protocol specifies that the server should be the first one to send the client it’s version string, and only then will the client respond.

I should probably rewrite it in C at some point. Anyway, here’s your hack for today. Maybe someone will find use for it, or even better, come up with a better solution (this one is certainly the worst!)

As a developer, I have to say that FOSS.IN/08 is possibly the most productive conference I’ve been to until now! In just 5 days, I’ve got more things done than I have in the last 5 months

Let’s start with the Beacon workout: Nandeep joined us via VoIP and we got started almost immediately, thanks to the dynamic nature and small size of our project – we didn’t have any infrastructural trouble as a few other C/C++ projects with huge codebases and complex build systems did. We had a list of 6 tasks in mind, and we managed to complete 3 of them. Salil Kothadia got started with writing a PDO data backend, and promptly submitted the patch to us next day. Thanks Salil, hope you continue to contribute to the development of Beacon (thereby increasing the development team size by 25%)!

I also attended Philip’s workout on porting HTML::Template to Javascript. As mentioned on the Wiki page, we mostly worked on the design during the first half or so, and then moved on to writing a skeleton for the whole framework. I think this is an extremely interesting project, and am very happy to be associated with its birth. Hope we can continue the momentum and work until it is finished.

Perhaps the biggest take-away from the conference for me was the ability to give a lightning talk about Glendix, with several kernel hackers present in the audience. Christoph then kindly offered to review some of the patches during the workout. Even the possibility of Plan 9 binary emulation being considered for inclusion into the main kernel tree is amazing, let alone the fact that I got the guidance of an experienced kernel hacker for a good 2 hours! I think the effort was largely successful – I now have a better idea of what I need to do in order to get a kernel patch in order, and also got a few hints as to how I can implement the missing bits.

My primary focus at the conference was to give a talk on Mozilla Labs and Innovation. I think I managed to stir up a decent amount of interest in the various Labs initiatives. I covered the different ways in which members of the community can contribute, specifically focussing on Weave, Ubiquity and the Concept Series. We even covered how easy it is to actually write an Ubiquity command. I now look forward to increased participation by the Indian Mozilla community in Labs projects. Don’t forget to thank Mary for all the goodies!

All this, apart from regular conference happenings like catching up with old friends, making new ones and free swag (great mugs and t-shirts this time around) makes FOSS.IN/08 one of the most successful conferences I’ve been to so far! I can’t wait for the 2009 edition

Glendix has been making the rounds on the web lately, with coverage from OSNews, Reddit, and even a Russian site, Linux.org.ru. It’s really motivating to see some buzz around the project, giving me the extra boost needed to push the completion of a usable release!

Some of the major obstacles that have to be crossed before making a beta-quality release are the completion of critical synthetic filesystems – especially /net and /draw. We also have to work out the kinks with per-process namespaces and union mounts. I don’t know if we’ll be able to get Rio running as a WM before an official release; if not, Plan9Port’s Rio and WMII are good candidates.

The biggest criticism of Glendix seems to be the reasoning that Plan9 user-space tools are somehow superior to their GNU counterparts, and several people have asked us to substantiate our claims. At this point, however, I don’t think that it is really important, or even relevant. Even if Plan9 user-space tools aren’t somehow better – I think it is generally a refreshing idea to see Linux combined with a user-space other than GNU.

We’re at FOSS.IN this year, where a large number of kernel developers have gathered; and I hope to get some of their valuable input on the problems Glendix currently faces, and maybe even write some code to solve them

Thanks for all the community support and critical comments, they are all very vital feedback for the project and are much appreciated!

It’s been a long night, but Weave 0.2 is finally out! Grab it while it’s hot

The last week was mostly spent sprinting towards this release. We’ve fixed over 75 bugs, the major features being: streamlined first-run experience; passwords, tabs and form history synchronization; switch to NSS-based component instead of openssl; better encryption (AES and RSA instead of DES3); and general performance improvements. For a more detailed overview of the changes, check out the release notes.

Ubuntu users: You can’t use the Firefox that the package manager installs, you need to get a stock build here and use that instead. (This is because Ubuntu doesn’t include some parts of Firefox that Weave depends on).

I’m really excited to be a part of this, because I’m very new to the feeling of being part of a team that writes software that a ton of people use. Within minutes of the release, we started getting feedback from users; which is really awesome, and I doubt I can get this kind of experience elsewhere.

I’m also a little nervous, because I got to work on some of the server-side code that Weave depends on, and I’ve actually never written code that is going to be tested by thousands of people. My only hope is that all goes well and that I didn’t break something. My best moment was when I wiped all the old accounts to start afresh (that means all 0.2 users have to register again, irrespective of whether they had an account before. Weave should automatically popup a registration wizard when you install it). All in all, it’s been a wonderful experience so far, and I definitely look forward to more releases working at the Labs!

On a side note, the Labs upgraded to a really snazzy looking website today, going hand-in-hand with the 0.2 release.

So, I’ll be heading to bed now, and check if the world hasn’t ended tomorrow morning. Enjoy the new Weave release!

It’s almost the end of my second week, and for the last 4 days I’ve been really frustrated trying to write some crypto code in Javascript. Just now, however, I got the 800-line beast to run successfully, and man, am I happy

A little background here, Weave encrypts data to keep it safe (duh). One of the primary goals of Weave is to ensure security of personal data, so at no point in time is the password ever stored anywhere (except if you ask the Firefox password manager to remember it for you, in which case it is also encrypted, but that’s outside the scope of Weave). The current trunk version of weave simply issues calls to openssl to encrypt the data using RSA. The decryption also happens with calls to openssl, so we don’t care in what format the data is encrypted, we just upload the Base64′ed version of the data to the server.

One of the ideas Chris had, however, was to create a web-based client so that users can access their data even when not using Firefox. And thus, one of the constraints here is that the decryption of the data *must* be done on the client-side, and thereby, entirely in portable Javascript. But that would also mean, that the uploaded encrypted data would have to be in a standard format.

Dan had written a WeaveCrypto component that uses PKCS #12 (not a fun standard, mind you) and DES to encrypt data. DES is the usual Triple-DES algorithm, and PKCS #12 defines how the key and initialization vector for DES can be generated from the password. This encrypted data is then encapsulated in a PKCS #7 format, which stipulates ASN.1 encoding. Well… now you know :roll:

It took me about 2 days to find out which standards the component was actually using, because the encryption was done by 3 simple calls to NSS. GDB to the rescue, I wrote a small console C++ application that linked against NSS and then traced the execution of my executable to see what was going on.

The Javascript decryption code is made of 5 basic parts:

• ASN.1 parser: I didn’t write a real ASN.1 parser, it only understands a particular type of ASN.1 encoding that NSS outputs for the very specific case of PKCS#12-Triple-DES encryption. This portion simply reads out the value of the salt, number of iteration and the encrypted text.
• PKCS 12 Generator: This portion takes the salt and number of iterations to generate a 64 byte key and 8 byte initialization vector.
• DES Decoder: This one takes the initialization vector, key and encrypted text to output a decrypted message.
• SHA-1 Hasher and Base64-Encoder/Decoder: Self-explanatory.

I only wrote the first two parts, but it took a whole week. That says something about the PKCS #12 and ASN.1 standards – like – they’re not meant to be used

Anyway, it was fun while it lasted. Hoping to get started on the next portions of the Weave web-based client. Have a fun weekend everyone, I know I will!

My first day at Mozilla was awesome. After a brief introduction session with Julie, during which I also caught up with Mary (I met her earlier at FOSS.IN, Bangalore) – I was shown the way to the ‘S’ building, which is were the labs people work. There are these cute scooters I’ve never seen before that can be used to ride between buildings ‘K’ and ‘S’ – they’re just too damn fun!

I met with my mentor for the Summer – Dan Mills (thunder) and he got me setup with this really huge monitor and power cables for my laptop. I’m still trying to get used to this OS X extended desktop feature, and it feels a little weird typing on the laptop keyboard but looking at the monitor (I’m not a touch typist, but I can type reasonably well without looking at the keyboard). Then, I met with the other folks at the Labs, a small but amazing group of people. Myk is also working at the Labs, and his desk is right behind me – Yay

Right now, the focus at Labs is to roll out a beta-quality version of Weave. As the Firefox 3 launch approaches, we’d like to make sure Weave 0.2 is ready to use around that time. One of the reasons is that Google Browser Sync isn’t going to upgraded for use with FF3 for an undetermined period of time. I’ve been going through the targeted bugs for 0.2 and am trying to fix as many as I can.

I finally checked-in to my alloted apartment yesterday, and since my roommates aren’t scheduled to arrive until next week, I have the entire place to myself for sometime. I saw an episode of How I met your Mother – hooray now I can watch all my favorite american TV shows at the time of airing (not weeks later, like before)

Ok, so my second day has begun and it’s almost lunch time. The american meal timings are a bit off from what I’m used to because most of their meals are compressed to be during the day, and then there’s a huge gap at night between 8pm and breakfast. I’m going to stock up on some knick-knacks for my midnight cravings.

Looks like everyone’s doing one of these around the blogosphere lately, so I’m joining in the fun:

[theghost ~]$ uname -a
Darwin theghost.local 9.2.2 Darwin Kernel Version 9.2.2: Tue Mar  4 21:17:34 PST 2008;
root:xnu-1228.4.31~1/RELEASE_I386 i386

[theghost ~]$ history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head
118 ls
81 cd
61 hg
39 exit
29 vi
24 ssh
24 mate
23 grep
19 rm
9 wget

And for the Linux virtual machine:

anant@tg-nix ~ $ uname -a
Linux tg-nix 2.6.24-gentoo-r1 #32 SMP Sun Apr 13 09:15:20 IST 2008
i686 Genuine Intel(R) CPU T2600 @ 2.16GHz GenuineIntel GNU/Linux

anant@tg-nix ~ $ history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head
142 ls
88 cd
83 sudo
48 vi
33 emerge
30 exit
8 rm
8 mv
7 startx
7 cmake

I’m going to leave it for you to figure out what mate’ is