Google's Infrastructure and Google Traffic

Google presented a Tech Talk at the ANU in Canberra today. Google comes to the university once a year to give some insights to how they do ICT and to recruit students.

This year's talk centered on Google's mapping function, which is developed in Australia. This can be used to easily develop mapping applications, such as for tracking vehicles.

Google's execution model uses racks of x86 computers, each of which has an IDE disk. A custom job scheduler allocates tasks to the CPUs, with the code written in C+. This is in many ways like an old fashioned job scheduling system, and in other ways like the systems used for managing scientific processing systems with very large numbers of CPUs. Google has centers distributed around the world, with the system routing jobs around problems, up to the loss of whole centers.

One interesting point is that HTTP is used for managing the jobs. Also there was a reference to "shards" of data. 64Mbyte chunks of files are allocated by the Google file system. Interestingly the same disk drives on the local clusters seem to be used both for the local temporary files and the long term Google file store.

One of the audience had used the open source version of the Google system for running a government application.

Google has jobs at its office in Sydney for software engineers, UNIX/Linux application administrators and product managers. There are also internships, graduate opportunities and a special scholarship for females. About one third of Google's staff come directly from university. Google has 60 engineers in Sydney and 140 other staff. Overseas students are welcome, provided they meet Australian immigration requirements. Google staff may also work at other offices around the world.

At question time I asked if there could be Australian involvement in the Sahana 2008 Google Summer of Code. This is to add mapping and other functions to the Sahana disaster management system (which I help with). The answer was that this tended to be a northern hemisphere event, but Google Australia is keen to foster Australian involvement.

Perhaps there would also be scope for Green ICT projects with the ACS Green ICT Group and Computers Off Australia (also Google has a new green building in Sydney). One possibility would be some online training materials to teach ICT people about power saving technology.

ps: Last year Google suggested ANU could replace its computer systems with Google's, giving each student a Google email address. There was also an emphasis on the Google free lunch and Google ethos. This did not go well with the audience and this year they did not make the same mistake. The presentation had much more of an Australian flavor to it.