Last night's ACS Canberra talk started with Funnelback, an Australian web search company formed to commercialize government funded research by CSIRO in 2005. The company provides hosted search services to the Australian Stock Exchange (ASX), Westpac Bank, and the Australian Government. The technology was developed from text retrieval research at ANU in the 1990s. My own small contribution to this work was to suggest the researchers suggest the researchers use web site content for their text search systems and to make available the web sites of the Australian Computer Society and the Australian Department of Defence for the test data. Up to that point the researchers do not seem to have thought of web sites as something worth searching.
Funnelback have now produced their Fluster Clustering Engine, which automatically finds words and phrases in related documents. This is producing "tags" to group information, but rather than someone having to go through all the data creating the categories manually, the system does it automatically. Funnelback can create "also of interest" links, with the aim of not only helping the reader find information, but keep them on the web site longer (without straying to another web site).
On 12 November, 2007 Funnelback announced it won a tender to provide enterprise search for Geoscience Australia. Unlike the public search engines, this will be used for staff to find information on the internal corporate system. As well as internal web pages, the system will search the staff directory, library catalog and a TRIM electronic document management system (EDMS). The last of these is most significant, with Funnelback having built an interface to Tower Software's EDMS.
As happened with the web, this search technology is likely top spread from scientific agencies, out to more administrative and business organizations. Geoscience Australia is a highly competent technical agency (I was at a meeting on metadata there some months ago). Their lead is likely to be followed by other agencies. This may happen in months rather than years.
A few weeks ago I was teaching a course for public servants on Electronic Document Management at the ANU. One issue discussed was how to provide access for the public to government records. The Australian Labor Party had announced its policy for the reform of Freedom of Information (FOI) laws, creating a new Office of the Information Commissioner. So I discussed that on the course, looking at how web based technology could be used to streamline the process. Then I bumped into one of the staff from Funnelback who mentioned thy had already built an interface between the EDMS and their search system.
In the past the way the Australian Government has worked (or not worked) is that each agency, and each part of each agency, guards "their" information from the public, from other agencies and from other parts of their own agency. This is done in the name of national security and personal privacy, but in most cases the information is not sensitive and it is hidden to protect the public servants.
After the November federal election there will be a change in government agencies (regardless of which political party wins the election). Normally this would involve files having to be moved from agency to agency and arrangement of IT systems. Perhaps this would be a good opportunity to change the way the government works. Instead of rearranging the small piles of information, the data could be made accessible online to those who have a need to use it. The system at Geoscience Australia could be expanded to all agencies. Any public servant could search all records of the government for information they had a need for and were authorized to see. At the same time all public servants would be given access to electronic mail and other common office tools via one standard interface. Voice over IP standard telephony could also be provided to all staff.
In the past such a corporate system would take decades of planning and would be unlikely to ever be implemented. However, the Internet and web based technology makes this very much simpler. An Internet based interface could be provided to a common, public service wide email, search and could management system. Government records in standard EDMS conforming to National Archives of Australia standards could be easily searched. IT staff of agencies who say that their systems can't be interfaced should be asked why they have produced incompatible systems, and told to produce an interface or find another job. Similarly those IT staff who say that their agency can't use email and office document standards should be replaced.
A standards based system with access to corporate records would provide the opportunity for a new Australian Government to greatly increase its efficiency and cut costs.