Audit of Learning Technologies in Government Schools

The Victorian Government conducted an audit of Learning Technologies in Government Schools. It was found that the Victorian government's policy for ICT in schools, titled the Digital Learning Statement, lacks a clear action plan and framework for investment in learning technology. As a result there is "... little guidance on how future learning technology initiatives can be appropriately planned and integrated". More positively the planning for a fibre-optic network for schools was found to have a "robust needs and options analysis". However, the software project, called "Ultranet", was "...poorly planned and implemented". The system is 80 per cent over budget, late, has a low uptake rate and limited functionality. It would seem to me that the Victorian Government should retain the network and computer hardware, but replace the software with open source, including Moodle and Mahara.

The audit report is the equivalent of 53 pages of very clear analysis of a complex social and technical issue. The report would be of value for those interested in the issues of technology for education at all levels, not just schools. The Auditor-General has published the report as a well formatted and efficient HTML web page as well as the more usual (and harder to read) PDF.

Conclusion

DEECD’s Digital Learning Statement (the Statement) does not provide a clear action plan or framework for investment in learning technologies. There is no supporting strategy or further detail to the Statement. This means that departmental staff and school leaders have little guidance on how future learning technology initiatives can be appropriately planned and integrated to build upon past and present ICT investments.
Planning for the VicSmart high-speed fibre-optic network for all government schools was underpinned by a robust needs and options analysis. Although the project was a less complex infrastructure rollout, it was well executed and is delivering its desired benefits. The high-speed connectivity that VicSmart provides is a key enabler of current and future digital learning in government schools.
In contrast, the Ultranet, the Statement’s key foundation plank and key enabler, was poorly planned and implemented. Six years after its announcement as a government priority, it is yet to achieve expected benefits for students, parents and schools. It is significantly late, more than 80 per cent over its first announced budget, has very low uptake by users, and does not have the functionality originally intended.
This audit identified a number of serious probity, procurement and financial management issues surrounding the Ultranet project. DEECD’s tender process lacked rigour and was seriously flawed. There is little confidence in the costing and financial management practices around the Ultranet project, and limited assurance that the selected outcome represented value for money.

Findings

Digital Learning Statement

The Digital Learning Statement—the government’s current policy document on the use of learning technologies—was not informed by robust and comprehensive research and does not make a clear and cogent case for government investment in learning technologies.
The Statement does not deliver on the directive in the 2008 Blueprint for Education and Early Childhood Development to provide a plan of action to use learning technologies in teaching and learning. A review of DEECD’s advice to the Minister for Education shows that the government was not advised that the Statement did not comply with the Blueprint’s directive to develop and deliver a strategy.
To date, there has been no accompanying detailed strategy developed to support the Statement, even though this was originally planned.

VicSmart high-speed broadband for schools

Planning for VicSmart was underpinned by a robust needs and options analysis, as demonstrated by its 2005 business case, which articulated the needs to be addressed and provided a clear rationale for the purchase of high-speed fibre-optic broadband connectivity. The business case provided confidence that the project was achievable and could be delivered as planned.
The VicSmart procurement process was streamlined by using a mandated whole-of-government single-source provider. The fibre-optic system is performing as expected and has been upgraded incrementally to meet emerging data and connectivity needs across the government school system.

Ultranet e-learning system

The Ultranet project was poorly planned and implemented. None of its three business cases had a well thought out needs analysis or gave considered options to deliver the project. The various business cases did not answer the ‘Why invest?’ question for the Ultranet, nor did they provide a sound basis for the project’s approval.
Some six years since its announcement as a government priority, the Ultranet has not delivered its main objectives:

• to improve responsiveness to individual learning needs
• to provide better information to parents, the school system and government
• to improve the efficiency of the learning environment and school administration.

Consistent with public sector practice, the Department of Treasury and Finance (DTF) and the Department of Premier and Cabinet (DPC) provided advice to government at key decision points over the life of the Ultranet project.
The project continued despite advice from central agencies that it should cease or be delayed. Further, there is no trail of documentary evidence to explain whether or how DEECD addressed the many critical issues raised by DPC and DTF.
It is difficult to understand why the Ultranet procurement was able to proceed to contract execution, given the significant concerns raised by DPC and DTF, as well as the many adverse ratings that DEECD had received from various Gateway reviews since the project first commenced.
Further, this audit detected a number of serious process and probity issues in relation to tendering and procurement for the Ultranet. DEECD has advised that it has commenced a number of actions and further detailed investigations in response to these matters.
There is little confidence that the financial management practices relating to the Ultranet were sound and that full costs have been adequately recorded. VAGO estimates that actual capital and operating expenditure for the Ultranet was approximately $162 million as at June 2012, and by June 2013 it is likely to have cost approximately $180 million. DEECD has advised that it is currently investigating the financial practices in relation to this major ICT project.
Despite this significant expenditure, no cost-benefit analysis has been conducted to determine whether the Ultranet provides value for money, or whether the same functionality could have been delivered more cost effectively.
Performance indicators for the Ultranet have been revised down over time and do not provide appropriate measures of whether the Ultranet is achieving what the government expected when it funded the project.
Use of the Ultranet is low, and declining. On average, only 10 per cent of students and 27 per cent of teachers logged in on a monthly basis from July 2011 to May 2012.
An underlying factor which has limited the effective implementation of the Ultranet is the significant discrepancy between the original scope of the project and expected benefits and what has actually been implemented and delivered. This underscores the urgent need for DEECD to review whether it should continue to invest in this project.
Further, DEECD did not adequately manage the change processes required to maximise the Ultranet’s acceptance and, therefore, the state’s return on investment. Teachers and parents were not appropriately trained and supported to use the Ultranet. Ultimately, the Ultranet is only a technology tool, and cannot by itself deliver the benefits intended from it.

Recommendations

The Department of Education and Early Childhood Development should:

1. develop a comprehensive and evidence‑based strategy or plan of action for use of learning technologies to underpin and guide the significant investment in ICT for government schools
2. develop performance indicators that measure both VicSmart’s ongoing operational performance and its achievement of intended benefits
3. urgently review its investment in the Ultranet, with a particular focus on:
   • assessing whether the contractor has delivered all functionality as required by the contract and what action, if any, needs to be taken to enforce the state’s rights
   • rigorously assessing its financial management practices and identifying the real, current cost of the Ultranet to determine the extent to which further investment is warranted
   • identifying and addressing the underlying causes of low take‑up rates across the school system by teachers, students and parents
   • providing advice to government on the cost-benefit of decommissioning the system now against continuing to fund and rectify the system so that it can be implemented as originally expected
4. conduct an agency-wide review of its internal tendering, probity and financial management practices in light of the serious issues identified by this audit
5. expedite the provision of guidance to schools on the current status of the Ultranet as the department’s key learning technology investment, and clarify the policy context of schools’ autonomy in purchasing non-Ultranet learning technologies. ...

From: Learning Technologies in Government Schools, Victorian Auditor-General, 12 December 2012