Book on the Future of Australian Universities

In "Raising the Stakes: Gambling with the future of universities" (University Of Queensland Press, 2013), Peter Coaldrake and Lawrence Stedman look at where Australian universities came from and where they might be going. This book is very up to date, addressing recent university reforms by the Australian government and putting them into context. Most usefully they discuss the role of e- learning and of Massive Open On-line Course (MOOCs). Like any good academic work, this book raises more questions than it answers.
But Coaldrake and Stedman are cautious about predicting the future and seem to see universities being much as they are now. 

ps: Thanks to the ADFA library for their copy.

Report on the Nature and Impact of MOOCs

The report "MOOCs: Cutting Through the Hype" by Kate Worlock and Laura Ricci of Outsell Inc, provides a good overview of where MOOCs came from, who is providing them, who is taking them and what the future holds. The report lists the business models for the MOOC providers, from simply acting as a way to market conventional courses, to offering add on services, such as certificates and tutoring for a fee. The authors point out that like other distance education students, MOOC students are older and are from different background to the typical university student and may not be looking for a small amount of training, not a full degree. This is a very good report for university administrators and academics wondering what all the fuss is about.

OUA Free Online Courses Look Good

So far, the Open2 Study Massive Open Online Course (MOOC) system from Open Universities Australia (OUA) looks good and much better than any of the US MOOCs I have tried. I enrolled in "Writing for the Web". OUA seem to have anticipated the sort of worries students have and are starting to introduce us to each other on-line, even before the course formally starts. I will discuss some of this in my "MOOCs with Books" series of talks:  Singapore 24 April, Colombo 28 April, CSIRO 8 July, and Canberra 12 November 2013.

MOOCs with Books in Canberra, Colombo and Singapore

I will be speaking on "ICT trends in Education" at the Australian Computer Society meeting in Canberra, 12 November 2013. Also at the CSIRO in Canberra on 8 July, ICCSE 2013 Colombo 28 April and in Singapore 24 April.

Cut in Federal Funding for Australian Universities

Craig Emerson , Minister for Tertiary Education, Skills, Science and Research, announced a $300M cut in annual university funding, in a "Statement on Higher Education", 13 Apr 2013. Also students will no longer receive a 10% discount on university fees for upfront payment and start-up scholarships will be replaced with repayable loans. A Fact Sheet with more details is available. The funding is to be redirected to primary and secondary education.

These cuts are relatively small, but do come at a time when universities are under threat from new technology. We are likely to see 75% of university education move on-line in the next five years. The question is if this will be a move of students from Australian physical campuses to Australian virtual ones,  rather than see 75% of Australian tertiary education disappear overseas. Australian universities need to make a large investment in re-training of staff, re-equipment and reconfiguration of buildings now, to remain competitive. This will be more difficult with fewer funds, although a reduction in funding may provide the rationale the vice-chancellors need to make rapid changes.

Online Higher Education Working Group

A Coalition Online Higher Education Working Group, chaired by Alan Tudge MP, is looking at how "... online technology improve existing campus-based teaching with all the benefits of interaction, in the classroom and beyond...". Here are my quick thoughts on the topics raised by the working group:

1. Trends in online learning both in Australia and internationally and how this might unfold over the next decade:

Online learning will be integrated into most university courses, with blended learning (on-line plus classroom), using the "flipped classroom" model being  the norm. Pure on-line learning with no classroom will be available as an option, particularly for more advanced and mature students. However, face-to-face instruction will still be required for younger, less experienced students. As a rough guide about one quarter of the students will turn up to optional face-to-face classes, with thre quarters finding the on-line mode adequate. The Australian Computer Society is a leader in on-line learning with its masters level, globally certified CPEP Program. The use of online learning at a traditional university is discussed in: "A Green Computing Professional Education Course Online", for 7th International Conference on Computer Science & Education (ICCSE 2012), July 2012, Melbourne, Australia.

2. Assess the benefits of online learning for Australian students and the Australian economy, including the potential:

a. Impact on cost, flexibility, customisation and quality: Online learning is lower cost when used with large volumes of students. However, this requires a high initial investment in course design and highly trained staff. Most Australian university academics are not currently qualified to design or deliver on-line courses.

b. Benefits of enhanced choice, including from global players: Global choice of online courses will be limited by accreditation requirements. There is likely to be a flood of poor quality, unaccredited courses.

c. Impact on workforce participation and democratisation of learning: Online learning allows for learning while employed and in remote areas. However, being a student is still hard work and requires considerable amounts of time.

3. Assess the challenges of online learning and how they could be overcome, including:

a. The challenges to existing institutions and their preparedness to face them: Institutions will require staff to be retrained in online pedagogy and use of technology. This retraining can make use of online technology. A stepwise approach, which would see graduate tutors trained in basic online teaching, is proposed in "Report on Incorporating Professional Skills in the ANU Master of Computing", February 2013.

b. The maintenance of quality and standards: Online learning can enhance quality and standards through more detailed and frequent monitoring. However, measures to prevent cheating by students will need to be enhanced. Also teaching staff and students will require explicit training in how to communicate online.

c. The technological and infrastructure requirements of online courses: The learning technology already being introduced to universities and vocational institutions can be ungraded for larger scale use. In particular, Australia is a world leader in learning management systems, by virtue of the locally developed Moodle system. Ways to leverage this technology for MOOCs is discussed in "MOOCs with Books: Syncronisation of Large Scale Asynchronous e-Learning",  for 8th International Conference on Computer Science & Education (ICCSE 2013), April 2013, Colombo, Sri Lanka.

4. Assess what policy measures are required to capture the benefits for Australian students and the economy, including:

a. How do deal with accreditation: Current accreditation processes for domestic courses will require minor modification for online courses. International accreditation will require government or industry agreements. Work by the Australian Computer Society in accrediting global professional qualifications can serve as a model.

b. How to best assist Australian providers: Governments can assist Australian providers by removing barriers which prevent online courses from being undertaken by students. Government can also set an example by allowing their own staff to undertake online courses and teach in them. As an example, the online engagement courses proposed by the Department of Finance for all Australian Public Servants could be delivered online by Australian tertiary institutions.

c. How to capture the opportunities that international institutions provide while fostering Australian ones: Australian institutions can be encouraged to join international partnerships for online course creation.

d. What other regulatory changes are required to capture the benefits of the emerging environment: Visa restrictions which limit International students undertaking online courses should be relaxed. Currently an international student is expected to be on campus for about three quarters of a full time course. This should be reduced to one quarter.

5. Determine how Australia’s tertiary institutions can best capture the growing online international market, particularly in Asia. This would include:

a. Assessing the size of the opportunity: Australian experience suggests that all university courses will be delivered at least in part online, with the typical student spending about 25% of their time in a face-to-face class and 75% online. Undergraduate degrees will have more face to face components, and  postgraduate courses less. This will result in a drop in Australian students on campus, but allow for millions more online from our region.

b. Determining how Australia can grow an online international market without compromising our on-shore market: With most education online, Australian and international students will be able to choose courses from anywhere in the world. Retaining a proportion of the Australian students and capturing part of the international market will require maintaining Australia's very high standards for accreditation. A national course recognition program, so students can combine courses from multiple Australian universities, would be useful to promote "Brand Australia"  (I undertook a combination of ANU blended courses and USQ pure online ones for a Certificate in Higher Education). The Australian government can also fund Australian academics to reach out to the region.In November 2012 I was invited to talk on "Sustainable Development Through Green ICT: The Role of Education and the Business Sector", in Indonesia. The proposed "new Colombo Plan" has potential as a way to help promote Australian education in the region. However, as with the existing Colombo Plan, this may become enmeshed in geopolitics.

c. Determining what measures should be put in place to help capture the opportunities. Australian universities are already reconfiguring their campuses with accommodation and "learning commons" to suit the new blended learning. Campuses are being redesigned to be pleasant places to be, not just study. See also: "On-line Professional Education For Australian Research-Intensive Universities in the Asian Century", November 2012.

d. Identifying regulatory barriers that need to be addressed: No specific regulatory barriers exist, with online courses being covered by existing regulations on education. However, it would be useful for the Australian Government to advance bilateral and multilateral agreements on mutual recognition of educational standards. Support to professional bodies to advance standards for disciplines would also be of assistance.

Student Loans

As someone from outside the USA I find the discussion of the “gainful employment” regulations made by the U.S. Department of Education a little hard to understand. As I understand it, the US government provides low interest student loans to students to study, much like Australia and other countries. These loan schemes are frequently controversial, with issues of if students take on more than they will be able to repay.

In "Now What?" (Inside Higher ED, March 21, 2013), Paul Fain writes that U.S. Department of Education “gainful employment” regulations,have been struck down by a court. The regulations were to measure performance of vocational programs in education institutions. One measure was the percentage of graduates repaying their loans. This seems a curious measure to use and one a court might well strike out, as the educational institution would have little say on if its graduates pay their loans or not. It would be reasonable to see if the courses provided a qualification in a job category in demand, what the rate of employment of graduates was and how much their salary went up. However, such measures are used in Australia only to provide potential students with information when choosing courses, it is left to the students to decide what courses to take.

As an example, here are the statistics for Computer Science graduates from the Australian National University (where I teach),  as reported by the Australian Government's "My University" website:

Overall satisfaction rate (opens in a new window)	79.2%
Good Teaching Scale (opens in a new window)	45.8%
Generic Skills Scale (opens in a new window)	85.4%

Graduate Outcomes (Average of 2009 to 2011)

Graduate Outcomes (Average of 2009 to 2011)	Value
Graduates in full time employment (opens in a new window)	77.9%
Graduates in full time study (opens in a new window)	9.6%
Median graduate salary (opens in a new window)	$54,000

The ANU rates well, except on the "Good Teaching Scale (45.8%). Staff have been encouraged to get trained in the latest education techniques and make full use of technology, to improve teaching and I have found this works Even before finishing the ANU's certificate in Higher Education, my student satisfaction scores increased, after applying what I learned on the program. I discuss some of what can be done, in "MOOCs with Books".

Tony Abbott Address to Universities Australia Higher Education Conference

The leader of the federal opposition, Tony Abbott, made an Address to Universities Australia Higher Education Conference in Canberra, 28 February 2013.  He started by referring to his studies at Oxford University and his time boxing there (but did not mention Oxford has been offering on-line courses since at least 2009).  at He pointed to education as Australia's biggest export, after minerals. He proposed a "... new, two-way street version of the Colombo Plan building on the original one that brought tens of thousands of students from around our region to study in Australia", with the the Menzies Research Centre hosting a policy development roundtable to work on a "... Rhodes Scholarship for our region".

Mr Abbott also expressed a wish for Australian universities to take advantage of online learning and Massive Open Online Courses (MOOCs), however no funding or other support was proposed for this. A Coalition Online Higher Education Working Group, looking at how "... online technology improve existing campus-based teaching with all the benefits of interaction, in the classroom and beyond..." (chaired by Alan Tudge, with Karen Andrews, Senator Fiona Nash, Senator Stephen Parry, Nola Marino, and Rowan Ramsey). These are very timely questions to be asked, as on-line courses are transforming education, including university education.

It is not clear if MOOCs will be more than a 21st Century version of the DOT.COM bubble, with no cl;ear business case. But it is clear that e-learning is an effective form of education. The question is how to implement it while maintaining the cultural and economic benefits of the Australian education system. For my own take on this, see: "On-line Professional Education For Australian Research-Intensive Universities in the Asian Century" and "Proposal for Teaching ICT Masters Students How to Teach On-line".

THE COALITION’S ONLINE HIGHER EDUCATION WORKING GROUP

Terms of Reference

Overview

Australia enjoys an enviable international reputation for delivering world class higher education.

International education is Australia’s fourth biggest export and our largest services export industry.

Like so many other parts of our society and economy, the Australian higher education sector can rightly stand proud of its achievements and international reputation. But resting on past achievements is never as important as looking toward the challenges of the future.

Perhaps the greatest challenge is the digital revolution and the global trend toward increased reliance on technology and online resourcing. Just as the digital revolution is transforming printed media and retail, it will almost certainly transform higher education.

The Coalition understands that the possibilities and benefits of online learning opportunities are enormous for all involved in higher education. We firmly believe that Australian higher education institutions are uniquely placed to take advantage of future online learning opportunities.

Our history, current international standing and world class reputation for excellence are all factors that potentially give Australia a competitive advantage when it comes to online education opportunities.
The biggest danger is for Australia to be left behind.

The Coalition knows that higher education institutions are best placed to deal with future challenges and opportunities.

The task for government is to ensure public policy provides the capabilities and incentives so that our students can tap the opportunities that online higher education can deliver. Most importantly, the Coalition believes it is crucial that government does whatever it can to ensure higher education providers are able to embrace online opportunities.

Traditional higher education models are vitally important and while they remain a central focus of government policy, we must also have an eye on the future.

We believe online learning has great potential to increase access and choice for students, enabling greater customisation and a better balance with work and family demands.

The online trend cannot be ignored The potential benefits of online learning for Australian students, higher education institutions and the economy are enormous.

Online education can allow students to access more customised, effective and efficiently delivered education at the time of their choice. For students living in remote areas, or possessing disabilities or juggling work or family responsibilities, greater online education will make access to higher education easier.

It also has the potential to speed up learning, allowing students to save money and enter the workforce more rapidly. The traditional university model is built around lecture-theatre and academic availability during a 26 week per annum schedule. People increasingly expect to be able to work, learn and study whenever and wherever they want.

Other countries are moving rapidly The most important evidence of how online education is transforming higher education can be found by looking at the growing involvement of some of the world’s most prestigious universities, including Harvard, Massachusetts Institute of Technology and Stanford. These institutions have begun to offer premium learning content in the form of “massive open online courses” (MOOCs).

MOOCs are offered by these universities through online providers such as Coursera and edX. Over two million people have registered for these courses. Within six months of joining Coursera, Melbourne University had more students online than on campus.

Currently, the vast majority of MOOCs are free of charge, are not assessed or graded and do not provide students with credit toward the completion of their university degrees. The next stage in the global development of online higher education is for online courses to be accredited and incorporated within the existing programs offered by universities.

Online challenges must be faced The increased choice, flexibility and lower cost will be beneficial to Australian students and our economic productivity, but it will be immensely challenging to existing Australian institutions. Our biggest danger is to simply ignore the overseas trends which, according to some forecasts, could have negative ramifications for current university models.
As well as producing the skilled graduates needed to drive productivity and long-term growth, Australia’s universities also educate hundreds of thousands of international students and in doing so generate $15 billion in export income each year (our leading services export industry).

There is potential to expand this greatly through online offerings. Over the next forty years, the demand for education from our region will be enormous. The OECD predicts the middle class in the Asia Pacific to grow six-fold in the next 20 years to 3.2 billion people. In India alone, the working age population will grow by 240 million over the next 20 years. There is insufficient capacity in their education system to cope, as McKinsey & Company have identified.
India’s online education market will be worth $40 billion by 2017, presenting huge opportunities for Australian institutions.

The opportunities from the online education trend are enormous, but so are the challenges.

Better approaches for Australia Labor has not capitalised on the opportunities that the online trend presents.

Labor has rigidly held onto the traditional learning model founded in “bricks and mortar” teaching and has neglected addressing the challenges facing higher education. Labor’s base-funding review of higher education ignored online opportunities and the potential for more effective and efficient learning methods.

For the past five years, the Government has focused on funding formulae, visas and regulations while Australia has declined in education export revenues. This has been particularly noticeable over the last two years despite other competitor countries continuing to grow their share.
This highlights the need to consider better approaches to ensure Australia is not be left behind.

Traditional higher education environments are important, but the challenges and opportunities of the future must also be considered.

Objectives and scope of the Working Group As part of the Coalition’s commitment to implementing innovative, forward-looking higher education initiatives that will drive increased productivity and economic growth, the Leader of the Opposition, the Hon Tony Abbott MHR, has established the Coalition’s Online Higher Education Working Group to examine the potential for greater online delivery to be incorporated within Australia’s existing higher education framework.

The Working Group will be chaired by Mr Alan Tudge MP and its other members will be Senator Fiona Nash, Shadow Parliamentary Secretary for Regional Education, Karen Andrews MP, Senator Stephen Parry, Ms Nola Marino MP and Mr Rowan Ramsey MP.

The Working Group will liaise closely with Shadow Minister for Universities and Research, Senator the Hon Brett Mason and Shadow Minister for Education, Apprenticeships and Training, the Hon Christopher Pyne MP.

The Working Group will examine the key issues and challenges related to the development of online higher education in Australia. The Working Group will consult widely with all relevant stakeholders, including tertiary providers, employer groups and the Tertiary Education Quality and Standards Agency, to inform its response to these issues and challenges.

Specifically, the Working Group will:

1. Assess the trends in online learning both in Australia and internationally and how this might unfold over the next decade.
2. Assess the benefits of online learning for Australian students and the Australian economy, including the potential:

a. Impact on cost, flexibility, customisation and quality.

b. Benefits of enhanced choice, including from global players.

c. Impact on workforce participation and democratisation of learning.

3. Assess the challenges of online learning and how they could be overcome, including:

a. The challenges to existing institutions and their preparedness to face them.

b. The maintenance of quality and standards.

c. The technological and infrastructure requirements of online courses.

4. Assess what policy measures are required to capture the benefits for Australian students and the economy, including:

a. How do deal with accreditation

b. How to best assist Australian providers.

c. How to capture the opportunities that international institutions provide while fostering Australian ones.

d. What other regulatory changes are required to capture the benefits of the emerging environment.

5. Determine how Australia’s tertiary institutions can best capture the growing online international market, particularly in Asia. This would include:

a. Assessing the size of the opportunity.

b. Determining how Australia can grow an online international market without compromising our on-shore market.

c. Determining what measures should be put in place to help capture the opportunities.

d. Identifying regulatory barriers that need to be addressed.

The Working Group will provide a report on these matters to the Leader of the Opposition by 15 June, 2013.

Contributing to the Working Group Written submissions to the Working Group are invited by 30 March, 2013 and should be addressed to:

The Office of Mr Alan Tudge MP
Suite 4, Level 1, 420 Burwood Highway Wantirna South, VIC 3152

Or sent electronically ...

To arrange a private meeting with the Working Group or to obtain further information about the Working Group, please contact ... the office of Mr Alan Tudge MP ...

From: THE COALITION’S ONLINE HIGHER EDUCATION WORKING GROU: Terms of Reference, Alan Tudge, 28 February 2013

Necessity of Student Course Evaluation

In "The Unnecessary Agony of Student Evaluations", Spurgeon Thompson (1 March 2013,The Chronicle of Higher Education) writes that American universities are treating students too much like consumers, with excess emphasis placed on student feedback evaluations of courses and teachers. They contrast this with the European approach, where universities are mostly free, but difficult to get into and argue that American universities do not have as much consumer choice as it may appear.

In Australia, my impression is that we have, as in so many fields, steered a middle course, between the USA and Europe, with some use of student feedback. While lecturers grumble about the feedback results, it is having a positive effect. Some of my colleagues who have for years looked with at best tolerance on my use of new teaching practices are now coming t me to ask how it is done. The reason they are asking is that my student feedback scores are very high and theirs are very low. It is not that I set out to get high scores, it is just a by-product of good teaching. While lecturers may be reluctant to acknowledge that their teaching practices are out of date, it is hard to argue when the Dean wants to know why you are dragging the faculty results down.

One point I differ with Thompson on is the choices USA student have. Thompson argues that US universities effectively have a captive market, a US student has no choice but to attend a US institution. However, a student can choose to study overseas if they wish, they can do this by physically moving, or, increasingly, study on-line. Not only will US campuses have to increasingly have to compete with overseas ones, but also with their own virtual campuses. While much has been made of MOOCs, the real revolution is in the use of new more responsive teaching methods, which can be used on-line , in a classroom, or a blend of both. One a student has experienced teaching which does not talk down to them and responds to their needs in a timely and supportive fashion, they will not want to go back to old fashioned "Shut up and take notes" lectures.

Australian Open Access Support Group

The Australian Open Access Support Group (AOASG) has  announced its new web site. AOASG is an open access to research advocacy group set up by six Australian universities (ANU, Charles Sturt, Macquarie, Newcastle, QUT, and Victoria). AOASG point to the Australian research already freely available on-line, including Australian research in Trove, provided by the National Library of Australia ( get a few entries in the system).

The launch of the new AOASG website was not without a few problems. The emailed media release I received actually took me to a Microsoft Outlook login screen at login.live.com. This was after my system warned me it might be a scam. I assume AOASG are using some system to track how successful the announcement was. This can be counterproductive, if it results in the message looking like a scam  or the links not working at all. Also ending up at the Microsoft site is not a good look for an "open access" independent organization, as it makes it look like a front for Microsoft.

I manually entered the web address and that worked okay. Here are the results of the usual checks I run on new websites:

1. W3C Markup Validation Service: 5 Errors, 2 warning(s) Not unusual and easily corrected.
2. W3C mobileOK Checker: 68% which is a good score (many websites score zero): 0 critical, 1 severe, 1 medium, 5 low failures.
3. AChecker: Accessibility Review (Guidelines: WCAG 2.0 (Level AA))" two "Known Problems", much less than the usual number and easily fixed.

The major question the home page doesn't answer is: Who is in the AOASG and what do they provide which I can use? I suggest AOASG need to put the list of organizations which is on the membership page also on the front page, preferably with the logos of these organizations. AOASG also need something on the front page which is of use to the average academic, if not member of the general public, such as a search box where they can find open access content from the AOASG members and a toolkit for implementing open access in their organization.

Nothing is Wrong With Our Universities

Richard Hil, author of Whackademia: An Insider's Account of the Troubled University, argued on ABC Radio  ("What's Wrong With Our Universities", Sunday 20 January 2013) that Australian universities are under-funded and its academics overloaded with administrative work and pressure to pass sub-standard students. He looks back to a golden age when academics where funded without any accountability and could sit around thinking great thoughts. However, I doubt such an age ever existed. Universities cost money and someone has to pay. If academics want to sit around and think great thoughts, then ether they will have to think up something very useful occasionally which justifies they salary, or do something else, such as teach, to subsidize their thinking time.

Hil expresses concern that there is are a countless number of part time university teachers who do not have the pay and conditions of permanent staff. This seems to contradict other parts of his argument where he calls for less administrative burden on academics and more flexibility. As an adjunct lecturer I like the relative freedom of not being included in university administrative processes. I don't have to go to administrative meetings, unless they concern my course. I don't have to fill out leave forms, because I don't get paid for leave, just for the courses I teach.

Hil claims there is pressure from administration, even from VCs, for academics to pass students who are not up to the required standard, particularly international students. I have never been under such pressure. Any academic who is, should report the matter to the appropriate internal review body, or if it appears to be a systemic problem to the independent commission against corruption (or equivalent).  There will always be legitimate debate about marks, but if inappropriate pressure is being applied for financial gain, then that is a crime which needs to be reported and investigated.

Universities have to balance the need to be financially viable with research and teaching. There is also the need to balance the need to turn out "work ready" graduates and to create new knowledge. There is never going to be enough money to do everything every academic wants and everything society expects.

US Education Bubble About to Burst

William Baldwin suggests a USA education bubble may be about to burst (The Scary Economics Of Higher Education, Forbes, 15 January 2013). He points to US universities and colleges investing in new campuses and buildings, funded by tax-exempt bonds. Baldwin suggests that also surplus educational places could drive down tuition fees to the point where educational institutions can't service their debts. While not as large as the previous the housing bubble, Baldwin suggests this is similarly caused by government subsidies and tax deductions encouraging over investment. Individuals  encouraged by government policies, are also investing in education which does not lead to jobs with high enough wages to pay back their loans.

The other risk pointed out is from on-line education providing low cost or free courses. Baldwin suggests that on-line courses do not provide the social experience of a campus, but my recent studies of education suggest that there are ways to provide the collegiate spirit on-line, with blended courses. University campuses are changing to look more like a mall, airport business lounge and a leisure center. Australian universities are turning their libraries into learning centers and providing on-campus accommodation and entertainment facilities, backed by private investment. There are fewer traditional lecture theaters being built, but labs and places for learning circles to meet are in demand.

Baldwin suggests a blended four year degree, with a 50% split between on-line and on campus study. I suggest this is more likely to be 75% on-line or more. The typical student will do most of their study on-line, visiting a campus only for essential lab sessions and intensive workshops. They may get together with other students, at a local learning center of a public library or a coffee shop.

One curious suggestion by Baldwin was the threat of the "European" three-year degree. Presumably Australia's three year degree and adoption of the European style Bologna Process for higher degrees would also be a threat to US institutions. However, the low US dollar may cushion US institutions from overseas competition, at least from Australia.

University Merchandise Suggestions

While visiting University of Queensland I browsed through the UQ branded merchandise in the bookshop. Back in Canberra on my way to Indonesia I looked at items in the ANU store. UQ had some useful "bucket" hats and ANU keyring torches. But here is my wish-list for university merchandise items:

Long sleeve batik shirt: This is referred to as "Territory Rig" in the Northern Territory and is the official dress code for semi-formal government functions. Uin Suska staff wear batik shirts as a uniform (and presented me with one on a visit) and this makes a very sensible alternative to a suit and tie for a hot climate. Perhaps Australian universities could have a batik shirt in their colors for staff and students. This could reduce energy use and carbon emissions, as it would allow the air-conditioning to be turned up a few degrees. Here is a photo of then Prime Minister Paul Keating and other leaders in Batik shirts at the 1994 APEC meeting in Indonesia.

Adjustable Boonie Hat: This is a soft cloth hat with a broader brim than a bucket hat. The adjustment strap around the hat results in one size fitting most people.

LED Flash-light on keyring  with one AAA battery, such as the MAGLITE K3A016 AAA Solitaire Flashlight. The torch itself doesn't needed to be branded with the university crest, just pair it with a keyring. Many of the tiny torches have button cell batteries which do not last long, are harder to get and cost more than AAAs.

ANU Ranked Oceania's Top University on the Web

The Spanish National Research Council puts out an annual "Web Ranking Universities". The July 2012 edition has the Australian National University as the top ranking university in Oceania. The rankings are based on measures of the depth and quality of the university's web presence. In general universities are ranked by web presence in much the same order as other measures of university quality. The web ranking has ANU, University of Queensland and Melbourne University, as the top tree in Australia. Other rankings have the three, but in different orders.

While ranking top in Australia overall, the ANU only ranked first on one of the four measures which make up the web ranking:

• Openness (15%): First, as measured by the number of "rich" files provided.
  
• Impact (50%): Second by the number of links from external web pages
• Presence (20%): Third for the number of web pages,
• Excellence (15%): Sixth for the quality of research publications.

One obvious area for improvement is in excellence. The compliers of the web ranking use data from Scimago group, which may not correctly reflect ANU's status, as measured by other rating systems.

Openness might also be improved by checking that document files end in pdf, doc, docx or ppt. There may be under-reporting where other file formats are used, such as Postscript (.ps).

Presence may be effected where web pages for the institution are hosted under a different domain name which the ranking system does not count, such as institutes.

Open University Discussion Radio National Life Matters

ABC Radio National Life Matters had a discussion of Open University today with Professor Simon Marginson, Centre for the Study of Higher Education at Melbourne University and Associate Professor David Glance, Director UWA Centre for Software Practice.

Mass open online courses, offered by American universities, are changing the university landscape. Australian students now have the opportunity to enrol in subjects prepared by Stanford or Harvard. So will this virtual campus reshape the offerings at Australian universities? Just how satisfying is the online learning experience compared to setting foot on a real-life university campus?

I posted this comment:

Tom Worthington :

28 Aug 2012 9:28:57am

Australian universities can compete to provide on-line courses. As well as courses for large number of students, there can be smaller ones, providing traditional small group tutorials for students.

Since 2009 I have been teaching "ICT Sustainability" on-line from the Australian National University in Canberra. The tutorial groups are small, with the students and staff getting to know each other and share experiences. Most students are on campus in Canberra, but others are around Australia and th world. They can share their work experience from their desk at work and use what they have learned immediately in the workplace.

An "Innovation in Education Showcase" will be held at the Australian National University in Canberra, 17 to 23 September 2012. As part of this I will be speaking on "A Green Computing Professional Education Course Online using eBooks", 12:30pm, 19 September 2012, in the Ethel Tory Centre: http://tomw.net.au/technology/it/green_computing_professional/

Research Supervision Guidelines

The Monash University Research Degrees Handbook has a chapter on research supervision. covering Appointment of supervisors and a Code of practice for supervisors. Like many codes of practice for research in Australia this calls up the Australian Code for the Responsible Conduct of Research although that is from the National Health and Medical Research Council and so is primarily concerned with experimentation on human subjects.

UK Professional Standards for Higher Education Staff

The UK Higher Education Academy has issued a "UK Professional Standards Framework" (UKPSF) to define the roles of teaching and learning support staff in UK universities. The framework has three Dimensions of Practice: Areas of Activity, Core Knowledge and Professional Values.

The Academy also provides four levels of accreditation, in increasing order:

1. Associate Fellow AFHEA
   
2. Fellow FHEA
3. Senior Fellow SFHEA
4. Principal Fellow PFHEA

The only formal qualification required seems to be a certificate in higher education teaching and similar Academy-accredited professional development.

Performance Measures for Australian Universities

Senator Chris Evans, Minister for Tertiary Education, Skills, Jobs and Workplace Relations issued three papers for comment on the performance measures for Australian universities, 9 December 2011. These measures will be reported on the the "MyUniversity" web site (modeled on the MySchool website) , starting during 2012.The discussion papers for comment are (excerpts appended):

1. Development of Performance Measurement Instruments in Higher Education – exploring the balance of performance measurement instruments, potential uses, deployment methods and participation and selection of students;
2. Review of the Australian Graduate Survey (AGS) – examining the strategic position of the AGS in its relationship with other survey instruments, administration methods, timeliness and capacity to measure the diversity in student experience; and
3. Assessment of Generic Skills – focusing on the development of an instrument appropriate for Australian universities in the context of the OECD’s Assessment of Higher Education Learning Outcomes (AHELO) project.

The proposed Australian measures are about student satisfaction and academic results. They do not appear to cover ancillary costs of studying or job success after graduation, as the UK Metrics for Universities do.There could be considerable gaming of the figures (as has happened with "My School" website). For example, in the case of costs of studying, universities could make blended courses look cheap, by assuming that students will be distance education students. That is the university can say it has a wonderful range of on-campus facilities and accommodation, but assume the students never use them, when calculating the cost of study.Of course that might be an accurate view of modern student life. I am a postgraduate university student at the moment. Six weeks into the course I just submitted my first mid semester assignment on Friday. The university has excellent facilities for students, but I have never been to the campus, as it is more than a thousand kilometers away.

Development of Performance Measurement Instruments in Higher Education

Discussion Paper

December 2011

Table of contents

1.Introduction 3

1.1.Performance Funding 3

1.2.Advancing Quality in Higher Education 3

1.3.Purpose of this paper 3

2.Principles and the student life cycle framework 4

2.1.Principles 4

2.2.Student life cycle framework 4

3.Existing surveys 7

3.1.National and cross-institution surveys 7

3.2.Links with new instruments 7

4.New instruments 8

4.1.University Experience Survey 8

4.2.Assessment of Generic Skills : The Collegiate Learning Assessment 9

4.3.Review of the Australian Graduate Survey 9

5.Issues 12

5.1.Administration of new instruments 12

5.2.Student selection 12

5.3.Central sampling of students 13

5.4.Uses of data 14

5.5.Intersection of existing and new instruments 15

6.Next Steps 16

6.1. The AQHE Reference Group 16

6.2.Discussion papers 16

6.3.Roundtable discussions 16

6.4.Next steps 17

Appendix 1 - References 18

Appendix 2 – How to make a submission 19

Appendix 3 – Terms of Reference for the AQHE Reference Group 21

Appendix 4 – Membership of the AQHE Reference Group 22

Appendix 5 – Summary of existing surveys 23

Introduction

In 2008, the Government launched a major review to examine the future direction of the higher education sector, its fitness for purpose in meeting the needs of the Australian community and economy, and options for reform. The review was conducted by an independent expert panel, led by Emeritus Professor Denise Bradley AC. The panel reported its findings to the Government in the Review of Australian Higher Education (the Review) in December 2008. The Review made 46 recommendations to reshape Australia’s higher education system.

In the 2009-10 Budget, the Government responded to the recommendations of the Review with a ten-year plan to reform Australia’s higher education system, outlined in Transforming Australia’s Higher Education System. The Government’s response was based on the need to extend the reach and enhance the quality and performance of Australia’s higher education system to enable it to prosper into the future.

To extend reach, the Government holds an ambition to increase the educational attainment of the population such that by 2025, 40 percent of all 25-34 year olds will have a qualification at bachelor level or above. The Government also seeks to increase the higher education participation of those people who are currently underrepresented in higher education. In particular, by 2020, the Government expects that 20 per cent of higher education enrolments at undergraduate level will be of people from low socio-economic backgrounds.

Reforms announced to achieve these ambitions included the establishment of the Tertiary Education Quality and Standards Agency, the introduction of the demand driven funding system and significantly improved indexation on grants, Performance Funding and mission based Compacts.

The Government’s response to the Review shares a number of features with reform agendas in other areas of “human capital development” such as health, employment services, and disability services that are being implemented in Australia and internationally. These common features include: opportunities for citizens to exercise greater choice between alternative providers; the introduction of funding that “follows the consumer” and thus gives them more power in the service relationship and strengthens incentives for providers to tailor their offerings to citizens’ requirements; improved regulation to ensure minimum quality standards; and improved information on performance to allow citizens to make better informed choices.

The Government’s efforts to improve performance reporting and transparency are aimed at enhancing the quality of information available to students, to give them greater confidence that the choices they make are the right ones for them. The performance of universities has a number of domains, including but not limited to: research, teaching, financial performance, student experience, the quality of learning outcomes and access and equity. Each of these domains has a specific mechanism or tool (sometimes more than one) designed to capture relevant information about performance in that domain. For example, the Excellence in Research for Australia (ERA) process captures information about research performance; access and equity outcomes are captured through student data collections that include markers for low-SES status; and TEQSA will be implementing teaching standards by which the performance of universities will be measured.

Similarly, the three performance indicators that are the subject of this paper are designed to capture information about how universities perform in the domains of student experience and the quality of learning outcomes. There are likely to be synergies and complementarities with other tools, for example, TEQSA’s teaching standards. They therefore should be seen as part of an overarching suite of performance measures and mechanisms that are designed to capture information across the most relevant domains of university performance, necessary for improving the information available to students as they seek to exercise the choices that are now open to them in the demand-driven system. It should be noted that the newly created MyUniversity website will be used for presenting information to students about performance across the various domains.

Performance Funding

In late 2009, the Department convened an Indicator Development Group comprised of experts in the higher education sector. The group assisted in the development of a draft indicator framework, outlined in the discussion paper, An Indicator Framework for Higher Education Performance Funding, which was released for consultation in December 2009. The paper proposed 11 possible performance indicators in four performance categories. 61 submissions from the sector were received in response to the discussion paper.

The Government considered the feedback received and refined the framework to include seven indicators in three performance categories: participation and social inclusion, student experience and the quality of learning and teaching outcomes.

The Government released draft Performance Funding Guidelines for discussion in October 2010. The draft Guidelines provided details on the proposed implementation of the Performance Funding arrangements. The Government received 44 responses to the draft guidelines.

In the 2011-12 Mid Year Economic and Fiscal Outlook (MYEFO) the Government announced that it would retain Reward Funding for universities that meet participation and social inclusion targets. The Government discontinued Reward Funding for student experience and quality of learning outcomes indicators in the context of the Government’s fiscal strategy and on the basis of feedback from the sector that there was no consensus on the issue of whether it is appropriate to use such indicators for Reward Funding.

Universities have acknowledged the need to develop a suite of enhanced performance measures for providing assurance that universities are delivering quality higher education services at a time of rapid expansion. The Government will focus on the development of student experience and quality of learning outcomes indicators for use in the MyUniversity website and to inform continuous improvement by universities. The Government has agreed that three performance measurement instruments will be developed over the duration of the first Compact period: a new University Experience Survey (UES), an Australian version of the Collegiate Learning Assessment (CLA) and a Review of the Australian Graduate Survey (AGS). The Government has removed the composite Teaching Quality Indicator (TQI) from the performance indicator framework since universities have made a reasonable case that university performance is best measured using output rather than input indicators.

Final Facilitation Funding and Reward Funding Guidelines and Administrative and Technical Guidelines will be released on the Department’s website in December 2011. These will provide an outline of the final Performance Indicator Framework and Performance Funding arrangements.

Advancing Quality in Higher Education

In the 2011-12 Budget, the Government released details of its Advancing Quality in Higher Education (AQHE) initiative designed to assure and strengthen the quality of teaching and learning in higher education. The announcement provided more information on the new performance measurement instruments being developed and the available funding for developing the instruments. The consultation processes for the initiative were also outlined including the establishment of an AQHE Reference Group to advise on the development and cohesiveness of the performance measurement instruments. The AQHE Reference Group will also assist in the development of discussion papers for each of the instruments. Roundtable discussions with universities, business and students will also be held later in 2012.

Purpose of this paper

This paper discusses key issues in the design of the performance measurement instruments, assesses their fitness for purpose and their ability to operate together in a coherent way to obtain a comprehensive view of the student’s undergraduate university experience and learning outcomes. This paper does not describe in detail how the measurement instruments will be implemented - these issues will be outlined in separate discussion papers on each of the instruments.

The second section of this discussion paper considers some principles to guide the development of new performance measurement instruments. It also proposes that a framework of the student life cycle be used to situate the development of new performance measurement instruments. The third section briefly discusses existing survey instruments. The development of new performance measurement instruments for use in performance reporting and for other purposes is discussed in the fourth section. The fifth section considers key issues that impact on the coherence and balance of performance measures. The final section outlines a proposed implementation strategy for the new performance measures. ...

Issues

This section discusses key issues that arise from consideration of existing and new performance measurement instruments. This includes the administration of performance measurement instruments, student selection, central sampling of students, uses and the intersection of existing and new instruments. Discussion of these key issues facilitates an assessment of the fitness for purpose of performance measurement instruments and their ability to operate together in a coherent way to obtain as comprehensive view as possible about the student’s undergraduate university experience.

Administration of new instruments

Student surveys tend to be conducted in Australian universities using one of two broad deployment approaches (ACER, 2011, p.14), specifically:

• an independent (or centralised) deployment, in which most if not all survey activities are conducted by an independent agency; or

• a devolved deployment (or decentralised), in which institutions and a coordinating agency collaborate on survey operations.

An independent deployment approach involves participating universities providing the independent agency with a list of all students in the target sample at their institution, including student’s contact details. After receiving institutions’ population lists, the independent agency would identify the target population, which could be either a census or sample of students, and invite students to participate in the survey. Responses would then be returned directly to the independent agency for analysis.

A devolved approach involves participating universities supplying the independent agency with a de-identified student list that excludes student contact details. A sample of students would be drawn, online survey links would be allocated to student records, and this list would be sent back to universities who would then merge in student contact details. Under a devolved approach, universities manage the deployment of the survey by sending invitations to sampled students and following up with non-respondents. Responses are provided directly to the independent agency for analysis.

Both deployment approaches have benefits and limitations. A devolved approach has benefits in that it can accommodate the needs and circumstances of a diverse array of universities. On the other hand, the fact that universities are primarily responsible for collecting data about their own performance can be seen as a conflict of interest, leading to perceptions that universities may ‘game’ the system. Given the stakes and uses to which the data collected from the new instruments will be used, on balance, an independent approach is favoured since this will promote validity, consistency and efficiency.

A key issue for any independently deployed survey is privacy and the responsibility of universities (and the Department) to preserve the confidentiality of student information they hold. Universities may be required to amend their agreements with students to permit disclosure of personal information to third parties for the purposes of conducting surveys. Providing privacy laws are satisfied in the development of an instrument, this approach has to date received broad support from the higher education sector, as measured through consultation in the development of the UES.

While the deployment approach is a significant consideration in terms of administration of a new survey instrument, there are other issues which should be considered. These include, but are not limited to, the administration method (online, telephone, paper-based), context and timing. These issues will be considered in the context of individual instruments.

Questions for Discussion

• What concerns arise from an independent deployment method?
  

• What are the obstacles for universities in providing student details (such as email address, first name and phone numbers) to an independent third party?

• Would universities agree to change their privacy agreements with their students to permit disclosure of personal information to third parties for the purposes of undertaking surveys?
  

• What are the other important issues associated with administration of survey instruments?
  

Student selection

The selection of students for the new performance measurement surveys is an issue which has raised a number of concerns. The burden of the existing range of survey instruments on both students and university resources is foremost in the minds of universities and therefore a balance needs to be struck between the need for the collection of new data for the purposes of performance reporting and to assure quality and the requirements on students and universities.

The surveys could be run as a census of all students in scope or by administering the survey to a sample of the students in scope. Deciding between a census and a sample is a complex process that necessarily takes into account many technical, practical and contextual factors.

Two major issues are non-response biases and general confidence in the precision of results, particularly at the sub-institutional level. The advantage of a sample survey approach is that a structured sample can be constructed that deliberately focuses on target groups for which it is necessary to generate meaningful results (for example at the course level or for particular demographic groups) and this may assist in overcoming non-response biases. A sample survey approach would require a relatively sophisticated sampling frame to give adequate coverage across fields of education and demographic characteristics. This process would be simplified if the Higher Education Information Management System (HEIMS) database could be used to construct the sample frame, given that it already records detailed information on student characteristics. Standard techniques to measure the precision of sample survey results could be systematically applied across all results, for example, calculating confidence intervals or standard errors.

On the other hand, given the small student populations sometimes under consideration (for example, courses where only a small number of students are enrolled at a particular institution), sample sizes needed to provide confidence in survey results would approach the total population. The intention to publish data from the new performance measurement instruments on the MyUniversity website disaggregated to subject level may influence the decision on whether to conduct a census or survey since a sufficiently large number of responses will be required to ensure data are suitably robust and reliable. In this case, it may be preferable to continue on a ‘census’ basis where the whole population is approached to participate.

Other issues for consideration in deciding between a census or survey approach include:

• Support by participating institutions;

• The size and characteristics of the population;

• Providing students with opportunities for feedback;

• Relationship with other data collections, in particular other student surveys;

• Analytical and reporting goals, in particular sub-group breakdowns;

• Anticipated response rates and data yield;

• Consistency and transparency across institutions;

• Cost/efficiency of data collection processes; and

• The availability of supplementary data for weighting and verification.

The method of student selection may vary between instruments, and regardless of whether a census or sample approach is used, proper statistical procedures will be used to evaluate the quality and level of response in the long term.

Questions for Discussion

• What are key considerations in choosing between a sample or census approach to collection of performance data?
  

Central sampling of students

As discussed above, an important issue regarding the introduction of new surveys within the higher education sector is the perceived burden on university resources and the students who are required to participate.

One method which has been proposed to assist in the reduction of this burden is the possibility of using DEEWR HEIMS (Higher Education Information Management System) data to better control student sampling and also to use stored student demographic level data to pre-populate survey questions where appropriate.

Using the HEIMS data in this way could potentially improve random sampling and avoid oversampling of students invited to participate in surveys, while also reducing the number of questions students are required to answer per survey through the ability to pre-fill and skip questions where data is already available. In addition, by having the Department involved at this level in the survey process, this could improve perceptions of the overall integrity of surveys through making clear that samples are independently constructed.

Note there are restrictions regarding the use of HEIMS data in the Higher Education Support Act. The Department, therefore, is investigating options for the use of this data in the context of individual performance measurement instruments and also as a suite of instruments.

Questions for Discussion

• What are the advantages and disadvantages of central sampling of students?
  

Uses of data

The collection of data from the new performance measurement instruments will be of major benefit to universities for continuous improvement and to assure the quality of teaching and learning in the sector. The Government has also indicated that, subject to their successful trial and implementation, it is intended that data from the new instruments will be published on the MyUniversity website.

While a number of uses are proposed for the new data collections, other uses may be discovered throughout the development of the new instruments. It will be important, therefore, to consider potential future uses of the data while being aware of the potential for misuse of the data.

By utilising the principles referred to in section 2.1 of this paper in the development of the instruments, the data available should be relevant, reliable, auditable, transparent and timely. These principles will also provide guidance as to the appropriate uses of the data. Further, the range of consultations being undertaken in the development of the new instruments will include roundtables where stakeholders will be able to discuss how the results from the new instruments can be used while raising any concerns regarding the future use of the data. In addition, it may be appropriate to establish codes of practice that guide appropriate uses and interpretation of performance information.

Continuous improvement

It will be important that the data collected from the performance measurement instruments is useful for universities, so that they can implement processes and policies for continuous improvement and maintain a high level of quality of teaching and learning.

To ensure the performance measurement instruments collect data that is useful and relevant to universities, the instruments are being developed with significant sector consultation. Stakeholders will be invited to provide feedback throughout the development of the new instruments to allow these to take account of their data and measurement needs.

Using the student life cycle model, consideration needs to be given to what information can and should be collected from the new instruments given their implementation at different stages of the life cycle, and how this information will assist to assure the quality of teaching and learning in Australian universities.

MyUniversity

It is expected that in future releases, the MyUniversity website may include performance data from the new performance measurement instruments. This would allow prospective students additional information with which they can assess the performance and quality of different institutions in the three performance categories. How the data will presented on the MyUniversity website will be a consideration once the instruments have been tested and the level of data analysis is known.

Information regarding the use of performance data on the MyUniversity website will be made available throughout the development process for the instruments and the website itself.

Another issue that arises in consideration of the MyUniversity website is the level of reporting. A major purpose of the website is to inform student choice about courses and subjects. In this environment, more detailed reporting is likely to be desired, for example, at the field of education level. There is likely to be a trade off between collection and reporting of data from the new performance measurement instruments at a finer level of disaggregation and adding to the complexity and burden of reporting.

Questions for Discussion

• What are appropriate uses of the data collected from the new performance measurement instruments?
  

Intersection of existing and new instruments

The development of the University Experience Survey (UES) has raised the issue of whether the new instruments should be focused instruments for the purposes of performance reporting, or whether they could potentially be expanded to replace existing surveys and institution/course specific questionnaires. For example, what is the potential overlap between the newly developed University Experience Survey and the Course Experience Questionnaire in measuring student experience?

This will be a consideration in the development of all new instruments to ensure there is balance between the additional burden on both students and universities of the new instruments and ensuring they are able to capture targeted and purposeful information. Further, there needs to be consideration of the data needs of individual universities and how these differ across the sector.

A key issue in considering the overlap between the new instruments and existing survey instruments is the uses to which performance measurement data will be put as discussed above. It is expected students will benefit from the availability of new data via the MyUniversity website. The new performance measurement instruments will potentially be used to enhance continuous improvement processes within universities. There is also the potential for international benchmarking.

A major dilemma is the need for broad level national data and the requirement for more detailed data to suit universities’ diverse needs and missions. Satisfying both of these goals raises the important issue of costs and resource burden. One suggested solution for the UES is that there could be a core set of items which would be asked of students at all universities, and an optional set of non-core items which universities could select to suit their individual requirements.

Ultimately, universities will decide in which instruments they participate, and this will hinge on a range of factors not limited to the type of data collected. By considering, however, the range of existing surveys, what data is most useful to universities, and what additional data universities would like to collect from the new performance measurement instruments, the development of the new instruments has the potential to make this decision by universities considerably easier.

Questions for Discussion

• Are there other issues that arise when considering the overlap of existing and new instruments?
  

...

From: Development of Performance Measurement Instruments in Higher Education, Department of Education, Employment and Workplace Relation, 9 December 2011

Review of the Australian Graduate Survey

Discussion Paper

December 2011

Table of Contents

1.Introduction 3

1.1.Policy context 3

1.2.Consultation 3

2.Principles and the student life cycle framework 4

2.1.Principles 4

2.2.Student life cycle framework 4

3.Strategic position, role and purpose of the AGS 5

3.1.Overview of the AGS 5

3.2.Role and purpose of the CEQ 5

3.3.Role and purpose of the GDS 6

3.4.Future strategic position of the AGS 6

4.Administration issues 7

4.1.Administrative model 7

4.2.Timeliness 7

4.3.Funding 7

5.Survey methodology and data quality issues 8

5.1.Methodology and standardisation 8

5.2.Data quality 8

6.Aspects of student experience 10

7.Next steps 11

Appendix 1 – References 12

Appendix 2 – How to make a submission 13

Appendix 3 – Current AGS survey instrument 15

...

Consultation

The Australian Graduate Survey (AGS) is a national survey of newly qualified higher education graduates, conducted annually by Graduate Careers Australia (GCA). A strengthened AGS is part of the suite of performance measurement instruments that were announced as part of the AQHE initiative. The Department of Education, Employment and Workplace Relations (DEEWR) is working with GCA and the higher education sector to review the AGS. The review is examining the strategic position of the survey, and aims to improve the survey content, data collection methods and timeliness of reporting. The review is also considering how to better capture aspects of student experience for external, Indigenous, international and low socio-economic status students.

Consultation for the AQHE initiative includes the establishment of an AQHE Reference Group to advise on the cohesiveness of the three instruments, and the development of an overarching discussion paper, Development of Performance Measurement Instruments in Higher Education. In addition, the AQHE Reference Group will assist in the development of discussion papers on each of the instruments. Consultations and roundtable discussions with universities, business and students will also be held later in 2011 and in 2012.

The Department has prepared this discussion paper based on consultation with and advice from the AQHE Reference Group. The paper raises issues and options for the future of the AGS, with the aim of canvassing views from universities and other stakeholders in the sector. Information on how to contribute to the process can be found below. ...

Future strategic position of the AGS

The Government announced as part of the Advancing Quality in Higher Education initiative the development of a suite of performance indicators for the higher education sector (a summary of the new indicators can be found in the Development of Performance Measurement Instruments in Higher Education discussion paper). These indicators will provide greatly enhanced information on university performance. At the same time, the sector is moving towards a student centred funding model. To assist students in making informed decisions about their tertiary education, it is intended that the MyUniversity website will from 2013 include detailed results from the new performance indicators. Higher education sector stakeholders, including institutions, students and Government, will therefore be responding dynamically to a greater range of information sources than has previously been available. Importantly in this respect, institutional level and institution by field of education level data will be made public. While this does not accord with current AGS practice, it is consistent with approaches to publishing performance information previously undertaken by the Department.¹

This changed environment presents a major challenge to the ongoing relevance and strategic position of the AGS. From being the prime source of nationally benchmarked data on university performance, the AGS will become one of several available data sources. In this context, the ongoing role and value of the AGS needs to be clearly articulated. The AGS may need to be modified to enable the survey to establish a coherent place among the range of new indicators, and to ensure it continues to meet the evolving needs of higher education sector stakeholders.

Given the increasing number of surveys in which university students are being asked to participate, and for which universities are being asked to provide administrative support, the additional value offered by the AGS needs to be clearly articulated. One option to reduce cost and respondent burden would be to move from the current census basis of the AGS, where all eligible students are invited to participate, to a survey sample. This question also has ramifications for data quality, as discussed below.

Consideration should also be given as to whether the CEQ should move to surveying students, rather than graduates, in line with the other performance indicators being developed. The CEQ was originally developed and tested for use with undergraduate students in the United Kingdom. In Australia, however, it has always been administered to graduates, which may lead respondents to focus on overall course experience (as intended), rather than specific subjects or instructors. Surveying graduates has also allowed the CEQ to be administered simultaneously with the GDS. Conceptually, however, the CEQ measures satisfaction across the whole of the student lifecycle, and there is no inherent reason why this need take place after graduation.

The most notable challenge to the ongoing relevance of the CEQ comes from the new University Experience Survey (UES). The UES will gauge student attitudes towards a number of aspects of their university course, initially at the end of their first year and potentially in their final year of study. The UES will measure aspects of student’s university experience associated with high level learning outcomes such as teaching and support, student engagement and educational development. While not identical to the information garnered by the CEQ, the UES will provide an alternative measure of student satisfaction and course experience perceptions across the student lifecycle. Consideration needs to be given to the value of continuing the CEQ as an additional survey instrument.

Information provided by the GDS will not be replicated by any of the new performance indicators. By its nature, the GDS is a measure of a university’s contribution to skill formation in relation to labour market outcomes and can only be administered at the end of the student lifecycle. Information on graduate outcomes will continue to be of value to the sector. Nonetheless, consideration should be given as to whether the GDS as currently configured is appropriate for the needs of the sector in the future.

Questions for Discussion

• Is joint administration of the GDS and CEQ under the AGS still appropriate?
  

• Will the GDS and CEQ adequately meet future needs for information in the student driven environment?
  

• Should the basis of the AGS be modified to improve fit with other indicators or to reduce student burden? Would a survey sample be a more appropriate option? What are the implications for the development of the UES for the CEQ?
  

...

Funding

The AGS is primarily funded by direct grant from DEEWR to GCA, made under the Higher Education Support Act. In 2011, this grant was around $660,000. Funding is also sourced from Universities Australia and from subscriptions paid by individual universities. In addition, universities provide substantial in kind funding by administering the survey instrument. In recent years, GCA has incurred financial losses in administering the AGS, and additional funding would likely need to be found if current administrative arrangements are to continue. ...

Questions for Discussion

• Is the current partially decentralised mode of delivering the AGS still appropriate?
  

• How can the timeliness of AGS reporting be improved?
  

• Are current funding arrangements for the AGS appropriate? What alternative funding arrangements should be considered?
  

...

Data quality

Consideration should also be given to the broader data quality issues. Conceptually, the AGS currently operates on a census basis, in that all eligible graduates are invited to respond. GCA procedures mandate that institutions achieve at least a 50 per cent overall response rate. For the 2010 AGS the national response rate was 53 per cent for the CEQ and 57 per cent for the GDS.¹ This is a high response rate compared with other surveys of university students, but is still low enough to raise questions as to data reliability. The two main issues are non-response biases and general confidence in the precision of results, particularly at the sub-institutional level. ...

Questions for Discussion

• Will AGS data continue to be reliable enough to meet the needs of the sector in the future? How can data reliability best be improved?
  

• Would moving the AGS to a centralised administrative model improve confidence in results?
  

• Would moving the AGS to a sample survey basis improve data quality?
  

Aspects of student experience

...

Questions for Discussion

• Does the AGS adequately measure the diversity of the graduate population and how might it be strengthened in this regard?
  

1 Graduate Careers Australia, 2011(a), p. 2; Graduate Careers Australia, 2011(b), p. 3.

2 Graduate Careers Australia, 2006, pp. 61-92.

1 Department of Education, Training and Youth Affairs (1998); Department of Education, Science and Training (2001).

From: Review of the Australian Graduate Survey, Department of Education, Employment and Workplace Relation, 9 December 2011

Assessment of Generic Skills

Discussion Paper

December 2011

Table of Contents

1.Introduction 3

2.Policy context 3

2.Policy context 3

3.Consultation 3

3.Consultation 3

4.Principles and the student life cycle framework 4

5.Principles 4

5.Principles 4

6.Student life cycle framework 4

6.Student life cycle framework 4

7.Purpose 5

8.Direct Assessment of learning outcomes 5

8.Direct Assessment of learning outcomes 5

9.Uses 5

9.Uses 5

10.AHELO – progress report 7

11.Outline 7

11.Outline 7

12.Generic Skills 7

12.Generic Skills 7

13.Progress 7

13.Progress 7

14.Australia’s participation 7

14.Australia’s participation 7

15.Issues 9

16.Quality assurance framework 9

16.Quality assurance framework 9

17.Discipline-specific assessments 9

17.Discipline-specific assessments 9

18.Measurement 10

18.Measurement 10

19.Participation 12

19.Participation 12

20.Next steps 13

...

Purpose

Direct Assessment of learning outcomes

Direct assessment of learning outcomes represents the ‘holy grail’ of educational measurement. Objective data on student outcomes provides direct evidence that higher education is meeting economic, social and community needs. Knowledge of what students have learned and achieved and that they have attained the expected outcomes of their degrees provides assurance about the quality of higher education.

External assessment of student’s cognitive learning outcomes, at least in the higher education environment, is rare and to date has been difficult to achieve. In the schools sector, the Programme for International Student Assessment (PISA) is probably the most notable example on an international scale of the measurement of reading, mathematical and scientific literacy outcomes. Alternative measures of learning outcomes in the higher education sector, such as employment and further study, are thought to be problematic because they may be confounded by factors such as the influence of field of education and differences in state/regional labour markets. In the absence of robust direct assessment of outcomes, reliance is frequently placed instead on student self-reports, for example, as measured through the Course Experience Questionnaire (CEQ) generic skills scale which records student perceptions of their achievement of generic skills.

The Bradley Review of Higher Education argued that, “Australia must enhance its capacity to demonstrate outcomes and standards in higher education if it is to remain internationally competitive and implement a demand-driven funding model.” (DEEWR, 2008, p. 128). As part of the new quality assurance framework, Recommendation 23 of the Bradley Review of Higher Education proposed:

“That the Australian Government commission and appropriately fund work on ...........

• a set of indicators and instruments to directly assess and compare learning outcomes; and

• a set of formal statements of academic standards by discipline along with processes for applying those standards.”

Uses

Direct assessment of learning outcomes has many uses and benefits including providing assurance about the quality of higher education, encouraging continuous improvement among universities, meeting employer needs for more skilled graduates and informing student choice.

Quality assurance

External assessment and reporting of the attainment of generic skills provides assurance about the quality of graduates from the higher education system. With significant public and student investment in higher education, the community is entitled to understand that graduates have acquired the skills expected of them when they have completed their degree. The performance indicator framework proposes that the Collegiate Learning Assessment, or some variant of the instrument, be developed for use as an indicator of the acquisition of generic skills. A key principle in the design of performance measures is that they be ‘fit for purpose’, and it is intended that an instrument assessing generic skills be designed that is capable of being used for performance reporting.

Continuous improvement

Assessment of learning outcomes offers the prospect of a virtuous circle whereby assessment and reporting inform improved teaching and learning, in turn leading to improved assessment and reporting of learning outcomes. For example, the Collegiate Learning Assessment offers assessment tools and additionally resources to support improvement of curriculum and pedagogy to promote the development of generic skills. The Council for Aid to Education (CAE) in administering the Collegiate Learning Assessment also conducts professional development activities that train staff in in the process of creating better teaching tools (and classroom-level measurement tools) aimed at enhancing student acquisition of generic skills.

Arguably, the process of teaching and learning generic skills is more effective if undertaken at discipline rather than university level. The issue of the appropriateness of university and/or discipline-specific assessments will be addressed in more detail below. In part, the focus on discipline follows the recommendation of the Bradley Review of Higher Education of the need for more work to be undertaken on academic standards by discipline and processes for applying those standards.

Employer needs

To be successful in the workplace, graduates must acquire generic skills that enable them to fully utilise their discipline-specific knowledge and technical capabilities. As suggested by the Business Council of Australia:

“The challenges involved in adapting to new and changing workplaces also require effective generic skills. Generic skills including communication, teamwork, problem solving, critical thinking, technology and organisational skills have become increasingly important in all workplaces.” (BCA, 2011, p.8)

Reliable empirical studies of employer needs and satisfaction with graduates are few and far between which is both unfortunate and somewhat surprising given the strongly vocational orientation of much of Australian higher education. An earlier study of employers found that the skills in which new graduates appear most deficient are the generic skills of problem solving, oral business communication skills and interpersonal skills with other staff (DETYA, 2000, p.22). The measure of skill deficiencies used in this study was the gap between employer ratings of the importance of skills and their ratings of graduate abilities in these skills. A study conducted by the University of South Australia gave broadly similar findings about employer demand for graduate skills and capabilities (UniSA, 2008).

Informing student choice

Greater transparency will inform student choice in a demand-driven funding model of higher education. The Collegiate Learning Assessment is part of a suite of performance measurement instruments designed to improve transparency in university performance. Subject to the successful development and trial of the Collegiate Learning Assessment, it is intended that universities results on this performance indicator will be published on the MyUniversity website from 2013 onwards.

Another issue that arises in consideration of the MyUniversity website is the level of reporting. A major purpose of the website is to inform student choice about courses and subjects. In this environment, it would be appropriate to report information at the discipline/field of education level as well as the institution level. This is another factor that impacts on the development of an appropriate assessment of generic skills. The issue of the assessment of discipline-specific generic skills is discussed in more detail below.

Questions for Discussion

• Are there other uses of the assessment of generic skills?
  
  

...

Issues

...

Questions for Discussion

• Which criteria should guide the inclusion of discipline specific assessments in the development of a broader assessment of generic skills?
  

• Are there other criteria, not listed above, which need to be considered?
  

...

Questions for Discussion

• What factors should guide the design of performance measurement instruments to assess generic skills?
  

...

Questions for Discussion

• Is value-add an appropriate measure of generic skills?
  

• Is it necessary to adjust measures of generic skills for entry intake and how should this be done?

• Are there other more appropriate measures of generic skills?

...

Questions for Discussion

• What would be an appropriate measure of generic skills for reporting university performance?

...

Questions for Discussion

• What level of student participation is desirable and for what purposes?
  

• What design features or incentives would encourage student participation?
  

...

From: Assessment of Generic Skills, Department of Education, Employment and Workplace Relation, 9 December 2011