Evaluating University Research Supervisors

Recently I have been looking at how to evaluate the quality of the supervision provided to research students at university. Some questions are:

1. Quality Assurance measures at the institutional level: What are these measures?
2. Individual practice: Supervision is a teach activity, so how are the views of the individual supervisors assessed? Are the also assessed as a group?

Readings

The examination of evaluation of supervision in Australian universities by Aspland, Edwards, O'Leary,  & Ryan (1999) is still relevant to today's conditions. Australian higher educational institutions were consolidated into larger universities with campuses which were previously independent teacher's colleges. Staff from these vocationally orientated institutions were under pressure to undertake research and supervise research students.
Now universities are under regulatory pressure to meet new new national standards and financial pressure to have higher degree students complete faster and more reliably. Student dissatisfaction with supervision, delays with feedback would seem as relevant in 2012 as they were in 1999.

 Aspland, Edwards, O'Leary,  & Ryan (1999) cite literature on the value of feedback on postgraduate supervisory practices and note that such feedback was not a routine part of postgraduate programs, in contrast to standard undergraduate coursework feedback surveys. Those postgraduate evaluations which were conducted tended to be for clinical work, rather than research. This would seem reasonable, as professional programs are attempting to produce graduates with a standard set of skills, much like an undergraduate coursework degree, but at a higher level of skill.

To investigate the topic Aspland, Edwards, O'Leary,  & Ryan (1999) surveyed Australian universities and carried out focus groups. These groups seem to have come up with the obvious: supervisors and their students need to agree what each will do, early in the student's program. The authors then produced a booklet containing guidelines for discussions between supervisor and student, plus evaluation tools to rate progress during the student's program (the tools appear to have been paper based). The authors claim that the pilot program with 75 students and supervisors contributed to the learning environment.

One curious aspect of this research, and much in this field, is the lack of discussion of the supervisor's formal qualifications and training in the field of research supervision and teaching more generally. If one booklet can make a significant improvement in the quality of research supervision, it raises the question as to what training and formal qualifications the supervisors had previously.

The paper by Boud and Costley (2007) does not appear to be on the topic of research supervision, as it instead talks about "project advising". They argue for a new relationship between the student and staff, derived from work-based learning (WBL) as applied in professional programs. The paper goes on to discuss short undergraduate projects and the value of having the student take on more of the management of these. I had assumed that the paper would go on to discuss how the techniques for these short projects could be applied to higher degree research programs, but the paper just seems to end after discussing short projects.

In my own supervisory practice I have found that short, one semester projects by undergraduate, honors and Masters students have been very successful. Examples are "Semantic Web for Museums" and "Evaluating Emergency Management Websites". The process for these involved negotiating refinements to a topic the student has selected from a list, setting deadlines for deliverables, having the student do a short literature search or user requirements (for software) the research (or software design), writeup and a seminar presentation. The practical and current topics provided to the students keep their interest during the projects. Essentially the same approach should work for longer masters and doctoral projects.

Being a supervisor

The "Being a supervisor" notes shared between ANU, Oxford and McGill Universities, provides a section on "Evaluation", which lists three kinds:

1. formative
2. summative
3. quality assurance
   

The use of these terms for research supervision seems to differ from coursework pedagogy, where formative and summative refer to assessment of the student, whereas quality assurance is used to refer to evaluation of the course. It is not clear from this document how all three terms are used for evaluation of research supervision.

Formative Evaluation

The section titled "Formative evaluation & reflective practice" unfortunately does not actually use the word "Formative" in the body of the document and seems to confuse reflective practice, which is something undertaken by an individual and mediated discussion between student and supervisor. It may be that the author, in summarizing concepts from source documents has missed some key points. As it is, it was not clear what the author had in mind. Unfortunately the section on summative evaluation was no clearer.

Institutional quality assurance

The Institutional quality assurance (QA) section describes QA in terms of evaluation of:

"... whole of Faculty, or whole of institution performance on standardized (often national) survey instruments.  ..."

This analysis could benefit from a more rigorous definition of quality assurance (QA), contrasting it with quality control (QC). QA is concerned with improving the production process, whereas QC is concerned with individual products. As the section fails to use these terms correctly, it is difficult to understand what is intended.

References

Aspland, T., Edwards, H., O'Leary, J., & Ryan, Y. (1999). Tracking new directions in the evaluation of postgraduate supervision. Innovative Higher Education, 24(2), 127.

Marsh, H. W. (2002). PhD students' evaluations of research supervision. The Journal of Higher Education, 73(3), 313-348.