As part of my learning about tertiary teaching I am looking at the process used for the selection of thesis examiners and the examination process for doctoral theses. After some readings (appended) I am to ask experienced academics how they select examiners for their students and how they go about examining others. I would welcome any comments on the topic.
Readings
Course Notes
The Research Supervision Notes (customised by a consortium of ANU, Oxford and McGill Universities) divide the topic into four steps:Preparing Students
The notes suggest that Australian examiners take a formative approach, to help the candidate, rather than a simple pass/fail summative approach. However, this seems to me to be at odds with the official rules for examination of a thesis (discussed later) which are strictly summative in nature. If a formative approach as intended, then the examiner would be brought in well before the end of the thesis preparation and would not be barred from communicating directly with the candidate.Selecting Examiners
The notes suggest the supervisor and candidate discuss potential examiners. However, university rules prohibit this: the candidate is not to know who the examiners are. The notes also suggest the examiners are potential sponsors for post-doctoral positions. This would also appear to be contrary to examination rules and expose all those involved to charges of misconduct.Examination Process
The notes suggest examiners are motivated by a wish to maintain standards, an obligation to academia in general and reciprocal oblation to have their own students examined. The last of these is a source of potential corruption, with examiners being tempted to pass students in return for having their students passed.It is curious that in the motivations of examiners there is no mention of it simply being part of their job. Such an incentive could raise the quality of examination as then the examiner would not be seen as doing a favour for a colleague, but instead doing a professional job and so could have formal requirements set which they are required to meet. In particular, if examiners do not complete an examination on time, then payment, or promotion could be withheld.
Following Examination
There appears to be no systematic process within or across universities to monitor thesis examiners. Given the importance of the examination, it would seem reasonable to at least keep a record of the mark given by each examiner for each thesis so as to detect trends, including official corruption. It would seem reasonable to have the marks of all examiners published, along with statistical trends for institutions and disciplines.Higher Degree Research Guide
The ANU Higher Degree Research Guide also provides an overview of the formal process. This sounds a somewhat dated, pre-internet process with the thesis "... despatched to examiners by the quickest and most secure means possible..." and examiners asked to provide reports "... within 2 months...".Recent discussion with collogues indicates difficulty in having examiners respond by the time limit, which is not surprising give the lack of structure to the process.The procedure discourages examiners from submitting a preliminary report. As a result the examiner would be discouraged from starting work on such a large monolithic task.
It would seem reasonable to update the process to a similar one to that now used for submission of papers to conferences and journals. That is the examiner would be provided with access to the thesis online and would enter their comments there. As a thesis is large, the examiner should be asked to enter comments for each major section separately, with a time limit for each. The work-flow system would automatically track the progress of each thesis examination and remind the examiner if they are behind schedule.
The university procedures allow for an oral examination, but his seems to be only if the examiner requests it.
The university procedures require that the examiners names are not revealed to the candidate or other examiners. Any communication is through the ANU Higher Degrees Research Officer. This process might be able to be updated now with a computer mediated system allowing for anonymous communication between the examiners and candidate (identified as for example "Examiner 1", "Examiner 2", "Candidate"). This would allow for a form of text based "Oral" examination. The Rules allow for "correspondence" between the examiners and the candidate for an oral examination.
There appears to be an inconsistency between university rules and practice, with the rules requiring an oral examination, unless there is an exemption, but the practice being to routinely exempt the oral. This appears to be done for administrative convenience, to save the cost and complexity of arranging the oral examination, given the candidate and examiners are likely to be spread around the world. The rules allow for telephone or video conference. I suggest a viable alternative would be a text based store and forward "oral" could be incorporated routinely into the process.
Examiners provide a written report, as well as "Quality of Work" and "Recommendation of Examiner" forms. In addition they can recommend the thesis for a University prize. The possible recommendations for the thesis are similar to those for a paper review:
- admitted to the degree;
- admitted to the degree, subject to amendments;
- submit a revised thesis for re-examination;
- failed: Failed PHD students may be offered a Masters.
Anonymous examiner's reports are provided to the candidate. Successful candidates are required to supply the university library with paper copies of the thesis. These are steps in the process which could be simplified with an online system (there is already a digital thesis system in place, but its use is voluntary).
One way this process might be simplified is where the candidate has has had refereed papers published and delivered conference or seminar presentations. Where these are judged to be at a suitable level, they could be considered as a suitable alternative to much of the written and oral examination. It is likely that examiners do this informally in practice, but it could be included in the rules.
Assessment Pedagogy for Thesis Examination
One striking feature of thesis examination is the limited level of resources devoted to it, compared to coursework programs where assessment is typically 40% of the staff cost.The Universities Australia recommended fee for examining Higher Doctorates is $425, where the thesis is more that 66% of the assessment. Assuming the thesis is 66% of the doctorate and there are three examiners, that would put the total cost of the assessment of a doctorate at $1,932. The international fees for a typical three year Doctor of Philosophy are $90,288. Even allowing for a 50% profit margin on the course and 50% staff cost, this indicates that the assessment is only 8% of the cost of the course.
Some allowance would have to be made for the supervision component of the degree, which provides a form of progressive assessment, but even so there would appear to be a lack of assessment in a research doctorate, as compared to coursework.
It may be argued that the fee for examining a higher degree is not intended to cover the true cost and academic assessors undertake this work for other than financial remuneration. However, this introduces the potential for corruption, with examiners passing a student in the expectation that the favour will be returned. This could be made worse due to the lack of a clear assessment process for a thesis, as compared with coursework. There appear to be no clear learning objectives and no assessment rubric. It might be argued that these are not possible as each thesis is unique, however in that case a unique set of criteria for marking should be developed for each thesis and agreed with the candidate in advance.
Other Readings
Mullins and Kiley (2002) looked at the experience of thirty examiners who had examined at least five theses each. They note that while there is a an increase in formal courses undertaken by research students, these courses is not considered in the final assessment, only the thesis.Mullins and Kiley (2002) states: "The research project on which this article is bored set out to answer two questions" (presumably "bored" is meant to be "based"):
- Is it possible to define one aspect of the pedagogy for postgraduate learning, i.e. assessment?
- What advice might there be for students, supervisors, examiners and institutions related to the examination of postgraduate research theses?
For the second point on advice on examinations, it is curious that the authors confine themselves to the examination of theses, rather than considering other ways to assess postgraduate research work. It would be very unusual to have a coursework program where the assessment consisted solely of one written examination at the end of three years, as applies to a thesis.
Mullins and Kiley (2002) provide a number of observations about what examiners find important in a thesis and this is useful advice for candidates. However, these do not show any significant differences from coursework experience. But it is useful to see that pedagogy developed for coursework can be applied to research supervision.
Tinkler and Jackson (2000) argue that PhD examination is less transparent than other levels of British higher education. They attribute much of the lack of transparency to the use of the "viva" (oral examination), wit only one "new" university where the candidate gets any indication of what they may be asked. The concern over the viva is interesting in the Australia context as some Australian academics see the use of an original examination as a way to improve the quality of assessment, but it may cause more problems than it solves.
The authors looked at processes in 20 UK universities (Scotland, England and Wales). They found common agreement as to what a PhD was and what the assessment criteria were. The results appear consistent with Mullins and Kiley's later work in Australia (2002).
Unusual Source of Input
On a flight back from Sydney on 8 October there was a tap on my shoulder and I was handed a note addressed to "Tom seat 16c". This was handwritten on an airline sick bag and was from my tutor (sitting several seats away), suggesting that the research supervisor provides the "real" assessment of the candidate and the external examiners are a form of conformation of this. That would seem a useful and workable approach, but university rules would need to be recast to say that is actually what is intended.Guidelines could be introduced to formalise the formative feedback and assessment provided to the student, mush as is done in coursework. Also the proportion of the staff time allocated to this could be accounted for (and most likely would be around the 40% for coursework). Supervisors could then also undertake the same courses which coursework lecturers undertake to learn how to design and deliver such assessment.
This might be hard for some research supervisors to accept as it would reduce the proportion of the assessment for the thesis and in particular the proportion for "originality" in the assessment. A workable arrangement might be that the thesis is 20% of the overall assessment (and 5% for originality), but that the student has to "pass" both these to pass overall.
References
Mullins, G., & Kiley, M. (2002). 'It's a PhD, not a Nobel Prize': How experienced examiners assess research theses. Studies in Higher Education, 27(4), 369-386. doi: 10.1080/0307507022000011507 Retrieved from http://www.tandfonline.com/doi/abs/10.1080/0307507022000011507Tinkler, P., & Jackson, C. (2000). Examining the Doctorate: Institutional policy and the PhD examination process in Britain. Studies in Higher Education, 25(2), 167-180. doi: 10.1080/713696136 Retrieved from http://www.tandfonline.com/doi/abs/10.1080/713696136
No comments:
Post a Comment