Online Support for Evaluating Postgraduate Student Skills

My reading of research on postgraduate students indicates that "generic" skills are important to the student and to prospective employers. The issue then is how students gain these skills and how they are assessed. The "Student Practice Evaluation Form" (SPEF-R) shows one approach. SPEF-R is a paper based and later online system for supervisors to evaluate students on occupational therapy professional practice placements (Turpin, Fitzgerald, and Rodger, 2011) As well as supporting Formative and Summative assessment, the tool also supports Criterion-referenced assessment (CRA), as outlined in the a policy at the University of Queensland's Assessment Policy and Practices. With this each individual student is assessed against set criteria, rather than by relative performance against other students in a program. SPEF-R targets eight learning domains:

1. Professional Behaviour
2. Self Management Skills
3. Co-worker Communication
4. Communication Skills
5. Documentation
6. Information Gathering
7. Service Provision
8. Service Evaluation

These have much in common with the generic skills described for higher degree research students by other programs:

National Postdoctoral Association six core competencies:

1. Discipline-specific conceptual knowledge2. Research skill development
3. Communication skills4. Professionalism
5. Leadership and management skills
6. Responsible conduct of research

UK Vitae "Researcher Development Framework" four domains:

1. Knowledge and intellectual abilities: The knowledge, intellectual abilities and techniques to do research.
2. Personal effectiveness:           
   The personal qualities and approach to be an effective researcher.
3. Research governance and organisation: The knowledge of the standards, requirements and professionalism to do research.
4. Engagement, influence and impact: The knowledge and skills to work with others and ensure the wider impact of research.

SPEF-R  has an "item bank" for each learning objective, made up of "items" describing how the student demonstrates they have the skill. While it is not clear from the documentation, the learning objectives do not appear to be hard coded into the software and others could be entered. There are different sets ("streams") of items students learning "Direct Service Provision" or "Project Management / Consultancy". This should be able to be generalised to apply to a broader set of professions which need variations of the same skills.

Feedback is provided in SPEF-R  by way of a numerical ranking on a five point scale, and text based qualitative comment’s. The five point scale may be programmatic, as it could be difficult to differentiate many of the skills to this level of detail. It is not clear if a different scale can be substituted, such as the  three point scale I use for ICT Sustainability.

SPEF-R  has minimum requirements for each learning objective and the supervisor provides a summary statement.

Perhaps more important than the recording function which SPEF-R provides is the introduction to modern teaching and assessment practices provided, such as the Feedback Cycle.

Lessons from SPEF-R for Higher Education

The approach used for SPEF-R might be generalised for Higher Education using off-the-shelf learning management software, in particular a Learning Management System (LMS), such as Moodle and/or a SCORM package.

Moodle 2 supports advanced assessment techniques, including rubrics. SCORM packages allow for portable assessment, across different learning management systems, such as Moodle and Blackboard.

The student could be provided with an e-Portfolio (such as Mahara), loaded with a template of skills definitions, which they are then required to fill in with evidence. The evidence would include results of assessment from courses, imported from the LMS (Moodle 2) and examples of their practical work, plus any papers or thesis produced.

The same system could be used for coursework postgraduate students and professional degree students, as well as research Masters and PHD students. Different programs would use different discipline templates of skills, but the same core set of "generic" skills.

References

Turpin, M., Fitzgerald, C. and Rodger, S. (2011), Development of the Student Practice Evaluation Form Revised Edition Package. Australian Occupational Therapy Journal, 58: 67–73. doi: 10.1111/j.1440-1630.2010.00890.x Retrieved from http://onlinelibrary.wiley.com/doi/10.1111/j.1440-1630.2010.00890.x/full