Implementation of eLearning: key factors for success

Greetings from the Australian National University  Science Teaching and Learning Colloquium on Research Led Education, where  Alex Webb is discussing online interactive eLearning
resources. She is showing materials for learning radiological anatomy at University of Southampton. The effectiveness of this material for undergraduate medical students at the University of Southampton and postgraduate-entry medical students at the ANU. While university study, these courses also have a vocational aspect, in that medical students are required to meet national standards for their practice, as well as learn theory and undertake research.

Dr. Webb mentioned that accessibility to the material was important. However, I tried looking at the "

Radiological Anatomy" from University of Southampton and was not able to access it. The materials required an Adobe x-director plug-in, which does not appear to be available for Linux. The  screenshots  have tiny text, too small for me to read. The UK, where University of Southampton is located,  has similar disability discrimination legislation to Australia and the university has a web page on how to design and assess accessible materials. 

Surface Expectations For Courses

Reading Martineau & Hannum's book "Evaluating the Impact of Leadership Development: A Professional Guide (2004) I was starting to think I was a bit thick, or from the wrong planet. The preface, introduction and chapter 1 did not make much sense. I had to check the author's website to work out what the book was about: how to assess management training courses.

I suggest skipping the Preface, Introduction and Chapter 1, as they use of the sort of jargon which Lucy Kellaway criticizes in her BBC broadcasts. At Chapter 2 "Focusing the Evaluation", it appears a different author took over and the book gets much more readable.

The chapter starts out with useful points: ideally the evaluation of a course should be planned while the course is being designed and it has to fit into the organization schedule. So it is not a good idea to be taking teachers away from their desks to ask them questions, when they are busy marking. Similarly it is not a good idea to ask students about satisfaction with a course when they are cramming for the examination.

Martineau & Hannum suggest these steps:

1. Identify stakeholders
   
2. Define purpose
3. Determine resources
4. Establish the desired impact and timing (they had impact and timing as two steps but I see them as the one)
   
5. Surface expectations
6. Determine evaluation questions

The chapter then goes into details, providing worksheet and examples. The one part I had difficulty with was "Surface expectations". After reading the detailed description I was still not sure what these are. They appear to be assumed impacts: that is results of the course which are not formally expressed because they are considered so obvious they need not be stated. A web search for the term "surface expectations" suggests it refers to how smooth a concrete slab should be, but I don't think that is the meaning intended here. ;-)

The chapter then goes on to sampling and data-collection methods and planning for use of results. At this point the chapter started to get interesting, but then ended. Hopefully there is more of the same in chapter 3.

Evidence Based Evaluation of University Courses

Groundwater-Smith and Cusworth (1998) point out that opinions about education courses are formed by individuals. Given the large amount of time and money invested in education by the state, parents, teachers and students, something more than opinion is needed to decide on how good a course, program or university is. The educational buzzwords for this are evaluation. , with evidence gathering and collegiate interpretation. But in the end it comes down to opinion, admittedly the opinion of many students, teachers, practitioners, employers and others combined together, but it is still subjective opinion. There is no objective way to evaluate a course: in the end someone has to decide what is important, based on their opinion of what is important.

Groundwater-Smith and Cusworth describe evaluation as: "... a process that allows school professionals to gather evidence in an orderly manner..." and "... well-informed judgment ...". So this is not just a matter of collecting results and adding them up.

Real Time Course Evaluation

Ideally courses are designed and tested before they are run with students. Teachers should not be using their students as involuntary experimental subjects, to try out new educational material and techniques (that would be unethical). The course should have been first designed and run through some form of evaluation, such as peers reading through the material and trying out the assessment items. But some course evaluation will could be take place in real time as a course is run, checking if the course meets the needs of the cohort of students and if there are changes needed due to circumstances.

The teacher will be looking at how students are progressing, by seeing what actvities they are undertaking. For a face to face class this would involve looking out at the class to see how many are actively engulfed. With a computer assisted class I have found I can use the Learning Management System to help with this, as I can see what students are doing. The =same can be done with distance on-line students: using the system to see who is up to where. However, the temptation to check every little thing the students are doing needs to be resisted: if the student get the feeling they are being continually watched, they will start to act in a way they think will keep the teacher happy (or they may rebel against the intrusion).

Assessed can also be used for evaluation of the course: if students have not done some of the formative assessment or are having problems with it, then the fault may be with the teaching, not the students. In my e-learning coruses I have weekly exercises for students and can see if there is a problem.

Link Between Assessment and Evaluation

It may seem obvious, but there is a link between assessment and evaluation in education. Some of the same theory and techniques can be applied to both and in some cases the same data. If an external form of assessment is used for several different courses then this can be used as a form of evaluation of the courses (allowing for differences in the students). If the same students do different courses, then their results can be compared for consistency. Also the result from the same course in previous years can be compared with the present (this is done routinely by university course examiner's meetings). If a course produces consistently lower results than other courses on the same topic or with the same students, then there may be a problem.

Action research

Groundwater-Smith and Cusworth go into detail on problems with Action research (AR), but without first explaining what it is. As far as I can work out from the Wikipedia entry, AR involves practitioners working together (in a "community of practice") to investigate current practices and improve them. But what is not clear is where the research comes into this. The average teacher does have the resources to conduct experimental research and using their students as subjects of an experiment would require ethical clearance. Having a group of teachers discuss teaching practice could be useful, but dressing it up with fancy terms does not make this "research".


Reference:

Groundwater-Smith, Susan. & Cusworth, Rosie Dobbins. (1998) Teaching : challenges and dilemmas Harcourt Brace, 1998: