Assessment beyond the degree factory

Greetings from day two of the Faculty Board Meeting of ACS Education, the part of the Australian Computer Society which provides education to computer professionals. There are about 30 people around a table in the ACS South Australia Branch office.

Assessment

We started with a presentation on the role of assessment in academic integrity. What I found interesting about this was the idea that both the teacher and the student have mutual obligations and are partners in the education process. Both the teachers and students have obligations as well as rights. It is academic integrity, based on years of experience , as well as qualifications, which allows a teacher to say what grade a student gets, not some set of assessment rules.

Collegiality

Part of academic integrity is collegiality and peer review. That is the teachers are not just employees who do whatever the boss says. The staff work out jointly what is to be done and what is reasonable. Senior staff act as coordinators, rather than dictators. Contrary to popular wisdom, I have experienced this approach working in a military headquarters. Senior military officers are trained to take advice from their junior staff. They will listen to the debate, with even the most junior allowed to disagree with the most senior. The staff officers (much like executive assistants) judges what the consensus is and issues the formal orders in the name of the commanding officer. A similar process works in traditional universities.

Forms of Assessment

Assessment is usually divided into Formative and Summative. The formative assessment is to help he student learn, whereas the summative assessment measures what the student has learnt against some objectives. Bloom's Taxonomy, has three domains: Cognitive, Affective, and Psychomotor. Increasingly there are also social issues.

Learning Objectives

The modern fashion is to express what the course is trying to achieve explicitly in learning objectives for a course and to present these to the student. I have some doubts as to the value of this, as often these are expressed in such a complex way with educational jargon that the student can't understand it. Large multicoloured tables with pop-up boxes are, in my view, not useful.

Building the Course Design Workflow into the LMS

The obvious thing for IT professionals to do is to design courses with the same sort of systematic process they use to design software. IT based systems can also be used to support the development of courses and their peer review. In the past this would be done by a bespoke system specifically designed for entering course designs and then having a work-flow for review and approval. However, it occurred to me that the LMS could be used for this as rapid prototyping. Rather than filling out forms to detail what it is proposed to put in the course, the details would be entered directly into the LMS, creating the skeleton of the course. The discussion forums and assessment feature of the LMS would then be used for the peer review.

Skills Framework for the Information Age

The ACS uses the Skills Framework for the Information Age (SFIA) framework for designing courses. This is mostly done at SFIA Level 5, the level of a person who has done an undergraduate course, been working for a few years and aspires to advancement needs.

Assessment for the Real World

The question then is how to assess if the student has achieved this. Because this is professional training, in my view this is conceptually very simple to assess: just get the students to demonstrate they can do whatever the professional has to do. As an example, I get Green ICT students to assess the greenhouse gas emissions form ICT in an organisation and to make recommendations on reducing them, as this is what they will be doing in the workplace. It will be more difficult for students to demonstrate the skills in other fields, such as project management. However, this can be done with simulations, just as airline pilots are mostly trained and assessed in a simulator, not a real aircraft.

As well as specific skills for specific jobs, SFIA has "generic levels", which can be used for defining core competences. Unfortunately there is a level of confusion with these sometimes being referred to as "generic skills". Here are SFIA "generic skills" defined at Level of Responsibility 5:

• Challenge range and variety of complex technical or professional work activities;
• Influences organisation, customers, suppliers, and peers within industry on contribution of specialisation;
• Work requires application of fundamental principles in a wide and often unpredictable range of contexts;
• Maintains awareness of developments in the industry;

From: Enterprise Architecture (Elective Subject), ACS Education, 2011

The SFIA generic levels of responsibility are listed in
"Framework reference SFIA version 4: Skill definitions in categories, subcategories and skills" (SFIA, 2008). However this is a large and difficult to read PDF document. The HTML version, designed by Text Matters is much more accessible:

Autonomy

Works under broad direction. Is fully accountable for own technical work and/or project/supervisory responsibilities. Receives assignments in the form of objectives. Establishes own milestones and team objectives, and delegates responsibilities. Work is often self-initiated.

Influence

Influences organisation, customers, suppliers and peers within industry on the contribution of own specialism. Has significant responsibility for the work of others and for the allocation of resources. Makes decisions which impact on the success of assigned projects i.e. results, deadlines and budget. Develops business relationships with customers.

Complexity

Performs a challenging range and variety of complex technical or professional work activities. Undertakes work which requires the application of fundamental principles in a wide and often unpredictable range of contexts. Understands the relationship between own specialism and wider customer/organisational requirements.

Business skills

Advises on the available standards, methods, tools and applications relevant to own specialism and can make correct choices from alternatives. Analyses, diagnoses, designs, plans, executes and evaluates work to time, cost and quality targets. Communicates effectively, formally and informally, with colleagues, subordinates and customers. Demonstrates leadership. Facilitates collaboration between stakeholders who have diverse objectives. Understands the relevance of own area of responsibility/specialism to the employing organisation. Takes customer requirements into account when making proposals. Takes initiative to keep skills up to date. Mentors more junior colleagues. Maintains an awareness of developments in the industry. Analyses requirements and advises on scope and options for operational improvement. Demonstrates creativity and innovation in applying solutions for the benefit of the customer.

From: Levels of responsibility: Level 5, ensure, advise, SFIA Foundation, 2005

ACS and SFIA had room for improvement in settling on a clear and consistent title for this is. However, ACS and SIFA are far more consistent in their use of terminology than universities with "graduate attributes".

Graduate Attributes

The Graduate Attributes are an Austrlaian term, first required by the Austrlaian Government in 1998 for universities to include in the Quality Assurance and Improvement Plans submitted to government. In theory each unviersity defines its own attribuites, but as the Government says, most follow the same pattern:

• Knowledge attributes - in general, graduates are expected to have good literacy and numeracy skills, the ability to communicate and listen and appropriate discipline-specific knowledge.
• Thinking attributes - graduates are expected to have good conceptual and problem solving skills, the ability to question, be creative and to combine theory and practice.
• Practical attributes - emphasises the ability of graduates to use information technology and be proficient in any other technical skills appropriate to their discipline. The ability to initiate and respond to change is also considered an important attribute.
• Personal attributes and values - graduates are typically expected to have a commitment to learning, be flexible and able to work in a team, have leadership skills and understand the concepts of ethical action and social responsibility.

From: Higher education report for the 2001 to 2003 triennium, DEST, 2008

An example are the ANU graduate attributes:

• have an in-depth knowledge base and comprehensive understanding of the conceptual and theoretical underpinnings of their disciplines;
• are well acquainted with the broader contexts of higher learning;
• trained in computer based technology, and relevant discipline-based technical and methodological skills;
• are independent, critical and creative thinkers with analytical and problem solving skills;
• are competent in literary and oral communication;
• value intellectual rigour, creativity, curiosity, integrity and lifelong learning; and
• understand major issues facing Australia and the wider world

From: The Australian National University, Department of Education, Training and Youth Affairs, 1999

While SFIA does not specify its attributes in great detail at least one standard set is provided unlike the Austrlaian unviersity system of a confusing and wasteful proliferation of non-standard terminology.

Short Courses

There were then two workshops, one for mentors on mentoring and one on teaching. I attended the teaching workshop on the topic of the human side of teaching. This reminded me of the issue of how to provide short professional development courses. The ACS is running an ACS Business Skills On-Line Pilot Program. I was unable to undertake a course due to technical problems but it struck me that the one on one for the course support was very good. Typically short online courses (of one to three hours) are solitary, with the one student reading prepared material, watching videos and doing multiple choice tests. One area I had difficulty with was selecting suitable courses and how these would form an overall plan.

The Open University short courses, differ from the typical audio slide-show course. These are more like shortened versions of the ACS CPEP modules. However, this requires a tutor and a a forum for the students to learn from each other. OU do provide a forum for each course, but not the tutor.

ACS might use the tutor, mentor and forum techniques from its longer courses and PPP for short courses. It is not feasible to set up the infrastructure used for a 13 week course for one lasting an hour. But ACS requires each 30 hours of professional development each year of its ACS Certified Professionals (CP). The ACS Computer Professional Education Program subjects each require 8-10 hours per week wrk from each student (as do typical tertiary courses). So the professional development each year requires the equivalent time to a three to four week course.

So my thought is to treat the annual requirement for development as a three to four week course undertaken over on year. The student would be assignment a tutor/mentor and be part of tutorial groups online. The student would get help planning what they needed to learn and selecting courses to do. The actual courses could be provided by ACS and by other providers. Where the courses lack their own tutor and forums, the ACS would provide this. The student would produce an e-Portfolio.

Tutors Needed for ACS Education

The last topic for the ACS Education Faculty Board meeting was about how to obtain the needed staff. To me the obvious source is previous students, who can be come first tutors and then course designers. Because the ACS CPEP uses a different approach to most tertiary level courses, the average teacher will not have the needed experience for these courses.

ps: The meeting got a little surreal, where at one point I was typing this blog post as it got a mention as an example of something a professional could claim professional development hours for. ;-)