Global J. of Engng. Educ., Vol.I, No.2
Printed in Australia

 

Copyright 1997 UICEE


 

Issues Related to the Use of Peer Assessment
in Engineering Courses
Using a Problem-Based Learning Approach*


Ian D. Thomas

Educational Services Branch, Monash University Caulfield Campus, Caulfield, VIC 3162, Australia

Roger G. Hadgraft
Peter S. Daly

Department of Civil Engineering, Monash University Clayton Campus, Clayton, VIC 3168, Australia
 

 
 

This paper addresses the general educational issues involved in the use of peer assessment in engineering courses that employ a problem-based learning approach. The paper provides a general framework, reports experiences in using this approach in a variety of engineering courses, and discusses the practical considerations involved in implementing this approach

* An expanded and revised version of a paper presented at the Congress

 

 
 

[UICEE home page] [Global Journal Home page][Table of Contents][Previous Paper][Next Paper]


TABLE OF CONTENTS


INTRODUCTION

The basic premise underlying the use of peer assessment and a problem-based learning (PBL) approach in engineering courses is that, in so doing, our students are being exposed to something closely approximating real life as a professional engineer. Students are given the opportunity to acquire knowledge and skills and to practise these skills, and to develop and be assessed on their performance of professional behaviour, attitudes and values [1]. Further, engineering educators have the opportunity to develop and evaluate learning experiences that cover a whole range of the learning objectives frequently included in course descriptions but mostly neglected in the traditional educational setting, geared as it is to the evaluation of academic performance of knowledge and skill.

A PBL approach which incorporates some component of peer assessment is, in effect, an elaborate simulation of real life performance in a professional situation.

The educational issues involved in this approach to course management are those bound up in a combination of the key elements of the situation – simulation, PBL and assessment (particularly the peer assessment component).

SIMULATIONS

The key educational issue in any form of simulation is how closely it models the real life situation. This means arranging the learning experiences and selecting assessment criteria very carefully to ensure that observed behaviour can be used to develop and assess real professional competence.

A variety of methods are used in engineering courses to simulate conditions in the professional workplace, eg laboratory exercises, field experience, computer simulations, individual research projects, group projects, to investigate engineering problems and multi-disciplinary teams working on industry generated problems [2][3].

The educational value of these approaches, in terms of the acquisition of professional engineering behaviour, skills, attitudes and values, depends very much on the criteria chosen for the evaluation process [4]. Reliance on paper and pencil tests alone, ie written reports and examinations which test academic knowledge and cognitive skills, is inadequate and inappropriate. To evaluate the performance of a multi-disciplinary team working on a real world engineering problem, it is necessary to evaluate a whole range of teamwork, group behaviour, leadership and personal performance skills. Logically, peer and self assessment must be part of this process due to their unique and intrinsic value in measuring these criteria and because this type of evaluation must (and does) occur in real life.

In an educational setting, the problems being investigated must be somewhat contrived (or tailored) to meet the educational objectives for a course of study and the administrative constraints of study timetables and commitments to other parts of the students’ academic programme. The problems cannot therefore be truly open-ended, driven by market forces, etc; ie they are simulations. Care must also be taken to ensure that all students can participate equally and fully in all the criterion tasks, ie they must know the rules, have sufficient information to undertake the tasks, and are given training in unfamiliar roles.

So the key educational issues emerging here are that:

PROBLEM-BASED LEARNING (PBL)

The important emphasis here is on learning. PBL represents a total approach to developing professional and life skills as well as supporting effective, self-directed adult learning [1].

PBL is concerned with:

There are many important concepts and educational values enshrined in these three points. The PBL approach is markedly different to the traditional lecture/tutorial/lab based engineering programme and demands different roles from the course leader. PBL implies a much more sensitive collaborator, fellow traveller, mentor, colleague, guardian role and a very tricky hands-off leadership role for the course leader. This role is both real and crucial to the success of the approach because it requires enormous effort and planning to ensure creation of an appropriate learning environment.

The educational issues about the learning environment to be created are that it should enable learning to be:

Achievement of this type of learning environment depends very much on arranging learning experiences which promote the development of a set of generalisable competencies such as those shown in Table 1.

These competencies not only enhance learning during a particular course of study, but are also important for continued development throughout their professional life. The key educational issues to emerge here are that:

Peer assessment

It is clear from the context developed above that traditional paper and pencil tests are inappropriate and inadequate for most of the competencies that a PBL approach is supposed to develop. Reliance on these tests in a course using a PBL approach gives all the wrong messages because it simply reinforces the shallow knowledge-based, algorithm-memorisation approach to learning.

One key assessment issue here is the setting of appropriate weights to the different components of the programme and to the different approaches that can be applied to assess them. Since objectives related to process, development of professional behaviour, attitudes and values are the major objectives of the PBL approach, it is logical to suggest that assessment should reflect this emphasis.

A second issue with assessment is the difficulty of finding a large enough pool of relevant problems to use as assessment items. The difficulty is that once a problem has been used for assessment and the solution publicised, the next time that question is used it will be assessing something other than problem solving. At best it will test knowledge of the rules, processes or applications needed to solve the problem, and at worst it will simply test memory of the actual solution.

A third issue is credibility. Many of the measures that have to be used to assess these process objectives are qualitative or subjective in nature and may need more than a single measure to get a true perspective on them. Engineers are traditionally suspicious of subjective measures and prefer to rely on data from so-called objective quantifiable assessments such as the marks awarded on traditional test items or the scores obtained on computer-marked multiple choice questions (MCQs). This position becomes untenable when these measures are used to infer assessment of performance on otherwise unmeasured criteria.

Evaluation of process type objectives requires quite close observation and analysis of actual on-task behaviour to develop a checklist of the attributes that one would expect to see demonstrated over time. There are at least four relevant sources of information to be used here: the assessment of the client or end user of the solution to the problem, the assessment of performance by other members of the team (peer assessment), self assessment of performance, as well as the supervisor’s assessment. Use of all these sources removes a considerable assessment burden from the supervisor and increases the relevance of the assessments made. It is these additional components of assessment that are most often ignored or actively discouraged in academic circles.

The most common arguments against peer assessment (and self assessment) are:

Evidence in the literature presents a more positive view. It indicates that peer assessment (and self assessment):

There are however a range of actions that can be taken by the course supervisor to enhance the value of the peer assessment as both a learning experience and as a component of performance assessment [8].

To this point, the paper has outlined a number of important issues to be addressed in using peer (and self) assessment in engineering courses using a PBL or contextual learning approach. We conclude with a brief discussion of three examples from different engineering courses which highlight these issues.

Example 1: A fourth year Mechanical Engineering course in refrigeration and air conditioning

The Department of Mechanical Engineering, at Monash University, restructured a traditional lecture based course in Refrigeration and Air Conditioning in 1994, and offered it as a laboratory based course [2]. The restructured subject comprised five main teaching components – laboratory work, computer-based learning units, problem sheets, text readings and some special purpose lectures.

There were four complex laboratory exercises based around the use of seven purpose-constructed modules which simulated components of a complex air conditioning/refrigeration system. Students worked in self-selected groups of 3-5 members. The laboratory tasks had defined outcomes, but were unstructured in the sense that the students had to specify for themselves the strategies to be employed and the data gathering techniques appropriate to the tasks. This meant that students were placed in a situation where they had to develop teamwork, leadership and cooperative task completion strategies.

In addition, some of the early exercises on equipment calibration and performance assessment of system modules required a sharing of information across groups before the more complex investigations could occur. This component forced the development of both written and oral communication skills, and highlighted individual and corporate responsibility for the quality of information produced. It also meant that each group had both to trust and evaluate the performance of its peers.

Peer assessment was used to evaluate group presentations of vital information and of performance within the groups of the various tasks, and represented about 20 percent of total marks. Within-group performance was also monitored by an independent observer present at all laboratory sessions with the specific objective of observing the group processes in action.

The peer assessment component resulted in almost all students gaining maximum marks for their participation in the group effort. This highlights an interesting dilemma with peer assessment. In this case, while the peer assessment contributed nothing to the variability and hence discrimination capability of the assessment of student performance, evidence from the independent observer did suggest that all of the group behaviour criteria in the course had been achieved and that the students had been appropriately rewarded. Even though it was possible to do so, no attempt was made to use the independent observer to moderate the assessments awarded by peers.

Observation of laboratory work showed that leadership in all groups tended to be democratic rather than autocratic or laissez faire. Student involvement with the investigations seemed genuine. There were positive patterns of interaction in discussion, questioning and debate. The language used in discussions about the investigations was indicative of high level reasoning with vigorous defence of experimental outcomes and suggested procedures. At each class all groups were well organised and well advanced from the previous class meeting, suggesting that some co-operative allocation of responsibility for tasks and individual initiative was occurring. The investigations engendered a sense of ownership of a piece of work at both individual and team level. While it would be instructive to include peer assessment of these outcomes in the final grades, no attempt was made to do so in the 1994 programme.

The same basic structures have been followed in 1995 and 1996, although some minor modifications have been included in the balance of assessment weights of the various components, and in limiting the scope of some investigations to make more economical use of student time.

A further outcome of this project was that students were expected to take a traditional written examination as part of their assessment. The examination was in all respects comparable to the examinations used in previous traditional course offerings. Analysis of the examination results indicates no significant difference in performance on the traditional examination questions between the two years [2]. The big plus for the 1994 course offering was the great increase in the available information about student performance on a whole range of criteria not previously assessed, and the knowledge that the programme had been successful in having students achieve criterion performance on these tasks. This is a very important educational outcome.

It would be nice to report that this initiative has been a raging success. In the sense of achieving project objectives it has, but in the sense of influencing the teaching of mechanical engineering by others, it has not, and survival of the programme is in question.

Example 2: A third year Industrial Engineering course

In 1995 a third year Industrial Engineering course, Design for Manufacture, employed a PBL approach in which students worked on a real design problem supplied by a plastics packaging manufacturer who assisted the project further by giving students extensive access to company information and production and management staff [3].

The project concerned an evaluation of the whole process of manufacturing an injection moulded plastic pail which had a printed label and a carry handle. The project was to focus particularly on alternative printing and handle installation processes that would be suitable for varying market sizes.

Two groups of 10 students (all industrial engineering students) tackled the same problem. Each group member chose or was allocated one of the ten key tasks which were determined by the nature of the project and the processes involved in the manufacture and marketing of the product. The designated roles assigned within the group were: project manager, injection moulding analysis, printing process analysis, costing of printing processes, handle insertion, materials handling and transport and plant layout, quality control, standards and inspection, costing of handle insertion and materials handling, market evaluation, occupational health and safety.

Fifty percent of the formal assessment was based on group performance. An individual’s performance in the group was graded by peer assessment to produce a weighting ranging from 0 to 1.4. An individual’s group performance score was determined by multiplying the group score by the individual weighting determined by his/her peers.

Each student had the experience of chairing and being minute secretary to a group meeting about the project. Both groups presented project reports to the company management at the end of the semester. It is interesting to note that the company later employed two of the students to further the investigation of the two solutions suggested in final reports.

In this project, engineering students had to perform unfamiliar roles requiring knowledge and expertise outside the engineering discipline. The results from this project indicated reasonable congruence of assessment of performance between the client, academic staff and peer groups of students, but there was some reluctance on the part of some students to assess peers.

In 1996 the approach was broadened by adding students from accounting, marketing, and industrial design courses to work with industrial engineering students in multi-disciplinary teams. In all, six teams were formed to work on projects generated by three separate plastics moulding companies.

One of the interesting assessment issues that arose early in the project was how to cope with different assessment requirements and criteria from the different academic disciplines involved, and some reluctance from academic staff to have students from other disciplines having an input into the assessment of the academic performance in their discipline.

Considerable negotiation and the involvement of a person from the Leadership and Management Development Unit within the Professional Development Centre resulted in an agreed process for engaging in peer and self assessment which also involved the students in the setting of the assessment criteria. The formal evaluation of these processes is yet to be completed, but should appear in published form early in 1997.

The problems with assessment highlight the organisational difficulties and prejudices that can impinge on the development of real cross-discipline project teams, and just how seriously the assessment tail can wag the educational dog.

One issue emerging from the 1995 experience and the planning for the 1996 project is the need to provide more training of group members in group processes and in the techniques and processes of peer assessment. Some attempt was made to do this in this project in the early sessions and through the use of two formative trials of the peer/self assessment rating scale. Some attempt to moderate this whole process was achieved by having independent observers provide the project team with an assessment of the group processes in the teams. We employed graduate students in an Organisation Psychology Programme as the observers and they used their involvement in the project as part of their assessment in their course, adding an interesting further dimension to the multi-disciplinary nature of the project. The final results of their observations are still to be collated and included in the project report.

Example 3: Fourth year Civil Engineering design

In 1995 the Department of Civil Engineering offered, for the first time, an integrated design subject at fourth year level. Previously, separate designs had operated in the areas of structures, water, soils and transport.

The form of the self- and peer-assessment used at each of the 4 stages of the group project required each student to rank each group member plus themselves on each of the following 4 items, using a 5 point scale from strongly agree to strongly disagree:

Students were also invited to comment on progress or the process of assessment in an open-ended question. The surveys were compulsory and confidential. Groups contained either four or five students.

Most well functioning groups answered these questions in a seemingly casual way, with all group members giving all other group members the same rating (Figure 1). In most cases this was a strongly agree rating, with 165 observations (95%) from the 174 responses shown in Figure 1 falling into this category. It was encouraging to note that students took the assessment more seriously as the marks for each stage increased. Stages 3 and 4 counted for 60% of the marks in the subject, and of the 9 ratings that were not strongly agree, 7 were recorded in these two stages.

Some of the comments made were quite hostile in a small number of groups however, indicating real difficulties in group processes. This was not backed up with significant differences in the closed-ended questions. Of the 43 different ratings shown in Figure 1, only 2 showed more than 4 scales difference (strongly agree to disagree) and the remainder consisted of 1 or 2 scales difference, with none of these giving a scale below undecided.

The inconclusiveness of the assessment results can be explained in part by structural deficiencies in the assessment model used. This is backed up by comments made by students in the open-ended assessment question and in a subsequent course survey. Criticisms of the model fell generally into 3 categories:

It is suspected that in 1995 many groups did not take the confidentiality component seriously, and, in at least one case, it was shown that one student had completed all forms for every stage.

In 1996 the assessment process was more transparent, with the students free to discuss self and peer assessment within the group. Students were given time at the completion of every week’s class for this task.

The deficiencies apparent in the model used in 1995 were addressed, with a more careful approach used in 1996. Woods emphasises a model for assessment based on objectives, measurable criteria and evidence [8]. This was handled in 2 ways, on a weekly basis:

Both the goals and performance indicators were recorded on pre-prepared sheets, bound together in a log book. The group kept this log book throughout the semester. Each week the completed entry was to be signed by the group leader and the staff member mentoring that group.

The revised model was presented to the class in a draft form in a dedicated session at the start of the semester, and students were invited to participate in further development. The purpose of this was twofold: to give students a sense of ownership (encouraging higher quality participation), and to utilise the very real skills and perceptions of the student body for further improvements. Changes suggested by the students related mainly to the administration and layout of the log book and these were subsequently included in the final version.

The changes in 1996 resulted in better quality, more honest responses in the majority of groups - probably due to the open nature of the exercise. It was planned that the accumulation of all this evidence throughout the semester would then form an important part of the final assessment. This did not eventuate however as staff were not diligent enough in ensuring all groups completed the log books satisfactorily. It was decided, as a result, not to use them for assessment purposes on equity and consistency grounds across the class as a whole.

More importantly, however, the log books provided a methodology for groups to set and reflect on their weekly goals. Most groups did complete the log books each week and, as such, these groups provided week by week formative assessment to all group members by which they improved their performance. It was observed that the groups who did this outperformed other groups significantly.

For 1997 a slightly different format is planned whilst still retaining the principles from 1996. The log book format will be modified so that the groups can record simplified minutes of their weekly meetings. The minutes will record ideas raised, decisions reached and goals/tasks to be undertaken. The group member responsible for each of these will be recorded and it is hoped that a pattern of functionality of each group member might emerge, eg chairman, shaper, plant, etc [9]. The log book itself will remain, with pre-prepared sheets to guide students and lessen the time taken to undertake the exercise.

The changes will again be presented to the class for possible modification. Staff mentoring each group will keep closer contact with the exercise to encourage full participation. It is hoped that the refinements for 1997 will continue to provide valuable feedback to both the group members and staff.

CONCLUDING COMMENTS

From the experiences reported in the three different examples of problem-based learning in engineering courses, it is clear that PBL courses not only require a very different approach to teaching and learning, but also require the solution of some tricky assessment and educational issues. The more closely PBL courses resemble real life situations, the more diverse the assessment issues become, and the more important the peer assessment component becomes in establishing a meaningful measure of performance.

It is also clear that evaluation of these projects bring to light compelling evidence of additional benefits (in terms of educational outcomes) for students that cannot be overlooked in cost-benefit analyses of PBL approaches to engineering courses employing peer and self assessment techniques of measuring student performance.

REFERENCES

1. Boud, D. and Feletti, G.I., Introduction. In: Boud, D. and Feletti, G. (Eds), The Challenge of Problem-Based Learning. London: Kogan Page (1991).

2. Eley, M.G., Hessami, M.A., Gani, R., Sillitoe, J.F. and Thomas, I.D., A Laboratory Based Approach to Teaching in Engineering. In: Zelmer A.C.L (Ed), Higher Education: Blending Tradition and Technology. Research and Development in Higher Education. 18, 284-289 (1995).

3. Wellington, P., Industrial engineering students address a real manufacturing problem. Research and Development in Problem Based Learning, 3, 449-462 (1995).

4. Frederiksen, N., The real test bias: influences of testing on teaching and learning. American Psychologist, 39, 193-202 (1984).

5. Marton, F. and Säijö, R., On qualitative differences in learning I – outcome and process. British Journal of Educational Psychology, 46, 4-11 (1976).

6. Marton, F. and Säijö, R, On qualitative differences in learning II – outcome as a function of the learner’s conception of the task. British Journal of Educational Psychology, 46, 115-127 (1976).

7. Scott, J., Watson, A. and Yerbury, H., Using peer assessment in grading undergraduates. National Teaching Workshop, ANU, Canberra, 67-71 (1994).

8. Woods, D. R., Problem-based Learning: how to get the most from PBL. Hamilton, Canada: McMaster University (1994).

9. Belbin, R. M., Management teams: why they succeed or fail. London: Heinemann (1981).


BIOGRAPHY

Ian D. Thomas

Roger G. Hadgraft

Peter S. Daly


[UICEE home page] [Global Journal Home page][Table of Contents][Previous Paper][Next Paper]