ICT4LT Module

Computer Aided Assessment (CAA) and language learning


Contents


Aims

The aims of this module are for the user to consider key issues in assessing language skills through ICT in order to be able to:

  1. assess language learning outcomes when ICT is involved in the learning process,
  2. use appropriate technologies to assess progress and provide feedback,
  3. use Computer Aided Assessment (CAA) as an integral part of the language learning process.

This Web page is designed to be read from the printed page. Use File / Print in your browser to produce a printed copy. After you have digested the contents of the printed copy, come back to the onscreen version to follow up the hyperlinks.


Authors of this module

Terry Atkinson, Freelance Educational Consultant, UK/France.

Graham Davies, Editor-in-Chief, ICT4LT Website.


1. Introduction

Contents of Section 1

1.1 What do we mean by Computer Aided Assessment?

Computer Aided Assessment (CAA) covers a range of assessment procedures and is a rapidly developing area as new technologies are harnessed. In essence, CAA refers to any instance in which some aspect of computer technology is deployed as part of the assessment process. Some of the principle examples of CAA in language learning are:

CAA is more than just a list of possible applications, however. Its importance is intimately bound up with raising achievement, since it can be argued that the role of ICT in raising achievement cannot be fully measured unless ICT is also used in the assessment process.

Just as oral skills cannot easily be assessed by a written test, so there are ICT-specific language skills that cannot easily be assessed through pencil and paper exercises and tests. Consider the following examples:

If students spend time practising the above activities on computers the intention is to raise achievement in language learning at a general level, and this might well be picked up in a conventional test or examination. However, as the balance moves further and most of students' reading is online and most of their writing is computer-based, using paper technology is unlikely to enable students to demonstrate fully their skills, particularly ICT-related ones such as using writing tools - spellchecker, thesaurus, etc. An integrated approach to teaching, learning and assessment is always likely to be more successful than a random approach. Thus, teaching methods, learning methods and assessment methods need to cohere if learners are to learn successfully and if valid and reliable results are to be achieved from assessment procedures. This is true whether or not ICT is to be used - extensive oral practice activities and no oral exam is still a problem in many classes, and so too is extensive use of computers and no opportunity to use computers as part of the assessment process. Moreover, the incentive for teachers to incorporate learning technologies into classwork is reduced if the examinations do not use these technologies or expressly prohibit their use.

1.2 Formative and summative assessment

Formative assessment: In general, CAA is used mainly for formative assessment rather than summative assessment because it is excellent for giving immediate feedback, e.g. in tests designed to measure students' progress in specific areas, either for self-assessment purposes or for the teacher, e.g. as in placement tests (see Section 1.4).

Summative asssessment: There is a good deal of discussion at present regarding the use of CAA - also dubbed e-assessment - in examinations at the end of a course and in national examinations such as the GCSE examinations in England. It's a controversial topic and has been subjected to a good deal of media hype, with outrageous claims being made regarding its possible uses. To what extent do you think CAA can be used to carry out summative assessment? Jot down your immediate thoughts on this question and the concerns that you might have if your own students were to be assessed in this way. Then read Section 1.3, Which skills can be assessed? Go back to your list of concerns and see which ones have been answered fully, partly or not at all. Send a message to us, using our Feedback Form, if you still have concerns that have not been answered.

Use of CAA for summative assessment is increasing, especially in higher education. Trainee language teachers in England and Wales are now expected to pass computer-based skills tests in literacy, numeracy and ICT, so some of the readers of this module have already experienced summative CAA and will be familiar with the procedures that make it possible. These include:

The above measures can ensure that there is as much security involved as in paper-based exams and the process is not too dissimilar to that used for carrying out orals. Perhaps a more important question is whether computer-based tests can really assess language skills and, if so, which skills are best assessed through CAA and which through other formats. These questions are addressed in the next section. Summative assessment is not likely to be widely implemented in the near future as there are still a number of concerns about its reliability, but it is likely that ingenious solutions and new technologies will bring about a much greater degree of summative assessment than is currently possible. Placement testing and adaptive testing are more likely to be introduced in some areas: see Section 2.3.

1.3 Which skills can be assessed?

Skill Assessment by computer Assessment of electronic output by human being
Listening Computer can assess a limited range of different types of responses to test comprehension. Listening tests can be presented on a computer, students' answers can be stored electronically and assessed by a teacher. Self-assessment and peer assessment are also possible.
Speaking Very limited as yet. Automatic Speech Recognition (ASR) software is developing rapidly but it is still too unreliable to be used in accurate testing. Students can record their own voices on a computer for assessment by a teacher. Self-assessment and peer assessment are also possible.
Reading Computer can assess a limited range of different types of responses to test comprehension. Reading tests can be presented on a computer, students' answers can be stored electronically and assessed by a teacher. Self-assessment and peer assessment are also possible.
Writing Very limited as yet, but spellchecking, grammar checking and style checking are possible, and some progress is being made in the development of programs that can assess continuous text. Students' answers can be stored electronically and assessed by a teacher. Self-assessment and peer assessment are also possible.

Listening

At a basic level it is simple to assess listening comprehension in much the same way as it is possible to assess reading comprehension, e.g. with multiple-choice, drag-and-drop and fill-in-the-blank tests. If well designed, this form of assessment works effectively and instant feedback can be offered to the student, which has a beneficial effect on learning. The main ways of assessing listening skills can be summarised as follows:

  1. Multiple-choice, drag-and-drop, and fill-in-the-blank tests with single-words or very short sentences, but these types of tests cannot easily assess more open-ended aspects such as the ability to infer; and in multiple-choices tests students can get the answers right by guesswork.
  2. Completely open-ended answers cannot be assessed. Single-word answers or answers consisting of very short sentences can be assessed to a limited extent.

Despite these limitations, assessment of listening comprehension by computer can be of great value to students, offering a form of comprehensible input (Krashen 1985). Moreover, computer-based listening comprehension can combine sound with text, still images, video, animation and on-screen interactivity which creates thereby a much richer environment than is otherwise possible: see Module 2.2, Introduction to multimedia CALL, and Module 3.2, CALL software design and implementation. A measure of student control to allow ease of navigation, options to retry or move to a different section, to attempt different tasks or roles etc are vital to ensure active participation. Good equipment is also vital: headphones, fast network/Internet access and/or networked CD-ROMs.

Speaking

Limited assessment of speaking skills is possible. Self-assessment and peer assessment can be managed if facilities are available (e.g. microphone and headphones) to allow students to record themselves and listen to the playback. A number of multimedia CD-ROMs have this feature: see Module 2.2. The Encounters series of CD-ROMs, for example, allows students to take part in a role-play by recording their own voices - and re-recording them until they are satisified with the results - and then saving the whole role-play on disc, with their own voices slotted into the appropriate positions in the role-play, for assessment by a teacher. The Encounters series of CD-ROMs was produced by the TELL Consortium, University of Hull and is available from Camsoft.

To assess speaking skills solely by a computer, using Automatic Speech Recognition (ASR), is a very complex task and research in this area is developing rapidly. ASR can be motivating for students working independently, but computers are still not completely reliable as assessors. For further information on ASR see:

See the software for automated testing of spoken English produced by Versant.

Reading

At a basic level it is simple to assess reading comprehension in much the same way as it is possible to assess listening comprehension, e.g. with multiple-choice, drag-and-drop and fill-in-the-blank tests. If well designed, this form of assessment works effectively and instant feedback can be offered to the student, which has a beneficial effect on learning. The main ways of assessing reading skills can be summarised as follows:

  1. Multiple-choice, drag-and-drop, and fill-in-the-blank tests with single-words or very short sentences, but these types of tests cannot easily assess more open-ended aspects such as the ability to infer; and in multiple-choices tests students can get the answers right by guesswork.
  2. Completely open-ended answers cannot be assessed. Single-word answers or answers consisting of very short sentences can be assessed to a limited extent.

More extended reading tasks are harder to set on computer. On-screen reading of longer texts is in any case inadvisable. Research indicates that people read around 25%-30% more slowly from a computer screen. Web guru Jakob Nielsen writes:

Reading from computer screens is about 25% slower than reading from paper. Even users who don't know this human factors research usually say that they feel unpleasant when reading online text. As a result, people don't want to read a lot of text from computer screens: you should write 50% less text and not just 25% less since it's not only a matter of reading speed but also a matter of feeling good. We also know that users don't like to scroll: one more reason to keep pages short. [...] Because it is so painful to read text on computer screens and because the online experience seems to foster some amount of impatience, users tend not to read streams of text fully. Instead, users scan text and pick out keywords, sentences, and paragraphs of interest while skipping over those parts of the text they care less about. (Source: Be succinct! Writing for the Web, Alertbox, 15 March 1997.)

More recent research by Nielsen, in which the iPad and Kindle were examined, showed that

The iPad measured at 6.2% lower reading speed than the printed book, whereas the Kindle measured at 10.7% slower than print. However, the difference between the two devices was not statistically significant because of the data's fairly high variability. Thus, the only fair conclusion is that we can't say for sure which device offers the fastest reading speed. In any case, the difference would be so small that it wouldn't be a reason to buy one over the other. But we can say that tablets still haven't beaten the printed book: the difference between Kindle and the book was significant at the p<.01 level, and the difference between iPad and the book was marginally significant at p=.06. (Source: iPad and Kindle reading speeds, Alertbox, 2 July 2010.)

See Nielsen's other articles on Writing for the Web.

Extended texts are more likely to be print-based unless they are in hypertext format, i.e. separate pages linked together as on the Web, or CD-ROM reference materials. In the case of hypertext, the computer may be a suitable medium for assessing information-gathering techniques. The skills needed to track down documents, follow links within and between them and find specific extracts are of increasing importance in academic life, in commercial settings and in leisure time. These skills are, to an extent, generic rather than specific to any given language although learners do need to know key terms involved in searching for information. Many teachers are aware of the need to ensure that learners are equipped with the appropriate language skills for Web browsing, e.g. foreign language terms for help, search, next page, OK etc.

Writing

Limited assessment of writing skills is possible. It is fairly straightforward to program computers to assess the accuracy of single words and short sentences typed at the keyboard, and work on parsing students' typed responses, diagnosing errors and providing appropriate feedback is in progress - see Section 8 in Module 3.5, headed Parser-based CALL, and see Heift & Schulze (2003). There are also features in modern computer software that can be used within the assessment process, such as spellcheckers to enable self-assessment of spelling, and also grammar and style checkers, which - although still imperfect - do pick up many errors that students can use to self-correct, such as errors of gender or number: see example below which is a screenshot from Microsoft Word. See Section 6.1, Module 1.3, headed Spellcheckers, grammar checkers and style checkers. As for assessing continuous pieces of text, some progress has been made in developing programs that can roughly grade short essays.

Spellchecker

Figure 1: Screenshot, Microsoft Word Spellchecker

The main use of computers in the assessment of writing is, currently, the use of on-screen marking as described in Section 3.

You can read more about grammar and style checkers in:

1.4 Exercise or test?

Computer-based exercises and tests often take the same kind of format. The essential difference between an exercise and a test is the purpose to which it is put. An exercise usually offers instant feedback to the learner and an opportunity to correct any errors that are made, whereas a test may offer little feedback to the learner apart from a raw score at the end of the test, or no feedback at all, e.g. where the results of the test might be stored for analysis by a teacher or examination body. Exercises are usually designed to offer the learner practice in specific areas and to motivate and encourage, whereas tests are usually designed to assess the learner's progress in specific areas, i.e. for self-assessment purposes, for the teacher or for an examination body. But sometimes these distinctions become blurred. The main kinds of tests include:

See Linguanet Worldwide website: This project has recently undergone expansion to incorporate an interface in a number of new languages and addresses in particular the needs of adult learners and independent learners. The site includes advice on ways of assessing and improving one's current ability in different languages (including links to websites that offer diagnostic tests and placement tests), communicating electronically with other language learners and finding appropriate resources. A substantial online catalogue of language learning resources is also being built up.

The Goethe-Institut offers an online placement test for learners of German.

The BBC Languages website offers basic placement tests. Choose the language first then try the placement test.


2. Types of Computer Aided Assessment and the Common European Framework of Reference for Languages

Contents of Section 2

2.1 Interactive exercises and tests

When reading this section bear in mind the distinction that was made earlier between exercises and tests. See Section 1.4, headed Exericise or test?

2.1.1 The feedback loop

The feedback loop refers to the process through which the learner is giving an opportunity to produce an answer and to receive feedback on his/her input. Feedback is essential to learning, but the nature of the feedback is also important and the computer has certain advantages and disadvantages in providing that feedback. See:

See also: Bangs (2003).

The two main advantages of feedback are that it can be given (i) instantly and (ii) in a non-judgemental way. The major disadvantage is that the feedback given may not necessarily be adequately discriminatory or differentiated. It is possible to provide detailed, comprehensive feedback but this is very time-consuming. Consider a multiple-choice test in which four answers are given for each question. There are lots of software packages that allow the authoring of multiple-choice tests of this sort and for specific feedback to be written in for each answer. However, constructing such a test takes a significant amount of time, both in thinking through the details of the feedback and in typing it all in: see Module 2.5 , Introduction to CALL authoring programs. Despite this disadvantage, the two main advantages of feedback do bring excellent learning opportunities which can be authored by anyone with basic computer literacy. Heather Rendall has shown how a very simple program in which jumbled sentences have to be re-ordered can be used to develop grammatical insight: see the report on her research project in Section 5, Module 1.4. See also Rendall (2001).

The immediate assessment that the learner receives provides the key to learning. Consider the following example:

the talks boys teacher to the

The task is to enter the text in the correct order, i.e.

The teacher talks to the boys

Consider what the learner must know in order to get this correct: Meaning? Word order? Verb forms? If the learner gets the answer wrong, he/she will be presented with the correct answer immediately. Whilst this is not the most sophisticated form of feedback, Rendall's research points towards the power of this type of feedback in developing an instinctive grasp of the working of the language. Her subjects justify their thinking by saying "it sounds right", so the effect of feedback is to develop a sense of what is linguistically correct. This takes lots of practice but with sentence forms like the one in the above example, banks of exercises can be quickly generated. In fact, it works best at short sentence or phrase level because that eliminates ambiguity. Here are some grammar points than can be tested through this simple procedure:

2.1.1.1 Learning task

Can you draft some examples of the above points for the language(s) that you teach? Can you think of other points that could be tested in this way?

2.1.2 Exercise types

The most common exercise types are listed below:

It should be borne in mind that since the advent of the multimedia computer tests and exercises can contain a variety of different kinds of presentation of the stimulus, student input and feedback, e.g. a multiple-choice test could consist of

The possibilities are enormous. See Section 5, Module 3.2, headed Template examples.

2.1.3 Sources of exercises

Collections of ready-made exercises exist for the more commonly taught languages. These can be found in software catalogues and are increasingly available online via websites: see the Websites list (below). Many exercises have been developed by teachers and are free for non-commercial use. As an alternative to buying ready-made exercises, it is possible to use authoring programs to develop your own exercises: see Module 2.5, Introduction to CALL authoring programs. The following authoring programs are all in common use.

Hot Potatoes
An authoring tool that was created by Martin Holmes and Stewart Arneil at the University of Victoria, Canada, and launched in 1998. It enables the speedy creation of Web-based exercises for language learners, including multiple choice, gap-filling, matching, jumbled sentences, crosswords and short text entry. This authoring tool has proved extremely popular with language teachers and it continues to be used extensively for the creation of interactive exercises and tests on the Web. Visit the Hot Potatoes website to find out more, download the software and see lots of examples: http://hotpot.uvic.ca. See Winke & MacGregor (2001) for a review of Hot Potatoes.

Quia
This is an online provider of exercises. There are many language learning exercises which are free to use. It is also possible to develop your own exercises. It is very easy to do this and to store the exercise on the Quia website, although subscription charges may apply: http://www.quia.com

Fun With Texts
This is one of the most widely used authoring packages for Modern Foreign Languages in UK secondary schools and there are many ready-made exercises available as well as full authoring functions within the program. Its use is primarily in total text reconstruction - so-called total Cloze - but it includes other exercise types too. Version 4.0 includes multimedia enhancements that allow you to integrate images and audio and video recordings into the exercises. See Section 8, Module 1.4, headed Text manipulation, and see http://www.camsoftpartners.co.uk/fwt.htm

Question Mark
The Question Mark company develops software for presenting interactive exercises and gathering/collating test information: http://www.questionmark.com

TaskMagic
An authoring tool, produced by mdlsoft, for the creation of a variety of exercise types, including.text match, picture match, sound match, picture-sound match, grid match, mix and gap, exercises based on dialogues.

Websites
See the Websites list (below).

2.2 The Common European Framework of Reference for Languages

The Council of Europe's Common European Framework of Reference for Languages (usually known as the CEFR or CEF) provides a common basis for the elaboration of language syllabuses, curriculum guidelines, examinations and textbooks across Europe, based on research conducted by the Council of Europe dating back to the 1970s. It describes in detail what language learners have to learn to do in order to use a language for communication and what knowledge and skills they have to develop in order to be able to act effectively. The description also covers the cultural context in which language is set. The CEFR defines six reference levels of proficiency, which allow learners’ progress to be measured at each stage of learning and on a life-long basis. The Council of Europe's publications specify the number of guided learning hours (GLH) for the six levels, e.g.

The following table shows the correspondences between the six CEFR Levels, Cambridge General English Exams, the Languages Ladder, National Curriculum (England), National Examinations (England) and the number of guided learning hours required for each level. Note: "England", not Wales, Scotland and Northern Ireland, which have different curricula and examination schemes - although they are broadly similar in most respects.

CEFR Level

Cambridge General English Exams

Languages
Ladder
National
Curriculum
(England)
National Exams (England)
Guided
Learning Hours
A1
Breakthrough
 

1-3
Breakthrough

1-3 NQF Entry Level 1-3 Approx 90-100 hours
A2
Waystage
Key English Test (KET) 4-6
Preliminary
4-6 Foundation GCSE Approx 180-200 hours
B1
Threshold
Preliminary English Test (PET) 7-9
Intermediate
7-8 & EP* Higher GCSE Approx 350-400 hours

B2
Vantage

First Certificate in English (FCE) 10-12
Advanced
  AS / A / AEA Level Approx 500-600 hours
C1
Proficiency
Certificate in Advanced English (CAE) 13
Proficiency
  University Degree Level and above Approx 700-800 hours
C2
Mastery
Certificate of Proficiency in English (CPE) 14
Mastery
  University Degree Level and above Approx 1000-1200 hours

The guided learning hours required may vary, of course, depending upon other factors such as the learner’s mother tongue, other languages already learned, the intensity of the course being followed, the inclination and age of the learner, and the amount of exposure to the language outside lesson times, particularly if the course takes place in a country where the language is spoken.

The CEFR document can be downloaded from the Council of Europe's CEFR website. A Self-assessment Grid relating to the CEFR is available as a Word DOC file. See also the Council of Europe's Language Policy Division website.

Many European countries have already adopted the CEFR as a standard for assessing language proficiency. In the UK the Languages Ladder leans heavily upon the CEFR. Although the Languages Ladder appears to duplicate to some extent what the CEFR has already produced, it offers a new approach to measuring proficiency in the four skills. The Asset Languages assessment scheme is related closely to the Languages Ladder and is designed to provide accreditation options for learners of all ages and abilities from primary to further, higher and adult education. Like the CEFR, the Asset Languages scheme offers sets of "Can Do" Statements that describe what the learner is capable of doing in the different skills.

The Cambridge University ESOL website is very informative regarding the CEFR, especially these pages:

See the website of the Association of Language Testers in Europe (ALTE). ALTE also provides a comprehensive list of "Can Do" Statements.

2.2.1 The DIALANG diagnostic testing project

What level of competence have students reached and how can it be assessed? As learners move through the system the need for accurate information on the level reached is vital. Increasingly, computers are being used for diagnostic testing.

DIALANG was a major EU-funded project aiming to provide effective diagnosis of language competence in 14 EU languages, based on the Common European Framework of Reference for Languages: see Section 2.2. Tests for Listening, Writing, Reading, Structures and Vocabulary were created for 14 languages: Danish, Dutch, English, Finnish, French, German, Greek, Icelandic, Irish, Italian, Norwegian, Portuguese, Spanish and Swedish. The website is under reconstruction and not accessible at present, but you can read about DIALANG in Wikipedia: http://en.wikipedia.org/wiki/DIALANG

DIALANG

Figure 2: Screenshot, DIALANG

2.2.2 WebCEF: collaborative assessment of oral language skills through the Web

The EU-funded WebCEF project (2006-2009) aims to enable the collaborative assessment of oral language proficiency through a Web-based environment: http://www.webcef.eu

Language learners and language teachers will be able to evaluate their own video and audio samples together with colleagues and peers across Europe. Like DIALANG this project is also based on the descriptors of the Common European Framework of Reference for Languages: see Section 2.2.

Registered members of the WebCEF community can create, upload, rate and comment on video and audio samples. Using low-threshold technology, they can create their own video and audio samples to be assessed by a community of language learners across Europe.

WebCEF gives:

2.3 Adaptive testing: WebCAPE

Adaptive tests aim to assess language competence by asking questions based on the student's response to previous ones. If the student gets the answer right, a harder question is asked and vice-versa. This process claims to quickly find the level of a student in order to assist schools in placing the student in a suitable class. Brigham Young University in the US has pioneered web-based adaptive language testing for placement using its WebCAPE system. An example in Spanish is shown below:

WebCAPE

Figure 3: Screenshot, WebCAPE

For a full discussion of computer adaptive testing in language learning see Chalhoub-Deville (1999).

This form of testing may become much more common in the future. As noted earlier, Examination Boards are already experimenting with computer-based examining for public examinations such as GCSE and A level. It may soon be commonplace for schools to decide on setting of students by such methods. Teacher training departments may use this form of testing for assessment or self-assessment of trainees' language skills.

2.4 Other examples of tests

BBC Languages: Basic tests in a variety of languages to enable learners to assess their knowledge, plus a range of online courses at different levels: http://www.bbc.co.uk/languages/

The Council of Europe European Language Portfolio provides an attractive, portable and motivating way for individual learners to maintain an up-to-date record of their language learning experience and to track their changing levels of performance. It also contains can-do statements which help learners to assess their language skills: http://www.coe.int/t/dg4/linguistic/

Goethe-Verlag: Free tests in 24 languages and 552 language combinations: http://www.goethe-verlag.com/tests/

TOEFL: A large market clearly exists for testing English language competence and there is already a thriving computer-based testing industry. See the Test Of English as a Foreign Language website: http://www.toefl.org/

Versant Tests: Formerly known as the Spoken English Tests (SET), the Versant Tests are delivered over a telephone or on a computer and scored by computer. Once administered, numeric scores and performance levels that describe the test-taker's ability to understand and speak the selected language are generated within minutes and can be viewed online. Tests can be taken anytime, anywhere: http://www.versanttest.co.uk/

See the website of Linguanet Worldwide. This project has recently undergone expansion to incorporate an interface in a number of new languages and addresses in particular the needs of adult learners and independent learners. The site includes advice on ways of assessing and improving one's current ability in different languages (including links to websites that offer diagnostic and placement tests), communicating electronically with other language learners and finding appropriate resources. A substantial online catalogue of language learning resources is also being built up here: http://www.linguanet-worldwide.org/

See also Graham Davies's Favourite Websites page, which contains an extensive list of websites that offer interactive exercises and tests.


3. Using a word-processor for marking and giving feedback

Contents of Section 3

3.1 Introduction

Once upon a time students wrote in exercise books in blue pen and teachers collected these up and marked them in red pen. Although it is true that much learning occurs through considering mistakes, the problem with the above system was that the students did not always have any real incentive to look back at their mistakes and correct them; they were only interested in their overall mark.

As the story moves on, students began to use computers to type their work, print it and hand it in to their teacher who would then mark the work, again with a red pen. The main gain here is the eyesight of the teacher, who now marks printed text rather than handwriting, which is not always easy to decipher.

As word-processing software has developed, new tools have become available that make it possible for students to type their assignments and submit them to their teacher electronically - via the Web, email or over a local network. The teacher can now use tools in modern word-processing packages to mark the work electronically. The screenshots below (Figures 4, 5 and 6) show the process of marking a composition in Spanish.

Figure 4: Microsoft Word screenshot, showing student's original, uncorrected work

Figure 5 shows corrections that have been made using the Tools / Track Changes facility in Microsoft Word and the facility for inserting a comment using Insert / Comment.

Figure 5: Microsoft Word screenshot, showing teacher's corrections and a comment

The teacher can then return the work electronically to the student. The process does not end there, for there is now an incentive for the student to consider and accept the teacher's comments and then print a final, error-free product (Figure 6).

Figure 6: Microsoft Word screenshot, showing student's corrected work

3.2 How to operate on-screen marking

Marking using the revision facilities in Word

You have seen above a piece of written work in Spanish, with the teacher's corrections and one comment already entered. Have a look at the corrections and the comment. Now choose one of the following pieces of work by students of a range of languages to mark electronically or, if you have samples from your own students, use one of them.

Sample files

Click on any one of the following underlined titles to open a Word document and practise correcting it: Set up the following options in Word:
  1. Set user name (Tools/Options/User Information) - this is so that the comments appear with your name.
  2. Show Reviewing tool bar (View/Toolbars/Reviewing) - this adds the toolbar that you will need to add comments.
  3. Click on the Track Changes icon on the Reviewing toolbar
  4. Type the corrections directly into the text of the document. You will notice that any corrections you make will now show up with annotations.
  5. Highlight some text and then use the Insert Comment icon on the Reviewing toolbar to make longer comments.
Different approaches to marking:
  1. Simply correct the errors.
  2. Attach comments explaining key points.
  3. Attach comments pointing out the error, e.g. accent / tense / ending for the student to self-correct.
  4. Comment on the content, e.g. Très intéressant, pourquoi tu as acheté des chaussettes pour lui?

When you have marked the work, it can be sent back to the student by email or across the school's network.  The student can redraft the text incorporating your suggestions.

3.3 Marking software

As an alternative to using word-processing to mark students' work, there is specialist software that offers additional facilities, such as Markin. The website for this package describes it as follows:

Markin is a Windows program which runs on the teacher's computer. It can import a student's text for marking by pasting from the clipboard, or directly from a document file. Once the text has been imported, Markin provides a comprehensive set of tools enabling the teacher to mark and annotate the text. When marking is complete, Markin saves it as an XHTML document, in which the teacher's marks and annotations appear as coloured text. When the student opens this document in a web browser (such as Internet Explorer, Firefox, Safari or Chrome), they can click on the marks to reveal more detail about the nature of the teacher's annotation or comment. Markin can also export the marked work as a RTF file, which is more suitable for students who want to view it as a printout.

Markin

Figure 7: Screenshot from Markin

This software might be easier to use for those not experienced with word-processing and email since it is a ready-made solution.

3.4 Learning task

Responding to marked work

Imagine that you are a student whose work has been marked electronically, you receive the work back from your teacher and go through the following process. You can use any of the files that you corrected in the previous exercise (see above):

  1. Click on the Track Changes icon to switch that mode off .
  2. Use the Next Change icon on the Reviewing toolbar to find the first change.
  3. You can now accept, reject or edit the change - use the Accept Change or Reject Change icon on the Reviewing toolbar.
  4. Complete all the changes in this way.
  5. Now click on the Next Comment icon. After reading the comment you can make any changes and then delete the comment.

How did this seem from a student perspective? Does it engage the student with the corrections more than conventional paper and pen marking?

3.5 Automating comments using the autotext function

Marking using word-processing may seem slow at first and if you can't type quickly, it may take longer than conventional marking. One way to overcome this, especially if you frequently make the same comments, is to use the autotext function in Word or its equivalent in other packages. This is a feature which allows you to set up coded comment, e.g. SP to stand for Spelling Mistake. Of course, you may have trained your students to recognise such codes but using autotext allows you to type in the abbreviation SP and have the computer convert it to Spelling Mistake. Here's how to do it:

  1. Using your word-processor (Microsoft Word most versions) type the comment in full, e.g. Spelling Mistake.
  2. Highlight the text you have typed.
  3. Click on the Insert menu on the main toolbar, select Autotext and then New.
  4. Type in the code for the comment, e.g. SP.
  5. To test out and use the autotext function, type the code, e.g. SP, and then press the F3 key.

3.6 Further Reading

See Bishop (2004). Graham Bishop covers the use of the following tools that are available in Microsoft Word:


4. Reporting and recording students' progress

Contents of Section 4

4.1 Report Writing

4.1.1 Factors to take into account

What is ethically acceptable?

Automatic generation of student reports by computer carries with it obvious dangers in regard to quality assurance and security. For that reason, there is not much actual implementation of such schemes. However, computers can be used to support report writing, to provide templates, a bank of comments and automated comments without surrendering the teacher's normal responsibility for the task.

What will be possible within an overall school policy?

Where schools have adopted computerised reporting systems, it may not be possible to use standard features of word-processing as described below. Inevitably, any system allows only specified features.

What does the technology offer?

There are a number of facilities built into standard word-processing packages that can be exploited for report writing. These include:

  1. Autotext: this is described in Section 3.5. It is simply a method of speeding up the typing of comments that are used frequently by replacing the full comment with a code. For example, using this method, the phrase must try harder can be replaced by mth.
  2. Comment banks: this is simply a file containing lots of comments from which to choose or adapt in order to generate reports and, as such, offers a solution to report writer's block. See the Learning Task below.
  3. Forms and templates: these allow you to make a standard document that can be completed for each individual student. A template is just a document that has some pre-entered text and formatting, e.g. a report card. The teacher can simply open a new version of the document and then edit at will. A form is a variant of a template in which certain areas of the document are protected and cannot be edited. It can contain check-boxes, drop-down menus and so on. Forms and templates are very useful for speeding up the production of standard documents, e.g. progress cards.

4.1.2 Learning task: using a comment bank

You will find a cateogrised Comment Bank for Report Writing here, which was compiled by student teachers: Comment Bank.

See also:

You can experiment with using these resources to write imaginary reports. You can also make up your own comment bank using similar categories and adapting the comments for your own students. You can use autotext to automate the process of typing in the most commonly used comments.


4.2 Recording Progress

Recording progress electronically can help make the process more efficient and can make the information collected more versatile. The downside of this is security: Is the information stored as securely, at least, as it would be in a conventional markbook and is it protected from loss, erasure, etc? There is also the question of being able to use the information meaningfully. Some school-wide systems demand a lot of energy for the inputting of information that cannot readily be extracted by teachers. Sophisticated, custom-designed systems may be less helpful to use than a simple spreadsheet, for which the following examples apply.

4.2.1 Whole class records

Using a spreadsheet as an electronic mark book enables you to:

Excel

Figure 8: Screenshot from an electronic mark book

The file for this was created in Microsoft Excel and can be fully viewed by downloading the XLS file below. The file is also available in CSV format for loading into other spreadsheets. Note the bottom row of the window in the screenshot (Figure 8), which displays tabs for accessing different pages of the mark book for reading, speaking listening, writing and attendance. This facility is not accessible via the CSV file.

Sample markbook files:

Open the file and have a look at how it works.  Try some of the following:

  1. Add new students.
  2. Add a new sheet titled "form list" which can be printed out just as a blank list of names.
  3. Change the marks on one of the sheets and see how the totals and averages change automatically.
  4. Change the width of the columns.
  5. Write new comments on the comments sheet.

4.2.2 Learning task

Make a list of the advantages and disadvantages to you in your current teaching situation of an electronic markbook compared to a paper-based one. Which are you going to use in future? Note that even if you don't use a spreadsheet for a markbook, it still provides a very useful format for generating a paper based loose-leaf markbook.

4.2.3 Individual records

Keeping students' records is a facility available in various computer assisted language learning programs. Multimedia CD-ROM's often have this facility and it is usually possible then for the teacher to interrogate the records to check on students' progress. Some software incorporates an email facility through which student records are automatically forwarded to the teacher via email at the end of a session. This is available as an option with registered copies of Hot Potatoes. These individual records are important to track what students have been doing in CALL sessions since there is often no other record available.


5. Using the Web to manage assessment

Increasingly, modern languages departments have their own website or section at their school or college site. To what extent can a website be used to manage the assessment of language students? For example:

  1. The website can be used to provide students (and parents) with information about the syllabuses of courses, their aims, and the criteria for achieving their aims.
  2. The website can be used to provide information about examinations, such as contents of papers, grade criteria, tasks, sample questions.
  3. Various items can be made available as downloadable files, such as revision guides, vocabulary lists and sets of questions for oral examination topics.
  4. Links to websites that may be useful in the revision process can be listed, e.g. links to sites offering interactive exercises and tests appropriate to a particular examination..
  5. Model answers can be posted on the website, for example model compositions for the GCSE writing examination.
  6. If a website is available with student access via individual passwords, the site itself can be used by students to upload completed tasks and to download work that their teacher has marked.

Take a Hot Potatoes test on Computer Aided Assessment by clicking here: CAATest.htm. The answers can all be found in Module 4.1!

There is an increasing number of websites that offer distance learning materials, including whole courses delivered via the Web and email, using so-called Virtual Learning Environments (VLEs). See the following modules that deal in more detail with the Web, distance learning and VLEs:


6. Modern Language Aptitude Testing (MLAT)

It takes many hours of study and practice to acquire a reasonable level of proficiency in a foreign language. This represents a substantial investment for any organisation considering the selection of people for language training. It would therefore be useful to predict with reasonable accuracy to what extent an individual who had never studied a language before would benefit from attending a language course. Businesses often underestimate how long it takes to acquire a reasonable level of proficiency in a foreign language. The Council of Europe has calculated, for example, that it requires 180-200 learning hours to achieve Waystage (A2) level and 350-400 hours to achieve Threshold (B1) level with the Common European Framework of Reference for Languages scheme, Threshold level being the level at which the learner begins to communicate with a certain degree of confidence. See Section 2.2.

It can take a lot longer to reach an operational level for a particular profession. Airline pilots have to be able to communicate in English, and other staff employed in the airline industries are expected to be be proficient in foreign languages. An international railway operator requires its English-speaking train drivers to follow a 600-hour course (plus homework) in French to get them from scratch to operational level. The training includes a 3-week residential course in France. An important factor to bear in mind here is the safety factor. A driver who misunderstands an instruction in French while he is driving a train at 300 kilometres per hour is simply not acceptable as a member of the profession. Language tests in such professions are therefore rigorous, and it would obviously be useful to predict individuals' aptitude for learning a foreign language before sending them off to follow an expensive course.

Success in learning a foreign language depends on a number of basic aptitudes, for example:

  1. auditory
  2. linguistic
  3. memory

But factors other than aptitude often make a successful linguist, e.g. attitude to learning, a good learning style, interpersonal communication skills, intercultural competence, etc: see Skehan (1989) & Skehan (1998).

There is a useful collection of articles on Modern Language Aptitude Testing (MLAT) in Volume 9, Nos. 1 & 2 (1998) of Applied Language Learning, a journal published by the Defense Language Institute Foreign Language Center, Monterey, California, USA. Available at: http://www.dliflc.edu/archive/documents/all9_new.pdf

The website of Second Language Testing Inc (SLTI) has comprehensive information on Language Aptitude Testing.

The Language Aptitude Tests suite by Paul Meara, James Milton & Nuria Lorenzo-Dus (2001) consists of five discrete tests:

See also:
Language Aptitude Test for Entrance in Classics, Oxford University, 1995.


7. Plagiarism: detection, deterrence and avoidance

Most teachers have been confronted at some time with a piece of work that contains elements that have been copied from another source without due acknowledgement. Arguably, plagiarists are sophisticated cheats, but at secondary school language learning level they are more commonly naive incompetents who think that copying a large chunk of text from a website or CD-ROM is somehow going to get them a good mark. New technologies have made plagiarism easier, but they have also made its detection easier. Responding to plagiarism requires a two-tier strategy: how to catch the cheat and how to train learners to use source material appropriately. Academic plagiarism is the subject of a publication by Decoo & Colpaert (2002).

Detecting plagiarism

The advent of the information age and the course requirements of GCSE and A-Level examinations called for new skills for language students. It is important for students to be able to evaluate information, select appropriately and acknowledge sources that they have quoted. Providing examples and practice in this can be a proactive way of preventing naive plagiarism. However, there is a dearth of good advice for students, which reflects the relative novelty of the problem. Teachers will probably be able to detect readily any occurrence of plagiarism in foreign language work because they know their students' level of language, but it is also useful for them to know how to use a search engine to detect copying from websites and to find the precise site from which the copying has occurred. An easy strategy is to type in a section of the suspected text into the search line of a search engine such as Google, enclosing the text in speech marks. Five to ten words should be sufficient. With luck, the search engine will find the text on the website it was copied from. There are many tools that help detect plagiarism, for example:

See also:

When downloading materials or copying from another website, it is most important that you pay attention to copyright. Above all, don't assume that just because material is publicly available on the Web you can do whatever you like with it.

Copyright infringement is a growing problem, which we deal with in detail in our General guidelines on copyright. See also:


Bibliography and references

Bangs P. (2003) "Engaging the learner - how to author for best feedback". In Felix U. (ed.) Language learning online: towards best practice, Lisse: Swets & Zeitlinger.

Bishop G. (2004) "First steps towards electronic marking of language assignments", Language Learning Journal (ALL) 29: 42-46.

Brown J. (1997) "Computers in language testing: present research and some future predictions", Language Learning and Technology 1, 1: 44-59. Available at: http://llt.msu.edu/vol1num1/brown/default.html

Chalhoub-Deville M. (ed.) (1999) Issues in computer adaptive testing of reading proficiency, Cambridge: Cambridge University Press.

Chalhoub-Deville M. (2001) "Language testing and technology: past and future", Language Learning & Technology 5, 2: 95-98. Available at: http://llt.msu.edu/vol5num2/deville/default.html

Council of Europe (2001) Common European Framework of Reference for Languages, Cambridge: Cambridge University Press. ISBNs: Hardback 0521803136, Paperback 0521005310. The CEFR document can be downloaded from the Council of Europe's CEFR website. A Self-assessment Grid relating to the CEFR is available as a Word DOC file. See also the Council of Europe's Language Policy Division website.

Decoo W. & Colpaert J. (2002) Crisis on campus: confronting academic misconduct, MIT Press / Bradford Books.

Department for Education and Skills - now known as the Department for Children, Schools and Families (2002): Languages for all: languages for life - a strategy for England.

Dunkel P. (1999) "Considerations in developing or using second / foreign language proficiency computer-adaptive tests", Language Learning & Technology 2, 2: 77-93. Available at at: http://llt.msu.edu/vol2num2/article4/

Godwin-Jones R. (2001) "Language testing tools and technologies", Language Learning & Technology 5, 2: 8-12. Available at: http://llt.msu.edu/vol5num2/emerging/default.html

Heift T. & Schulze M. (eds.) (2003) Error diagnosis and error correction in CALL, CALICO Journal Special Issue 20, 3.

Krashen S. (1985) The input hypothesis: issues and implications, London: Longman.

Language Learning and Technology 5, 2 (2001) Special Issue on Computer Assisted Language Testing. Available at: http://llt.msu.edu/vol5num3/default.html

Nielsen J. (1995) Multimedia and hypertext: the Internet and beyond, Academic Press: Boston.

Nielsen J (1997) Be succinct! Writing for the Web, Alertbox, 15 March 1997.

Nielsen J. (1998) Fighting Linkrot, Alertbox, 14 June 1998.

Nielsen J. (2010) iPad and Kindle reading speeds, Alertbox, 2 July 2010.

Meara P., Milton J. & Lorenzo-Dus N. (2001) Language Aptitude Tests (book and CD-ROM), Newbury: Express Publishing. Search for the title "Language Aptitude Tests" in the Catalogue section at the Express Publishing website.

Pachler N. & Byrom K. (1999) Assessment of and through ICT . In Leask M. & Pachler N. (eds.) Learning to teach using ICT in the secondary school, London: Routledge .

Rendall H. (2001) Developing a sense of gender in French: ICT integration at initial learner level. In Atkinson T (ed.) Reflections on ICT, London: CILT.

Roever C. (2001) "Web-based language testing", Language Learning and Technology 5, 2: 84-94. Available at: http://llt.msu.edu/vol5num2/roever/default.html

Skehan P. (1989) Individual differences in second-language learning, London: Edward Arnold.

Skehan P. (1998) A cognitive approach to language learning, Oxford: OUP.

Winke P. & MacGregor D. (2001) Review of Hot Potatoes, Language Learning and Technology 5, 2: 28-33. Available at: http://llt.msu.edu/vol5num2/review3/default.html


Websites

CAA Centre website: A site that has been designed to provide general information and guidance on the use of Computer Aided Assessment (CAA) in higher education.

Council of Europe, Common European Framework of Reference for Languages

Graham Davies's Favourite Websites: An extensive list of websites relating to language learning and teaching, including many links to websites that offer interactive exercises and tests.

Language Testing Resources: A comprehensive and very interesting set of Web pages maintained by Glenn Fulcher at the University of Leicester. Extensive information about all aspects of language testing.


Feedback and blog

If you wish to send us feedback on any aspect of the ICT4LT website, use our online Feedback Form or visit the ICT4LT blog.

ICT4LT_at_Blogger

The Feedback Form and a link to the ICT4LT blog can be found at the bottom of every page at the ICT4LT site.


Document last updated 19 April 2012. This page is maintained by Graham Davies.

Please cite this Web page as:
Atkinson T. & Davies G. (2012) Computer Aided Assessment (CAA) and language learning. Module 4.1 in Davies G. (ed.) Information and Communications Technology for Language Teachers (ICT4LT), Slough, Thames Valley University [Online]. Available at: http://www.ict4lt.org/en/en_mod4-1.htm [Accessed DD Month YYYY].

© Sarah Davies in association with MDM creative. This work is licensed under a
Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Creative Commons Licence