CLIPP Board

Turnitin – turnitup!

Turnitin; our text matching tool of choice are currently beta testing two new additions to their ever growing suite of features:

First is the direct integration – useful for those (like us) who have Turnitin fully integrated with their VLE.    We will be joining the beta testing process in the near future. I am attending a demonstration of this next week, and will report back soon.

Secondly is the beta release of translated paraphrasing. This relatively new method of converting text into different languages – then back into English, can be now be “checked” to detect this practice.

Training news

Turnitin are also  running a series awareness sessions running during February and March, covering a wide range of academic misconduct and plagiarism topics. Most are free, and can be attended via an online webinar. Find out more and book onto these here: http://community.turnitin.com/events/event_list.asp

Issues and solutions regarding student assessment

This blog is my feedback from the Student Assessment and Classification Working Group (SACWG) meeting, held in London on 24 Nov 2011. SACWG is an informal group of UK academics and administrators who have a shared interest HE assessment, including assessment criteria, grading, and the honours degree classification. The meeting was attended by about 50 people from various universities, the HEA and QAA. Aston was represented by Professor Alison Halstead (invited speaker) and me (Stuart Wallis, delegate). I attended, because I know that assessment & feedback is such a big issue in HE, and I wanted to hear about other universities’ problems and solutions.

The day consisted of presentations, discussion groups and a question-and-answer session. The presenters were:

• Dr. Marie Stowell (Director of Quality & Educational Development, University of Worcester)
• Professor Alison Halstead (Aston’s Pro Vice Chancellor of Learning & Teaching Innovation)
• Professor Chris Rust (Associate Dean for Academic Policy, Oxford Brookes University)
• Professor Paul Hyland (Head of Teaching & Learning, Bath Spa University)

Marie Stowell set the context, by stressing the importance of good assessment and feedback (e.g. for student retention and completion) and by describing some of the challenges facing those who assess. Alison Halstead painted a very positive picture of how we are addressing the challenges here at Aston. Her PowerPoint slides are below. Edited highlights include: personal response systems (Optivote) being used by Audiology and Psychology to check students’ understanding in class, and the consequent increase of student engagement; the use of lecture capture (Aston Replay) – students like being able to review lectures, and lecturers do not report reduced attendance; the identification of the reasons for staff resisting e-marking and the consequent provision of large computer screens to staff in Psychology. In response to a question about how to manage the adoption of e-marking, Alison disagreed with another panel member who suggested a top-down decree, and stated that if management can’t demonstrate the benefits from within the discipline, then they should not be forcing a change. As I listened, I felt glad to be working at Aston.

Here are some of the issues/tensions/problems that were discussed by the presenters and delegates throughout the day, and possible solutions:

Issue: A lack of coherence across a programme, due to the tendency for lecturers to design module assessments in isolation.
Solution: Move away from assessment of a module towards assessment of a programme. For example, Coventry Business School uses an assessment that is marked by 3 lecturers, each of which is looking at different aspects of it. But, the increase of modules size (e.g. 40 credits) means that a students who fails it ends up being re-assessed on more material than if the module was small (10 credits).

Issue: How do we ensure good quality assessment & feedback when class sizes are increasing?
Solution: Have fewer summative assessments. This may release time for more formative assessments, which can be ‘quick and dirty’.

Issue: How can we get students to engage with peer/self assessment, if they think that assessment is the lecturer’s job?
Solution: Spell out the benefits of being a peer-assessor, and spell out that these skills are graduate attributes; give training on how to assess; make it a compulsory part of the module.

Issue: How do we get students engaged with formative assessment that carries no marks?
Solution: Specify that it is a stage towards a summative assessment (“Here’s your chance to practice before the real thing”); make it interesting.

Issue: How can early re-assessment be managed?
Solution: The University of Worcester has recently moved re-assessments from August/September to June. The idea was that lecturers are more likely to be available to help students (and not at conferences), and that this would lead to better pass rates. At Worcester it did, but at another university it did not. Also, this system placed increased pressure on staff during June.

Issue: How do we encourage students to be co-producers of learning, rather than consumers of teaching?
Solution: Manage their expectations by providing clarity about what is being offered, e.g. through a student charter.

Issue: How can we encourage all (rather than just some) of the students to read the feedback comments on their work?
Solution: Initially provide the feedback without the mark, and only supply the mark after a response has been made to the feedback (e.g. “What I have learnt from these comments is…”); if supplying audio feedback, embed the mark at an unexpected place in the audio (not at the beginning or end).

My overall impression is that it was a very useful day. I particularly enjoyed the opportunity to network, and by meeting people from other universities I realised that Aston really is a refreshingly responsive place to work.
If you would like to know more about the SACWG event, or wish to respond to any of these issues, please post a comment on this blog, or contact Stuart on extension 5171 or at s.a.wallis2@aston.ac.uk.

Alison Halstead’s PowerPoint slides

electronic feedback

In this short video CLIPP, Dr Stuart Wallis explains how he and his students have benefitted from the GradeMark tool, when using it to assess work and give timely feedback.

See also my notes on the  presentation  by Dr Cath Ellis we posted earlier in the year.

Supporting learning

Dr Matthew Olczack explains how he used, and  subsequently researched web based resources to enhance traditional teaching and learning methods.  The research is of particular interest as Dr Olczack investigates aspects of self selection bias.  The overall results were very favourable, both in terms of the postive impact upon students learning, and their overall achievements.

How to double your student exam scores (yes, really!)

On 13 May, Deslauriers et al (2011) published a paper in Science (impact factor 29.7) where they demonstrated the hugely beneficial effect of a teaching method called ‘deliberate practice’ (see chart below).

They employed this method for 3 × 50 mins in week 12 of a module – each of the previous 11 weeks had contained 3 × 50 mins of traditional lectures. When contrasted against a cohort that had received traditional lectures in week 12, the subsequent scores in a 15-20 minute MCQ class test were amazing… particularly as deliberate practice was taught by relatively inexperienced teaching assistants, whereas the traditional lectures were given by a highly experienced lecturer who often attained high student feedback.

What is deliberate practice?

It is an active learning approach, which uses the whole of the lecture time for problem-centred learning, and where the students frequently receive feedback.

How did Deslauriers et al implement deliberate practice in their teaching sessions?

One instructor and one assistant ran the class of 271 students. Before each session, the students were instructed to read a 3-4 page reading and answer a related on-line true-false quiz. At the start of the first teaching session, the students were told the rationale for the use of deliberate practice (and supporting research was cited).

A typical session schedule was:

An MCQ question was posed, and students discussed the answer with other students (2-3 mins).

The students gave their answers using ‘clicker’ handsets (like Optivote), and the instructor gave feedback to the whole cohort (4-6 mins).

This process was repeated with another 5 different MCQs.

Sometimes, an MCQ was presented twice in succession, presumably so that the effect of the instructor feedback and 2^nd student discussion could be measured. Note that this seems to suggest that the instructor feedback did not always give the correct answer.

Twice in each teaching session, there was a small-group active learning task (6 mins), followed by instructor feedback (4-6 mins).

How long did it take to prepare each 50-minute teaching session?

Including the pre-session reading quiz, 20 hours at first, dropping to 10 hours for the 3^rd session. But the authors estimate that this would fall to 5 hours with practice. Note that these times include piloting the materials with one or two students, and subsequent editing.

How was deliberate practice received by the students?

Deslauriers et al surveyed the students and found that they enjoyed it more and thought that they learnt more than in traditional lectures. Interestingly, students thought that they learnt much more in class than from the pre-reading with quiz, even though the majority (all?) of the new information came from the pre-reading.

What conclusions can be drawn?

An active learning approach with frequent feedback…

• is better than a traditional lecture.
• can be applied with large cohorts.
• does not need many teaching assistants.
• does not take a lot longer to prepare (when practiced).

Where can I find out more?

A hard copy of the Deslauriers et al paper is in Aton’s library (but is not available electronically yet). The supplementary material is available at http://www.sciencemag.org/content/332/6031/862/suppl/DC1.

Reference

Deslauriers, L., E., Schelew, E. & Wieman, C. (2011) Improved Learning in a Large-Enrollment Physics Class. Science 332(6031): 862-864.

GradeMark – nose to tail

The presentation given by Dr Cath Ellis here at Aston last week really illustrated how the GradeMark tool (part of Turnitin) can be used as an effective feedback and diagnostic tool.  Before I delve into the way Dr Ellis has been using GradeMark in quite innovative ways, I should first mention the overarching e-marking/e-submission strategies at Huddersfield. I’ll also share with you our own Aston University guidance notes for those considering larger scale adoption of using Turnitin, to complement exisitng academic misconduct procedures.

The University of Huddersfield is four years into an e-submission policy that allows for paper submission alongside those who want to adopt e-submission methods. They chose Turnitin to fulfil the needs of both e-submission and text matching.  This thoughtful rollout process caters for the nurture and support of early adopters, and also supports for those who still want (and need) to use paper.  This parallel process ensures that organic (peer driven) growth is supported, whilst ensuring no one is alienated or marginalised.   The new student body is also pivotal in the process of pushing for greater use of e-marking and e-feedback.  A print contract with a local firm means that scripts can be bulk downloaded from Turnitin and sent to the printer for paper copies within 24 hours – at a competitive cost. Therefore the onus is not placed on the student to print out paper versions. This means that everyone can use Turnitin to submit, and then outsource the printed versions for those whom still prefer to mark on paper. The print process is still an overhead to consider, but their outsourcing contract does offer a more flexible and innovative approach.

Just as important as this outsourced print contract, is the streamlined workflow process. The entire process is being continually monitored and refined to ensure that administrative bottlenecks can be gradually eliminated from the entire submission /marking process, leaving support staff to concentrate on other tasks.  Dr Ellis refers to this important factor as a “nose to tail workflow.”
So we have a number of important facets pivotal to the large scale adoption;
1)    A university wide e-submission  policy
2)    Continual refinement of administrative workflow processes
3)    Methods that ensure that e-submission and paper marking can happily coexist
4)    Early adopters that can share the vision
5)    Pressure from the students to adopt more flexible practices

Dr Ellis demonstrated the use of the (drag and drop) GradeMark rubrics as an efficient way of adding comments to student scripts, which helps to ensure a consistent quality of feedback for all. Sometimes she used a fairly lengthy comment saved in her own list of rubrics, thus eliminating the need to rewrite feedback, common to particular mistakes.  But the most impressive use of GradeMark was the ability to download and analyse the comments left on entire cohort’s scripts to diagnose recurring problems.

This diagnostic aspect (see graphs in presentation) of analysing the frequency of rubrics enabled Dr Ellis to plan teaching interventions based upon a higher frequencies of for example a particular grammatical misinterpretation. By downloading the frequencies of rubrics and displaying them as simple histograms (see above), Dr Ellis was able to quickly pinpoint recurring problems across and between academic years.

“..go into your inbox and click on GradeMark Report which is next to the roster synch button. You can then drill down to the grademarks and export the data as excel files.”

The last tip Dr Ellis demonstrated was the shortcut method (see above) to overlay the originality report over the GradeMark screen. This means that whilst you are marking online and providing feedback, you can be checking the script for text matches highlighted – all on the same screen. Time saving indeed!

ae-pig 02

During our second e-portoflio interest group, the vision and tangible outputs of our fledgling community of practice were discussed. The following themes were agreed;

•    collaboratively create assessment rubrics for e-portfolios, both process and product,
•    Identify key trends and drivers for UK research based HEIs,
•    Identify and share practice across Aston in respect to teaching reflection, electronic feedback, and the pragmatics of assessing large cohorts <link to process review>,
•    How to scaffold reflective practice for all students,
•    Logisitcs of marking large co-horts (assessment design), links to current process review outputs.
•    Electronic feedback (mp3) methodologies, the why and the how
•    Facilitating online collaboration
•    Ensure we achieve adequate SMT support, and targeted promotion the CoP to potential members (i.e. placement officers).
•    Effective support procedures for both staff and students

Read more

Piloting Practice

We recently ran another in a series of effective practice days here @ Aston where staff disseminated their findings on small scale funded [pilot] projects.
The day was filled with a series of some really innovative uses of technology and full of recommendations and much food for thought.  The first I watched was by Christine Hirsch who talked about their use of the e-portfolio tool PebblePAd with MPharmacy students. The study concentrated on what types of support was required to implement such a learning platform and how the students used the tools.  Christine referred to a recent review of e-portfolios by Birmingham University which concluded that e-portfolio tools help promote reflection and PDP.
The aim was to replace paper based systems of reflective evidence submitted at the end of the periods of study with a much more responsive and immediate e-portfolio system.  The “Ability” And “Action plan” asset types were used that allowed students to attach evidence into their personal competency audits.  Students were given some [not enough] F2F instruction together with supporting  screen-casts of the PPAD interface.
The results we very favourable, resulting in some media rich web ‘folios evidencing some truly reflective writing.  Further recommendations were made relating to improving staff and student support for the software, and having a local (PPAD) expert at hand to help. There were also difficulties with the level (quality) of reflection submitted for evidence.  Moving from description to reflection is always difficult. I’d suggest using something like the Driscoll reflective framework (what?, so what?, then what?) as a mode of scaffolding learning for (all??) students requiring to submit deeper reflective evidence. This could be achieved by using a standard template – issued from the (shared) gateway.

Read more

e-maths Versus e-CAF

In my very first few weeks here at Aston , I attended two project presentations covering similar themes. Namely effective assessment and feedback . Nothing new I hear you yell! But, wait for it….. these are Aston University born and bred e-assessment “systems”.
They were both funded from various teaching and learning funds to develop web based platforms to address specific requirements, which only a slick e-assessment system could cope with.  Interested? Then please read on.
e-CAF
The first system is called e-CAF – which has its own Aston web page, plus a separate website: http://ecaf.aston.ac.uk/ – with a demo.  If you take a look at the blurb there you’ll see it’s already 4 years old, and needs just a little more work to pass muster.   As the web page says…
“The e-CAF system is a web-based coursework assessment system. Its aim is to address the issues in providing timely, informative and consistent assessment and feedback to learners.”
The presentation covered various technical aspects of its development (which was very impressive) and also covered how it has been used at both formative and summative assessment levels.  It appears to have been developed to assess uploaded student written computer scripts.  So the intention (I believe) was to ease the burden of marking lines and lines of code, but also be able to annotate student scripts and allow students to see this feedback in a much timelier manner. So it’s a win<>win for both staff and students.
The admin side (interface only) requires some tweaking, but it allows different kinds /sizes/groups of cohorts, and also allows you to create your own marking schemes complete with weightings, etc. Being server based, it can link with existing password database systems (i.e. LDAP), or cope with bulk registrations (Excel spread sheets/CSV files). So it’s got potential, and let’s hope the final polish will ensure others will pick up on this and utilise it in other subject areas.

e-maths
The second e-assessment tool being developed is a much younger beast, only in the early stages of development. But the demo I witnessed from an interface perspective (see top image) and from a functionality perspective shows great promise. This system has been funded by some LLN money by the Learning Development Centre (Ellen Pope & Mike Peters). It reminded me of a “lite” version of Question Mark Perception, but I was informed that this system will be able to handle maths equations (unlike QMark).  And indeed that’s why it was constructed – as a diagnostic tool to assist foundation degree students.
Again it’s a web based system split in two distinct parts;
1)    Pre-entry diagnostic tools to aid transition of students between learning/award levels.
2)    A learning tool – packed with various quiz and feedback types – (very much like Qmark)
The fully fledged version will be able to handle a wider range of feedback and assessment methods, and because it’s browser based it requires the mathplayer plug-in to run.  So expect a launch date soon once the final stages have been tested.
So there’s two very good bespoke  “e-assessment” tools available to all staff to use.  Not forgetting our very capable Blackboard suite of “quiz” tools, plus the online marking feedback tools offered by Turnintin – both very capable e-systems straight off  the shelf.
It does appear to indicate in my short time here that there is a raised feeling that some of these “e-enabled” systems can improve the assessment and vital feedback process by virtue of their ability to be timely and web based. Which means giving feedback can be instant, and can also be done remotely.  Now the next thing to investigate is the peer assessment tool part of Turnitin! But that’s for another time !O)

Text matching = Turnitin

CLIPP are pleased to announce two workshops aimed at raising awareness of the popular text matching tool Turnitin.
The first workshop aimed at training support staff (sub administrators) – is already full, but we can still accept bookings if you are involved in that role, or soon to be? Monday 28th June morning session. Book here :

http://staffdev.aston.ac.uk/bookings/fullCourse.aspx?courseID=312

The second workshop is aimed at anyone interested in this text matching software (subsequently used to help checking for plagiarism and collusion).  The second workshop is being held on Monday 28th June from 1 > 3:30 pm.  Please find more details and book onto this workshop here:
http://staffdev.aston.ac.uk/bookings/fullCourse.aspx?courseID=313

Note:  we are covering the online marking features and peer assessment features of Turnitin during this workshop.
For those of you who are unsure of exactly what Turnitin can offer to improve teaching and learning, please go to : http://www.submit.ac.uk or http://www.plagiarismadvice.org
For an extensive list of effective practice across the UK on the use of such software please read the links/case studies/research papers, etc in my wiki page (three years in the making!):

• http://wiki.rscwmsystems.org.uk/index.php/Plagiarism ,
  …and also please browse my shard bookmarks on the topic :
• http://delicious.com/deliboy/plagiarism