Title: Building Resources Through Electronic Peer Review: The Expertiza Platform
1Building Resources Through Electronic Peer
ReviewThe Expertiza Platform
- Edward F. Gehringer
- Dept. of Electrical Computer Engineering
- Dept. of Computer Science
- North Carolina State University
- efg_at_ncsu.edu
- A shorter version of this presentation can be
found at http//research.csc.ncsu.edu/efg/expertiz
a/presentations/Expertiza-short.htm
2Outline
- Introduction
- Rationale
- The Expertiza platform
3Introduction
- Electronic peer review is students reviewing
other students work over the Web. - Building resources through electronic peer review
- gets students working together to improve others
learning experiences, - helps them learn, by performing tasks that are
similar to real-world responsibilities, - gives them experience in writing their ideas up
for an audience of their peers, - allows each cohort to stand on the shoulders of
students in earlier classes.
4Homework, traditionally
- Heretofore, the homework process has served only
for students to demonstrate mastery of the
subject. - Every student does the same thingredundant
effort. - Work is graded and thrown away, never benefiting
anyone but the student who did it. - Now the best work can be automatically
published, to help others learn.
5Why Electronic peer review?
- Only electronic peer review can produce these
benefits. - Overhead of doing it manually is too great.
- Assuring that enough students do each piece of
work - (otherwise, none may be good enough).
- Grading the work, which is time consuming
students submit solutions to different problems. - Assembling the best submissions into a single
unit, and getting permission to publish them for
others.
6Why Electronic peer review, cont.
- Having the resources doesnt help unless the
instructor can determine which are best. - The student reviews show the instructor exactly
where to look for good materials. - Manual implementation would consume faculty/staff
time, instead of saving it.
7Outline
- Introduction
- Rationale
- The Expertiza platform
8The Rationale
- Improve student learning
- Improve teaching
- Better utilize resources
9Improving student learning
- The Expertiza platform improves student learning
in these ways - Integrates active and cooperative learning
- Extends active learning to out-of-classroom
activities and distance education - Discourages plagiarism
10Integrating active and cooperative learning
- Active learning allows students to take
responsibility for their own learning. - E.g., short in-class exercises with one groups
reporting to whole class. - Active learning keeps students more engaged and
interested - Active learning encourages nontraditional
students to stick with a program - Larger of milestones helps those with poorer
study habits - Double-blind electronic review with Expertiza
helps overcome biases against women,
underrepresented minorities.
11Extending active learning outside the classroom
and to DE
- Provides a structure for out-of-class activities
that encourages persistence and helps avoid
distraction - Students give each other feedback on how to
improve. - The larger number of deadlines and milestones
makes it harder to be a slacker. - Diminishes difficulties of finding students with
compatible schedules - Asynchronous out-of-class collaboration is now
possible.
12Electronic peer review allows active learning and
DE to mix
- DE has been a roadblock to the use of active
learning students viewing lectures remotely can
work only by themselves. - Online discussions can help, but they are no
panacea. - The instructor or TA needs to monitor them,
interact with students, grade student
submissions. - Time-consuming Needs to be done on a timely
basis, several times per week.
13Discouraging plagiarism
- Multiple deadlines and milestones make it
impossible to submit a finished product obtained
from an external source. - Student feedback on peers contributions makes it
easy to catch cheaters.
14The Rationale
- Improve student learning
- Improve teaching
- Better utilize resources
15Improving teaching
- The Expertiza platform improves teaching in these
ways - Helps create weekly mastery quizzes
- Increases the supply of examples/homework
problems/test questions - Focuses students on explaining/understanding the
concepts that are hardest to master - Helps courses keep up with technology
16Periodic mastery quizzes
- Students can be assigned to create
machine-scorable questions over lecture material. - Peer review determines which questions are the
best. - Having a supply of such questions makes it
possible to give daily/weekly quizzes via an
online testing system, such as Webassign, WebCT,
LON-CAPA, or Mallard.
17Increasing the supply of good examples and
questions
- Students can be assigned to make up a
peer-reviewed homework or test question. - Increases the supply of worked-out problems
- Improves homework and/or exams, by adding to the
supply of good textbook questions - (This is of interest to textbook publishers,
too!) - Some questions can be used as examples in class
or given to students for self-study.
18Focusing students on mastering the hardest
concepts
- Students can be assigned to create materials that
address subjects that are difficult to master. - Annotate lecture notes with references to
background material from the Web. - Identify the hardest concept from Lecture k and
write an example that clearly explains it. - Improve the diagram that illustrates x.
- Add animation to the PowerPoint slides from this
lecture. - All of these, of course, can be peer-reviewed.
19Helping courses keep up with technology
- Students can be assigned to research emerging
areas of technology. - E.g., how communication is managed in a new
parallel computer architecture. - They can be asked to create lectures, or slides,
describing the material, that would fit in with
the course they are taking.
20The Rationale
- Improve student learning
- Improve teaching
- Better utilize resources
21How Expertiza improves resource utilization
- Some work is peer-graded, so teaching assistants
can spend more time working with students and
less time grading. - Not only does this help the students, but TAs
learn skills that will benefit them in the
workplace. - Having inadequate TA support no longer limits the
amount and kinds (e.g., design problems) of
homework that can be assigned. - Students rely more on their peers for help, less
on the course staff.
22How Expertiza improves resource utilization, cont.
- Makes teaching large classes an advantage!
- Large classes can produce more and better
resources. - The usual disadvantages of large classes (lack of
engagement and personal attention) are mitigated
by - active learning,
- better materials,
- giving TAs more time to work with students
23Outline
- Introduction
- Rationale
- The Expertiza platform
24The Expertiza platform
- Can accept submissions in almost any format.
- Web is a familiar interface.
- Web-creation skills are important to students.
- Web interface allows use in DE.
- Facilitates the production of Web resources.
25Overview
- The implementation
- The review cycle
- How peer grading has been used
- Student reaction
- The Expertiza platform
26The Implementation PG
- Students develop homework in the form of one or
more Web pages. - Students submit their pages to the system.
- The system copies them to a new Web address,
concealing the submitters identity. - Reviewers are assigned semiautomatically.
- Reviewers and authors communicate via a shared
Web page. - Reviewers assign grades, and the system averages
them.
27The PG Login Page
28Choosing an Assignment
29Selecting Submission to Review
30Submitting and Reviewing
- A student logging in has a choice of whether to
submit or review. - If submitting
- can use a browser to select a file.
- can submit one file at a time, or
- a whole network of pages in a single Zip file.
- If reviewing
- is presented with a set of pages to review.
- can click on one and type in comments.
31Reviewing a Submission
- Review is based on a rubric.
- A numeric score is assigned to several questions.
- Grade is calculated from these scores.
- Ample opportunity for comments.
32Overview
- The implementation
- The review cycle
- How peer grading has been used
- Student reaction
- The Expertiza platform
33Submit-Review-Publish Cycle
- Signup phase. A limited number of students
allowed to sign up for each choice. - Initial feedback phase. Students given 27 days
to make initial comments. - Resubmission phase. 27 days to revise work in
response to reviewer comments. - Grading phase. 37 days to make final comments
and assign scores. - Review of review phase. Students review each
others reviews. - Web-publishing phase. PG creates a Web page with
the best assignment in each category.
34Overview
- The implementation
- The review cycle
- How peer review has been used
- Student reaction
- The Expertiza platform
35How Peer Review Has Been Used
- Researching lecture material.
- Find links related to each lecture.
- Annotating on-line lecture notes.
- Writing research papers.
- Reviewing papers from the literature.
- Making up homework problems.
- Making up machine-scorable questions.
- Weekly reviews.
36Annotating a Lecture
- Students electronically sign up to review a
particular lecture, - then add hyper-links to instructors on-line
notes.
37Research Papers
- Can include hyperlinks to Web documents.
- Different students can sign up to write on
different topics.
38Madeup Problems
39Choosing Assignment Types
40Producing Resources
- A divide and conquer approach to large
projects. - Lecture annotations for an entire semester.
- Machine-scorable questions for each lecture.
- Madeup problems for the Computer Architecture
Course Database (cf. WCAE 1998, WCAE 2000).
41Overview
- The implementation
- The review cycle
- How peer grading has been used
- Student reaction
- The Expertiza platform
42Student Reaction
- Students have reacted quite positively to peer
review. - When asked to rate on a scale of 1 to 5, Peer
review is helpful to the learning process, - five classes rated it 3.41 to 4.24, with the
highest score given by one of the classes that
did the most peer-reviewed assignments. - By a score of 3.9, students said that reviews of
reviews motivated them to do careful reviews.
43Overview
- The implementation
- The review cycle
- How peer grading has been used
- Student reaction
- The Expertiza platform
44The Expertiza Platform
- PG is one of three components of the Expertiza
platform. - Shimmer for signing up for assignments,
allowing a task to be divided into individual
parts - PG for peer-reviewing work
- Conoscenza a Web-based database that makes the
best student work accessible to registered users
over the Web.
45The Expertiza Platform (cont.)
- Using Expertiza, it is possible
- to organize classes efficiently
- to produce resources that will
- enhance the educational experience of others.
PG
Conoscenza
Shimmer