Technical Note - Assessment System

Summary

The ECS Assessment System allows staff and tutors to mark and provide comments for submissions that students made using the Submission System. This document illustrates how the assessment system works. Please report any problems with this system to bugs@ecs.vuw.ac.nz

Details

The Assessment System is available at http://apps.ecs.vuw.ac.nz/cgi-bin/marking.

This application provides several functions for staff and tutors to support online assessment submission. Some of the functions provided for staff:

  • Add and configure online assessments to make them available on the online Submission System.
  • Display a list of all submissions as well as summary submission information for the assessment.
  • Display students that did not submit.
  • Approve late submissions for students.
  • Show marks for the assessment.
  • Upload and download marks (available from the assessment marks page)
  • Allocate tutors to mark a submission.
  • Display late submission information.
  • Approve tutor comments and marks so that students can view it on studentmarks.
  • View a calendar of all assessments. (There is also a full Assignment Calendar, updated automatically from the Assessment System.)

The following functions are mainly for tutors:

  • Take an submission to mark.
  • View the submission files in a browser or download it (individually or as a combined zip file).
  • Upload a file to the submission's marking directory.
  • Add and edit marks and comments.
  • View the marking guide for the assessment.
  • Flag a problem with a submission and add administrative comments.

How to add a new assessment

Staff can set up an online assessment by going to http://apps.ecs.vuw.ac.nz/cgi-bin/marking and following these steps:

  1. Select the course from the list by clicking on the course name.
  2. On the course page the current assessments will be listed. Click on the "Add new assessment" link at the top.
  3. Fill in the assessment name and optionally add a due date, due time and submission message. Click on save to complete the setup.

The new assessment will immediately be available in the submission system. If the course you want to configure is not on the initial course list, please submit a job to jobs@ecs.vuw.ac.nz.

How to add marking items to assessments

From Trimester 2 2016 marking items need to be setup before tutors can mark an assessment. This is done by following these steps:

  1. Go to the assessment's Config page and then click on the Marking tab at the top.
  2. Click on 'Add new marking item'
  3. Add the item's name (eg. 'Question 1', or 'Core', etc.)
  4. Optionally add a maximum mark. The tutor's mark will be tested to not exceed this value. By default the allowed values are A-F with an option plus or minus sign and values ranging from -100 to 100.
  5. Optionally set an item as an admin mark. These items are used to import marks and are not displayed to tutors. They can also be used as marks that you do not want the tutor to enter, eg Penalty. Students will see these marks on their studentmarks page.

Complete these steps for each marking item. The tutors will see a list of these items on their marking page and can only complete an assessment's marking when all marking items have a value.

How to add a final mark calculation to an assessment

Final marks can be automatically calculated and displayed to students. To configure this do the following:

  1. First setup the marking items as explained in the previous section.
  2. Next enter a simple arithmetic formula in the 'Final mark calculation' field.

Please note that only approved marks are used to calculate final marks. The minimum calculated mark is automatically limited to 0 so the formula does not need to compensate for this. The following functions can be used in the formula: floor, ceil, round, min and max. (Please note: these functions must be lower-case)

Example formulas:

  • Average of Item 1 and Item 2: (M1 + M2)/2
  • Turn a single mark out of 30 into a percentage: M1 / 30 * 100
  • Limit an assessment with extra marks to 100 maximum: min(M1+M2+M3,100)
  • Add Item 1 (out of 15) as 70% and Item 2 (out of 12) as 30% of the total: (M1/15)*70+(M2/12)*30
  • Deduct 20% absolute for each day late from three marks (out of 100 each): (M1+M2+M3)/3 - ceil(M0)*20
  • Deduct 10% of total for each day late from three marks (that sum to 100): (M1+M2+M3)*(1-ceil(M0)*.1)

Configuring a marking guideline

The marking guideline is displayed to tutors on each submission's marking page. To configure it enter the marking guide using plain text. By default a tutor needs to add at least one comment to complete marking of a submission. To remove this requirement add the text 'no comments needed' anywhere on the marksheet

How to edit an existing assessment

Go to http://apps.ecs.vuw.ac.nz/cgi-bin/marking and following these steps:

  1. Select the course from the list by clicking on the course name.
  2. On the course page the current assessments will be listed. Click on the wrench icon in the Config column of the assessment you want to change.
  3. Change any of the editable fields and click save to complete. (To change the assessment name requires a job to jobs@ecs.vuw.ac.nz)

Automatic Late Penalties

The Assessment System can automatically calculated late penalties for submissions. This is useful for courses that allow a grace period (a number of allowed late days before penalties will be applied) for late submissions during the whole trimester.

When activated the number of late days will be calculated for each submission and if late a 'Late Penalty Days' mark will be added to the student's marks. This will also be displayed to students on studentmarks. These marks can be used in the final mark calculation of assessments.

To configure this please log a request at jobs@ecs.vuw.ac.nz specifying that you want it activated for your course and also specify the number of late days before penalties will be applied. Note that these days are a total for the course and not for each assessment. After this has been done you need adjust the final mark calculation formula for each assessment and also configure the weight of each assessment. Setting the weight higher means some assessments are worth more than others and are prioritized for late submission days. Setting the weight to 0 disables late penalties completely for an assessment - it will also not contribute to the late submission day total.

Detail of late penalty calculation

Assessments are ordered by weight and then due date. Then for each assessment:
  • Determine if and how late an submission was by using the submitted files' dates compared to the assessment due date and factor in any extensions given.
  • If the submission is late, calculate if any allowed late days are left for the student (considering previous late submissions ordered by the same method) and subtract that from the late time.
  • If the remaining result is positive a penalty mark called 'Late Penalty Days' is added to the student submission. The format of this mark is a decimal number - for example 1.75 means the assessment was late by one day and 18 hours. This mark is automatically approved.

The late penalties for the course's assessments are calculated each time all marks are approved for an assessment. Since the order of assessments depend on weight and the later due assessments can potentially weight more than earlier due ones, late date marks for an assessment can change over the duration of the course. This will benefit students because the higher weighting assessment can use the allowed late days first.

Final mark calculation using late days.

To use the 'Late Penalty Days' in the final mark calculation use the variable M0. For example to deduct 10% from the total for each full day late use the following formula:

(M1 + M2) * (10-floor(M0))/10

In this example the mark will be 0 if the submission is later than 10 days. If M0 is not used in the formula late days will have no influence on the calculated mark.

Example of a more complicated formula:

(M1+M2+M3+M4+M5)*(0.9**ceil(M0))

which gives an exponentially decaying 10% penalty for each late day.

Other notes for automatic late penalties:

  • If you approve a late submission for a student all late penalties for the student will be recalculated, so this is safe at any stage.
  • On the course marks page all the marks that contain a late penalty will be flagged with a light red background.
  • The assessment weight is configured on the Marking tab of the Assessment Config page.

Mark approval

Marks need to be approved before students are able to view them. This serves several purposes such as releasing marks for all students at once, allowing lectures to resolve issues and also checking the marks in general. After tutors have marked assessments the marks are in an unapproved state. After all issues have been resolved and the marks checked, marks can be approved by clicking on the 'Approve all' button on the submissions page. There are also options to approve only marks or only comments. After this process has completed all marks (and comments) should be approved and any submissions with further issues will have a red flag in the Approved column (the number in brackets is the unapproved mark count). Marks and comments can also be approved individually on the student's submission page.

The criteria for the above process to approve marks for a submission are as follow:
  • All questions must be marked (except for admin marks which are optional).
  • All questions must have only one mark. For instance if two tutors marked a student, and the same question was marked twice it cannot approve both.
  • Unless specified in the marksheet at least one comment is required.

Please note that imported marks are never handled by this process, it is intended for marks made by tutors (and staff) on the student's submission page. Imported marks are approved if the option to do so are checked while importing.

Submission location

Submissions can go wherever you please, but generally they go into /vol/submit/[coursecode]/[assessment]/. For security reasons, this area is not available for writing to from a workstation.

Marking directories

When students submit code you will often want to be able to change or compile their code. To do this, you will need to copy their submissions to their marking directories. This is a sub-directory of a student's submission that has write permission for staff. This way we keep the original submission intact without any changes. To do this click on the Admin functions button on the submissions page and then on the Copy to Marking button.

How to add marks and comments

  • Go to the Submissions list page by selecting a course and then an assessment's marking option.
  • On this page you can go to a specific student's submission page by clicking on their username, or if you want to mark it click on 'Start marking' in the status column. (If you are already assigned by a staff member this column will say 'Assigned (username)' with your username.)
  • The submission page is divided into 5 sections:
    1. At the top is the summary containing information such as student name, number of submitted files, late submission information, whether late submission was approved and problem status.
    2. The next section list all submitted files. You can view/download files here and also download all of them as a zip file.
    3. The Marks section is next. In this area will be a list of all the items to mark along with a field to add the mark. This field is updated immediately when you press tab or click outside it. Also displayed is the approval status and a delete button (only available if not yet approved). If more than one tutor are marking an submission (only for some courses) any marks added by other tutors will also be displayed here, they can be distinguished from yours by the username in the tutor column.
    4. The next section is for Comments. Existing comments are listed first and underneath that is a text area followed by a 'Add comment' button. New comments are only added after pressing this button. There is also an option to flag a comment as an 'admin' comment. These are intended to send messages to lecturers concerning the submission and are not displayed to students. Existing comments can be edited (only if not yet approved) by clicking on the pencil icon at the bottom. The comment will update immediately when you press tab or click outside this area.
    5. The last section will display the Marking guide if available. It can be displayed as a popup dialog to keep open next to the Marks or Comments section.
  • After you have marked all items (see 3 above) and added comments (see 4 above) required by the Marking guide, complete the marking by clicking on the 'Complete marking' button in the Summary section. Complete marking will only be successful if you have marked all the items and added the required comments. Next to this button is also a 'Give back' button. This will 'unassign' you from the submission so that another tutor can complete the marking.