RSS Feed

Posts Tagged ‘evaluation’

  1. Unit evaluation – composing a questionnaire

    5 April 2015 by shartley

    As part of my Masters of Education at Macquarie University I am studying EDCN800 Introduction to Educational Research.  Yesterday’s blog post lamented the frustrations I was having with the statistical aspects of the course. Today I was looking at the construction of questionnaires and the exercise set was basically the writing of some questions for evaluating a unit of study.  The thinking behind these questions is a little influenced by John Hattie’s Visible Learning (2009) which is currently guiding a research project with which I am involved, but mainly they come from an accumulation of experience and study of a wide range of material.  The areas I’ve noted and the questions themselves are not even close to being exhaustive lists. I actually hope to later develop this into a real questionnaire to evaluate my own courses.  I’d be interested in knowing what you would include.

    Aim: Evaluating a unit of study within the Masters of Education program.

    Research participants: Students undertaking the unit of study

    Key areas:

    The learning environment

    • Online
      • Ease of understanding/following procedure and instruction
      • Allows for communication and interaction with instructor and peers
      • Feels inclusive of all (gender, disabilities, internal/external students, etc)
    • On campus
      • Inviting environment (temperature, lighting, seating comfort, etc)
      • Allows for communication and interaction with instructor and peers
      • Feels inclusive of all (gender, disabilities, internal/external students, etc)

     

    The curriculum

    • Knowledge, understanding and skills
      • Made clear
      • Covers a range of lower-order and higher-order thinking skills
      • Challenging but achievable
    • Relevant
      • Builds on prior knowledge
      • Practical application to workplace and/or experience
      • Includes contemporary issues
    • Assessment
      • Both the assessment and related criteria are easy to understand
      • Aligns with goals for knowledge, understanding and skills
      • Offers choice
      • Fair, equitable and achievable (task and weighting)
      • Provides opportunity for feedback during the process
    • Transparent
      • Purpose of the learning is clear
      • All elements/tasks of the course are clear and upfront
      • What lies ahead in the course is clear
      • Timing is clear (module lengths, due dates, etc)

     

    The teacher

    • Knowledgeable
      • Course content
      • Student needs
      • Teaching methods
      • Contemporary context
    • Communication
      • Approachable/personable
      • Quality feedback
      • Clear in expectations
    • Strategies
      • Variety
      • Engaging
      • Motivates
      • Relevant
      • Easy to follow/do
      • Involves collaboration (at times)
    • Flexible in approach
      • Adapts to the learning needs of students
      • Adapts to changing circumstances

     

    The impact

    • Students know and understand more and can do more as a result of the course
    • Students are engaged, interested and/or enjoy the course
    • Students increase their desire to learn
    • Students obtain a sense of achievement from completing the course

     

    Examples of research questions to be included in a questionnaire conducted at the conclusion of a unit of study.

    The learning environment

    To assess the experience of the student in using the online page I would include these questions:

    The online course page was easy to navigate.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The online course page clearly presented the unit’s requirements.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The online course page supported collaborative learning.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    To conclude the online environment section I would ask an overarching question such as:

    How satisfied were you with the online environment in this unit of study?

    • Very dissatisfied
    • Dissatisfied
    • Neither satisfied nor dissatisfied
    • Satisfied
    • Very satisfied

     

    Knowledge, understanding and skills

    To assess the experience of the student’s learning process I would include these questions:

    The course outcomes were easy to understand.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The course was challenging.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The course extended my understanding of the topic.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The course helped me to think more deeply about the topic.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    To conclude the knowledge, understanding and skills section I would ask an overarching question such as:

    How satisfied were you with what you learned in this unit of study?

    • Very dissatisfied
    • Dissatisfied
    • Neither satisfied nor dissatisfied
    • Satisfied
    • Very satisfied

     

    Assessment

    To assess the student’s experience of assessment I would include these questions:

    The assessment requirements were easy to understand.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The assessment criteria was easy to understand.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The assessment criteria matched the assessment requirements.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The assessment requirements were challenging.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The assessment process extended my understanding of the topic.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The assessment process included helpful feedback before final submission.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree
    • Don’t know

     

    The assessment process was fair for all students.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree
    • Don’t know

     

     To conclude the assessment section I would ask an overarching question (or two) such as:

    How satisfied were you with how assessment was marked in this unit of study?

    • Very dissatisfied
    • Dissatisfied
    • Neither satisfied nor dissatisfied
    • Satisfied
    • Very satisfied

     

    How satisfied were you with the process of completing assessment in this unit of study?

    • Very dissatisfied
    • Dissatisfied
    • Neither satisfied nor dissatisfied
    • Satisfied
    • Very satisfied

     

    Communication

    To assess the student’s experience of the communication process I would include these questions:

     The teacher was easy to understand.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

     The teacher provided clear expectations.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The teacher provided helpful feedback.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree
    • Not applicable

     

    The teacher was approachable.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree
    • Not applicable

     

    The teacher provided timely responses to questions asked.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree
    • Not applicable

     

    To conclude the communication section I would ask an overarching question such as:

    How satisfied were you with how the teacher communicated in this unit of study?

    • Very dissatisfied
    • Dissatisfied
    • Neither satisfied nor dissatisfied
    • Satisfied
    • Very satisfied

     

    The impact

    To assess the impact of the unit of study I would include these questions:

    I know and understand more about this topic area as a result of completing this unit of study.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

     I found this unit of study interesting.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

     I want to learn more about this topic area.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

     I obtained a sense of achievement from completing this unit of study.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

     To conclude this section I would ask an overarching question such as:

    How satisfied were you with what you learned in this unit of study?

    • Very dissatisfied
    • Dissatisfied
    • Neither satisfied nor dissatisfied
    • Satisfied
    • Very satisfied

     

    Construction of the Questionnaire

    These questions were designed to align with what I thought was most important for assessing a unit of study.

    They were written to be easily understood (“natural and familiar language” with “clear, precise and relatively short items”, Johnson & Christensen 2014, p.193) and allow for an appropriate range of options.

    To keep the questions easy to follow I used a fully anchored scale for all questions and there were only two styles of rating scales: (1) Agreement – for students to assess the elements of the course (2) Satisfaction – for students to assess the impact of the course on themselves in a broader and more personal sense.

    On occasion an option of “Don’t know” or “Not applicable” was added to allow for students who had not experienced that particular aspect of the course.

    The number of points were kept to five to assess students’ ambivalence and avoid irritating participants by forcing a stance.

    The wording was kept consistent where possible between sections, such as asking if the course/assessment/convenor were easy to understand, to allow for direct comparisons between the elements being researched.

    I believe I have avoided leading or loaded questions although at times I was tempted to add an adjective or two which would have broken this principle.

     

     

    REFERENCES

    Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement.  London: Routledge.

    Johnson, R. and Christensen, L. (2014). Educational research: Quantitative, qualitative, and mixed approaches. (5th Edition). Thousand Oaks California: Sage.


Skip to toolbar