RSS Feed

Posts Tagged ‘research’

  1. Mind the Gap: Research and Schools

    1 November 2016 by shartley

    217048717_08e3e922f1_m

    Image by Mikel Ortega at https://www.flickr.com/photos/mikelo/217048717

    Last week I attended the AIS Education Research Symposium.  When I worked at Oakhill College I was part of the team that created and implemented REAL (Relevant Engaging, Active Learning) and then we researched its impact.  Unfortunately I left Oakhill before the end of the two year research project but thankfully I remained involved to an extent.  We designed REAL to create a paradigm shift in the teaching and learning environment at the school through a transparent curriculum in a detailed but clear (student-friendly) format, structured for students to know each lesson or week:

    • What is to be learned
    • How it is to be learned, and
    • The evidence of successful learning

    The research project examined the impact of REAL on student outcomes but effectively discovered the effect it had on teachers too.  The research revealed REAL as a significantly successful program, although teachers found it hard to adapt their practices, and I’m proud to have been associated with it.

    But this post is about the symposium.  The main participants at the symposium are from schools that have received a funding opportunity from AIS and are at the start, middle or end of a two year research project.  It is also for any other teachers who are keen to make stronger links between research and teaching practice.  And the first keynote speaker, Dr Stacey Waters, was exactly on this topic.

    It is extremely hard to make a cultural shift in any institution, let alone a school which has so many entrenched perceptions of what it should be like.  Nearly everyone in the world experiences school and our experiences are reinforced by pop culture’s representation of it, this image being a teacher up the front instructing students sitting in rows, resisting the learning process (except when they have a superstar teacher like Michelle Pfeiffer or Robin Williams).

    This is not the one best way to learn.  Academic research has indicated this repeatedly, yet many schools resist changing (much like their students resist learning).  I think part of it is the conflicting voices dominating the discourse of how schools should operate. As mentioned before, there is the entertainment industry image but there is also the political viewpoint that schools must do better, meaning better at publicly published score achievement (PISA, NAPLAN, HSC).  In education conference circles there are a few (mainly men) who have a certain popularity, often stringing out their one concept stories for decades (I’m thinking the hole-in-the-wall story and the idea that schools kill creativity).

    The academic voice is often lost in all this.  There are some who are picked up by politicians and/or the media. For example, John Hattie and his Visible Learning ideal is lauded by politicians and the media.  However, even the John Hatties often remain ignored in the day-to-day classroom, for a number of reasons.

    Personally I think it stems from a dominating need of isolated teachers in their classrooms to have control and order.  Control and order is easier if learning is considered to be the mere accumulation of knowledge.  It casts a dark shadow on education.  Teaching is better if it is collaborative, learning is better if it is collaborative, but it is much harder to maintain control and order in a collaborative environment when there isn’t a desire to learn in the first place, by teachers or students.  It is harder to know what individual students are doing in a collaborative environment and if it is known that they are not participating in the learning process, it is hard to know what to do about them.  There are plenty of theories, methods and systems that can be put in place but really, it all comes to dealing with individual motivations and desires to learn.  And to be honest, the better learning environments take more time and energy to plan and monitor.  I love an active collaborative learning environment in my classrooms but boy, it’s hard work a lot of the time.  

    Prof Ruth Deakin Crick shared this quote as part of her presentation about the use of technology to change school culture from exam-driven teaching to a learning culture with dynamic pedagogy and engagement (image, however, was clipped from elsewhere):

    screen-shot-2016-11-01-at-6-05-25-pm

    Teachers are so busy and comfortable with what they know well and what they’ve been rewarded for in the past, they don’t want to become even busier and take risks that result in lost control and disapproval.

    So now we can see why some resistance to change exists, let’s return to the academic voice and other reasons why it is difficult to implement research findings into school practice.

    To be able to research in a detailed, scientifically credible manner, academics focus on narrow areas to reduce multiple confounding factors in their study. They generally want to discover a cause and effect, and the size of that effect, but to be specific and certain they can only concentrate on one cause and one effect at a time, whereas education is much much more complicated than that.  It is like reducing economic modelling to two products in the market.  It illustrates a concept but an economy with just two products is not real life.  This is why Hattie’s study is so appealing, it throws hundreds of studies into the mix and calculates which causes have the most effect.  However, one point worth noting, is how long this all takes.  A research project can take years, and then months to write and publish in an academic journal, and then even longer to garner the attention of those who it really affects, people in schools.  I’m not sure of the age of the studies Hattie includes in his meta-analysis.  Due to the narrow focus and time-lags the question was raised at the conference as to whether the academic rigour should be sacrificed, to an extent, to allow more pertinent and practical studies be undertaken in a more timely manner.  I don’t know the answer.

    The publish or perish pressure on academics is a significant factor too.  The money associated with grants often comes with a proviso of what must be researched. Dr Stacey Waters also referred to the Excellence in Research for Australia (ERA) system which awards points according to the category of grants.  To be published, it helps if the research is something new and different, rather than going deeper into something that has already been examined.  The dream situation for an academic though, is selling books.  And books sell if they have something novel and easy to implement.  We all know how hard change management is, but just how many books advocate just ‘x’ number of steps (eg Kotter’s 8 steps) like it is a simple linear process?  It’s not.

    Even if academics find something that is commonly considered important to implement in schools, the process is slow and usually ineffectual. Researchers are already balancing teaching and research and then they need to market their findings so that schools will take their ideas on board, when so many ideas are already present.  Dr Stacey Waters says that publishing and even training people is not enough. One of her presentation slides said:


    Implementation is most successful when…

    • Practitioners receive training and coaching
    • The organisation provides the necessary infrastructure for training and coaching and regularly evaluates
    • The community is fully involved in the selection and evaluation of programs

    (Fixsen, Naoom, Blasé, Friedman & Wallace, 2005)


    She argued that schools should build relationships with universities and establish a research culture by providing easy access to academic journals and having a forum to discuss what these journals have to say.

    Before Oakhill obtained the research funding opportunity from AIS, we asked some universities to help us perform academic research.  We discovered that not only did they want to be paid consultation fees, that some even wanted to virtually take over the whole process.  We backed away quickly and were relieved and grateful to receive the AIS funding opportunity.  As part of the inaugural recipients of research funding there was only minimal guidance but still, some important structural requirements, including an academic advisor.  However, we ended up only spending 50% of the budgeted amount we allocated for the advisor. He was extremely helpful for implementing research that produced quantifiable data and how that data could be used, but partly due to how late we were and partly due to the lack of need, we didn’t use as much of his services for the writing part as we thought we would.

    So here we were at a conference that was bringing together academics and teachers, albeit those already interested in research. As Terrie Jones tweeted, “Teacher practitioner researchers in partnership with academics bridging the research practice divide? This room. #AISRandD16”.  One of the sessions I enjoyed the most was about Case Study as a form of research.  From what little I have been involved in research at a tertiary level, I have found case study methods frowned upon because it is not appropriate to extrapolate something that works in one circumstance as being applicable to a wider range of circumstances.  Dr Kimberley Pressick-Kilborn argued that yes, three case studies are better than one, but much can be learned from them and proceeded to demonstrate how powerful they could be but also recognising their limitations.  I tweeted that I’d love to work on a case study with Dr Kimberley and she responded with a “Let’s talk” which I will probably take her up on some time next year (see screen grab below).  However, I’m not looking to do research like this until at least 2018.  I started a Master of Research (MRES) at Macquarie University this year but stopped just a few weeks in due to lack of time. I’m currently not working full-time so I can help my son through his HSC (he has autism and anxiety issues).  I considered studying also during this year ‘off’ but I need to make him the priority, not my own work. After that I’ll be open to all sorts of projects!

    screen-shot-2016-11-01-at-6-30-03-pm

    The people at this conference are the core people of their organisations interested in learning and continuous improvement.  What we actually need to learn as researchers within schools, is how to gather-in colleagues to a learning mindset and join with us.

    When we first attempted researching the REAL project ourselves, we applied to present at academic conferences in Seville, Spain, and New York, USA, and were approved for both based on our ‘abstracts’.  This was before the AIS research funding program even existed and we were without any support from universities.  We then had to write papers for these conferences, the New York paper being subject to a peer review.  It failed one reviewer but the other advised us to change the format significantly, which we did, and it was then published <http://shanihartley.cgpublisher.com/product/pub.264/prod.58>*.  We also had to review other papers which was a great learning experience for us.  At these conferences there was much said about education at all levels but over 90% of the presenters were from universities.  Universities seem to take a more hit-and-run approach instead of working in tandem with the schools they research.  It would be nice to see or even experience more collaboration between schools and universities in the research process at academic conferences.  Perhaps we were better off without a university’s own agenda. We were unusual in being school teachers conducting and presenting our own research.  There should be more of it.  

     

    * Despite numerous emails to the publishers, the bios of the authors were not corrected.
    This post is also found at http://squibsandsagas.blogspot.com.au/2016/11/mind-gap-research-and-schools.html

  2. Unit evaluation – composing a questionnaire

    5 April 2015 by shartley

    As part of my Masters of Education at Macquarie University I am studying EDCN800 Introduction to Educational Research.  Yesterday’s blog post lamented the frustrations I was having with the statistical aspects of the course. Today I was looking at the construction of questionnaires and the exercise set was basically the writing of some questions for evaluating a unit of study.  The thinking behind these questions is a little influenced by John Hattie’s Visible Learning (2009) which is currently guiding a research project with which I am involved, but mainly they come from an accumulation of experience and study of a wide range of material.  The areas I’ve noted and the questions themselves are not even close to being exhaustive lists. I actually hope to later develop this into a real questionnaire to evaluate my own courses.  I’d be interested in knowing what you would include.

    Aim: Evaluating a unit of study within the Masters of Education program.

    Research participants: Students undertaking the unit of study

    Key areas:

    The learning environment

    • Online
      • Ease of understanding/following procedure and instruction
      • Allows for communication and interaction with instructor and peers
      • Feels inclusive of all (gender, disabilities, internal/external students, etc)
    • On campus
      • Inviting environment (temperature, lighting, seating comfort, etc)
      • Allows for communication and interaction with instructor and peers
      • Feels inclusive of all (gender, disabilities, internal/external students, etc)

     

    The curriculum

    • Knowledge, understanding and skills
      • Made clear
      • Covers a range of lower-order and higher-order thinking skills
      • Challenging but achievable
    • Relevant
      • Builds on prior knowledge
      • Practical application to workplace and/or experience
      • Includes contemporary issues
    • Assessment
      • Both the assessment and related criteria are easy to understand
      • Aligns with goals for knowledge, understanding and skills
      • Offers choice
      • Fair, equitable and achievable (task and weighting)
      • Provides opportunity for feedback during the process
    • Transparent
      • Purpose of the learning is clear
      • All elements/tasks of the course are clear and upfront
      • What lies ahead in the course is clear
      • Timing is clear (module lengths, due dates, etc)

     

    The teacher

    • Knowledgeable
      • Course content
      • Student needs
      • Teaching methods
      • Contemporary context
    • Communication
      • Approachable/personable
      • Quality feedback
      • Clear in expectations
    • Strategies
      • Variety
      • Engaging
      • Motivates
      • Relevant
      • Easy to follow/do
      • Involves collaboration (at times)
    • Flexible in approach
      • Adapts to the learning needs of students
      • Adapts to changing circumstances

     

    The impact

    • Students know and understand more and can do more as a result of the course
    • Students are engaged, interested and/or enjoy the course
    • Students increase their desire to learn
    • Students obtain a sense of achievement from completing the course

     

    Examples of research questions to be included in a questionnaire conducted at the conclusion of a unit of study.

    The learning environment

    To assess the experience of the student in using the online page I would include these questions:

    The online course page was easy to navigate.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The online course page clearly presented the unit’s requirements.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The online course page supported collaborative learning.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    To conclude the online environment section I would ask an overarching question such as:

    How satisfied were you with the online environment in this unit of study?

    • Very dissatisfied
    • Dissatisfied
    • Neither satisfied nor dissatisfied
    • Satisfied
    • Very satisfied

     

    Knowledge, understanding and skills

    To assess the experience of the student’s learning process I would include these questions:

    The course outcomes were easy to understand.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The course was challenging.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The course extended my understanding of the topic.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The course helped me to think more deeply about the topic.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    To conclude the knowledge, understanding and skills section I would ask an overarching question such as:

    How satisfied were you with what you learned in this unit of study?

    • Very dissatisfied
    • Dissatisfied
    • Neither satisfied nor dissatisfied
    • Satisfied
    • Very satisfied

     

    Assessment

    To assess the student’s experience of assessment I would include these questions:

    The assessment requirements were easy to understand.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The assessment criteria was easy to understand.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The assessment criteria matched the assessment requirements.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The assessment requirements were challenging.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The assessment process extended my understanding of the topic.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The assessment process included helpful feedback before final submission.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree
    • Don’t know

     

    The assessment process was fair for all students.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree
    • Don’t know

     

     To conclude the assessment section I would ask an overarching question (or two) such as:

    How satisfied were you with how assessment was marked in this unit of study?

    • Very dissatisfied
    • Dissatisfied
    • Neither satisfied nor dissatisfied
    • Satisfied
    • Very satisfied

     

    How satisfied were you with the process of completing assessment in this unit of study?

    • Very dissatisfied
    • Dissatisfied
    • Neither satisfied nor dissatisfied
    • Satisfied
    • Very satisfied

     

    Communication

    To assess the student’s experience of the communication process I would include these questions:

     The teacher was easy to understand.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

     The teacher provided clear expectations.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

    The teacher provided helpful feedback.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree
    • Not applicable

     

    The teacher was approachable.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree
    • Not applicable

     

    The teacher provided timely responses to questions asked.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree
    • Not applicable

     

    To conclude the communication section I would ask an overarching question such as:

    How satisfied were you with how the teacher communicated in this unit of study?

    • Very dissatisfied
    • Dissatisfied
    • Neither satisfied nor dissatisfied
    • Satisfied
    • Very satisfied

     

    The impact

    To assess the impact of the unit of study I would include these questions:

    I know and understand more about this topic area as a result of completing this unit of study.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

     I found this unit of study interesting.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

     I want to learn more about this topic area.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

     I obtained a sense of achievement from completing this unit of study.

    • Strongly disagree
    • Disagree
    • Neutral
    • Agree
    • Strongly agree

     

     To conclude this section I would ask an overarching question such as:

    How satisfied were you with what you learned in this unit of study?

    • Very dissatisfied
    • Dissatisfied
    • Neither satisfied nor dissatisfied
    • Satisfied
    • Very satisfied

     

    Construction of the Questionnaire

    These questions were designed to align with what I thought was most important for assessing a unit of study.

    They were written to be easily understood (“natural and familiar language” with “clear, precise and relatively short items”, Johnson & Christensen 2014, p.193) and allow for an appropriate range of options.

    To keep the questions easy to follow I used a fully anchored scale for all questions and there were only two styles of rating scales: (1) Agreement – for students to assess the elements of the course (2) Satisfaction – for students to assess the impact of the course on themselves in a broader and more personal sense.

    On occasion an option of “Don’t know” or “Not applicable” was added to allow for students who had not experienced that particular aspect of the course.

    The number of points were kept to five to assess students’ ambivalence and avoid irritating participants by forcing a stance.

    The wording was kept consistent where possible between sections, such as asking if the course/assessment/convenor were easy to understand, to allow for direct comparisons between the elements being researched.

    I believe I have avoided leading or loaded questions although at times I was tempted to add an adjective or two which would have broken this principle.

     

     

    REFERENCES

    Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement.  London: Routledge.

    Johnson, R. and Christensen, L. (2014). Educational research: Quantitative, qualitative, and mixed approaches. (5th Edition). Thousand Oaks California: Sage.


  3. Research

    4 April 2015 by shartley

    20150404-220801.jpg

    Image source: author’s own textbook

    I’m currently studying EDCN800 Introduction to Educational Research at Macquarie University.  It is the only compulsory subject in my course but I put it off to last because it seemed so dry, and well, boring.  I have my regrets, however, since it would have been quite useful to know what I should be doing before I submitted abstracts on behalf of my team to international conferences.  When we were accepted to these conferences I had to write academic level articles on the basis of haphazard and amateurish research.  One of these articles was for a peer-reviewed journal and one of the two peers who assessed the paper slammed it for not being written in the acceptable academic format.  I had avoided the more academic format because I didn’t want to pretend that the research was formally conducted.  I have now resubmitted the piece into a more acceptable format but it still awaits final approval.

    More recently, I have been trying to support a friend who has been designing real proper research under the guidance of a university professor.  The professor’s critiquing of the attempts to write a research question and plan the research methods was a painful process but the frustration was worth it in the end because I think there is a very valuable research project currently underway.  As I do this course I can, in retrospect, see more clearly what was required and if I had completed this subject before this year it might have been a much less painful process.  Now, as I study the ethics of research I wonder how much more should be done to cover ethical considerations in my friend’s research.  It is also giving me more depth to my knowledge of research methods for when I teach Society and Culture.

    In the first semester of my Masters of Education I chose one subject (curriculum) because a friend was also doing it and another subject because I felt knowledgeable in that area (pedagogy).  I had enrolled in the course just so I could obtain the piece of paper and letters to look good on my CV but within a couple of weeks of participating in these two subjects I was enjoying myself immensely and did quite well as a result.  However, the one aspect that I was continually criticised about was the negligible evidence to support my (soapbox) statements.  I have improved a lot in this area since then.

    Now as I study EDCN800 I expect high achievement from myself but I’m not succeeding. I’m engaged in the subject because of the afore-mentioned application but despite being quite numeracy literate I struggle with the statistical concepts and analysis of data.  I only received 65% for the first of five assignments.  Today I battled with the concepts of reliability and validity with all their different coefficient measurements.  The concepts in themselves are fine but when I have to apply them to a technical academic article it becomes all muddled up and difficult to navigate.  Not only do I need to understand these concepts for EDCN800 but I am also writing a literature review for EDCN806 which requires an examination of the reliability and validity of the articles I am including in the review.  It is all driving me insane and I question my ambition to complete a PhD down the track.  As a result I’m feeling a fair bit of empathy with my students at the moment.

    Anyway, that’s enough complaining, I need to attack a question about evaluations using numerical ratings and then write some of my own questionnaire items to assess student experience in studying the Masters of Education.  It is so much easier to help my students design their research for their Personal Interest Projects (PIPs) in HSC Society and Culture than to do it myself at a university level.  But here I go…

    Stay tuned.


Skip to toolbar