Guskey+Discussions

**Guskey Chapter 4- Evaluating Professional Development **

Level One: Participants’ Reactions

Level One is primarily concerned with how participants regard their professional development experience including their reactions to formal presentations, workshops, courses, seminars, or institutes. This may also apply to their perceptions of curriculum development activities, study group experiences, or action research projects (p. 94).

**Question 1- What questions are addressed?**

Content Q’s- address the relevance, utility, timeliness of topics covered; they focus on the new knowledge, skills and understandings that are the basis for the program; can also relate to scope, magnitude, credibility, and practicality of the change required. *Participants generally express more positive perceptions toward the content if they have some say in determining what that content will be AND when content address specific problems and practical, relevant solutions that can be implemented immediately.

(Examples of content questions can be found on p. 95)

Process Q’s- related to the conduct and organization of the professional development experience- how were things done. In formal settings, process questions focus on program leaders or specific activities. In less formal settings, process questions generally related to the structure and format of the activities. Responses may vary wildly due to different learning styles. For this reason, it is wise to try to include a variety of activities and learning formats.

(Examples of process questions can be found on p. 96)

Context Q’s- generally related to the setting including facilities, lighting, technology, room size and temperature, etc. These factors can be a necessary prerequisite to a successful professional development session. It may also be beneficial to ask the participants to rate themselves as learners- this causes them to reflect more carefully on their conduct.

(Examples of context questions can be found on p. 98)

**Question 2- How will the information be gathered?** Usually information is gathered towards the end of the professional development session- the last 10-15 minutes. These forms are usually collected as people leave to ensure information is obtained from everyone who took part. The format of these evaluation forms can vary greatly. Most forms usually include both multiple choice items and open-ended questions. Things to consider when gathering information__.__

__All participants vs. a sample of participants__- how much data can you effectively analyze? Allowing the evaluator to choose who is sampled creates bias, so the best method is to randomize the participants, e.g. choose every 5th person.

__Signing evaluation forms__-requiring signatures can allow for follow-up or clarification but may increase reluctance to answer questions honestly. According to Guskey, the most important attributes of the evaluation form are honesty and candor.

__Delayed evaluations__- when participants evaluate a PD right after the session they tend to rush through the form providing inaccurate or missing data. Some evaluators allow participants to wait a few days to answer which can allow them to reflect more deeply on the experience. However, rarely do all participants take the time to complete the forms and return it. So, contingencies can be put in place to guarantee their submission, e.g. Credit is only given once the evaluation is submitted.

__Alternative methods__- focus groups and interviews can be alternative methods of evaluation, but they breech anonymity, so evaluators can present the answers of the participants in summary form. Reflective journals or professional learning logs, etc. can also be considered alternative forms and participants can be given guiding questions to direct their responses. These methods provide rich information from which to draw conclusions about the experience.

__Ongoing evaluations__- Gathering several evaluations over time in phases or in components provides more in depth information to be evaluated. It is important to be strategic about time and place of gathering this information, as both variables impact the type of information collected.

(Examples of many common evaluation forms can be found on pages 108-114)

**Question 3- What is measured or assessed?** Initially, the point is to measure participants’ initial satisfaction with the experience. There is no exemplar form and professional development administrators are encouraged to create one based on the specific professional development being offered. It is possible to evaluate professional development as “criterion referenced” or a “norm referenced”. Criterion-referenced refers to reactions to a professional development based on professional standards, specific criteria, or quality indicators. Norm-referenced refers to evaluation which occurs based on participants experience in relation to the other professional developments. Usually, it is better to create criterion-referenced evaluations as participants past professional development experiences can vary widely.

**Question 4- How will the information be used?** There are many uses for the information gathered including modifying content offered, choosing follow-up activities, guide further professional development activities, and evaluating the evaluation form itself. Each level of evaluation builds on itself. All evaluations can provide crucial data that inform all other levels of professional development.

National Staff Development Council (Learning Forward): [] **//__Level 2: Participant’s Learning__//**

“In addition to liking it, we would also hope that participants learned something from their professional development” (Guskey, 2000, p.83).

· Range from paper and pencil assessment to demonstration of skills. · Learning can be documented in many permanent products (reflections, portfolio, case study analysis). || · Use of a pre- and post-test to measure participant knowledge · Guskey recommendation to assess learner //perception// of knowledge acquired may seem counterintuitive, but he’s no dummy (p. 131). || A: 1)validates relationship between intended teaching and final learning, 2) indicates effectiveness, and 3) formative data to modify PD
 * Focus || Measure the knowledge, skills, and perhaps attitudes that participants learned. ||
 * Purpose || To use information gathered to revise content, format, & organization of PD. ||
 * Assessments || · Customized to the specific learning at hand (not standardized)
 * Considerations || · Openness to “unintended learnings”
 * Questions & Quick-n-Dirty Answers || Q: Why is it important to assess participants’ learning?

Q: What questions are addressed at this level? A: Did they learn the planned objectives?

Q: What types of learning are assessed? A: 1) Cognitive, 2) Psychomotor, and 3) Affective

Q: How will information be gathered? A: Procedures should be based on intended learning goals

Q: What is measured or assessed? A: Knowledge, skills, attitudes, and/or beliefs

Q: How will the information be used? A: Used to guide improvements in content, format, and organization of PD. ||

**//__Level Four: Participants’ Use of New Knowledge and Skills (Guskey, 2000) __//** Level Four addresses whether professional development (PD) participants have reflected on what they learned through their experience and how the experience has resulted in changes in practice. Evaluation of Level Four can only be conducted after program participants have had a sufficient amount of time to reflect on ideas learned and fit newly acquired skills and knowledge to their own settings (see pg. 180 for discussion). Level Four, then, is addressing __how PD affects practice__.

Gathering and analyzing information about whether or not new practices are used, and how well they are used, are essential activities in evaluation professional development programs and activities.

**Question 1- What questions are addressed? **

1. //Stages of Concern //: The assessment of the affective domain involved when individual(s) change their practice. “Assessing participants’ concerns is crucial for both formative and summative evaluations of professional development” (p. 184). Generally, six stages of concern are assessed across four categories.


 * Category || Stage || Label || Description ||
 * Impact || 6 || Refocusing || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Evidence of exploring benefits resulting from change indicated. Includes major alteration or adaptations of practice. ||
 * || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">5 || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Collaboration || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Evidence of coordinating with others regarding the change in practice indicated. ||
 * || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">4 || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Consequence || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Evidence of how the change is affecting student outcomes (how, which outcomes are affected), and necessary adaptations) indicated. ||
 * <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Task || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">3 || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Management || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Evidence of a focus on processes and required to apply change in practice (best uses of information/resources) indicated. ||
 * <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Self || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">2 || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Personal || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Evidence of a focus in on the demands of the change and the individual’s ability to meet those demands. Reward structures, decision making, and potential conflicts are assessed. ||
 * || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">1 || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Informational || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Evidence of thought regarding the details involved with implementing the change indicated. “need more information” stage. ||
 * <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Awareness || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">0 || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Awareness || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Little concern about the change is indicated. ||

<span style="font-family: 'Times New Roman','serif'; font-size: 12pt;">2. //<span style="font-family: 'Times New Roman','serif'; font-size: 12pt;">Levels of Use //<span style="font-family: 'Times New Roman','serif'; font-size: 12pt;">: Levels of use involve varying actions or non-actions related to a participant’s use of the new knowledge or skills gained through PD. Generally, there are three levels of non-use and five levels of use measured.


 * <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Category || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Stage || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Label || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Description ||
 * <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Use || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">6 || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Renewal || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Reevaluation of efforts indicated with needed alteration modification to practice. ||
 * || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">5 || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Integration || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Indicates an attempt made to collaborate with colleagues/disseminate practice. ||
 * || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">4b || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Refinement || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Reflection indicated/Varies the use of practice within context to improve student outcomes. ||
 * || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">4a || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Routine || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Pattern of use indicated but no thought/reflection on the practice. ||
 * || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">3 || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Mechanical || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Superficial day to day use of practice indicated. No reflection on practice. ||
 * <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Non-use || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">2 || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Preparation || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Evidance of preparation for use. ||
 * || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">1 || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Orientation || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Information and resources for use investigated. ||
 * || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">0 || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt;">Non-use || <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">No involvement and no interest in becoming involved in practice. ||

<span style="font-family: 'Times New Roman','serif'; font-size: 12pt;">3. //<span style="font-family: 'Times New Roman','serif'; font-size: 12pt;">Differences in Practice //<span style="font-family: 'Times New Roman','serif'; font-size: 12pt;">: Concerns whether current practice is truly different from what was being done in the past. *PD must involve differences from what was being done in the past. Otherwise, teachers will not implement any changes because they cannot see what is different from what was already being done.

<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt 0.5in;">“Another, more complex issue is determining whether observed differences in practice are actually due to the professional development experience and not to other, extraneous factors.” (p. 187).

<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 0pt 0.5in;">“Evidence gathered before serves to establish a baseline of part practices, whereas evidence gathered afterwards shows the extent of change or innovation” (p. 187).

**<span style="font-family: 'Times New Roman','serif'; font-size: 12pt;">Question 2- How will the information be gathered? **

<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Instruments used to gather information in Stage Four seek to capture the presence or absence of critical indicators. Critical indicators answer the question, “What would we expect to see if effective implementation were taking place?” Identifying five or six critical indicators is recommended to effectively capture the nature of change occurring. <span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">Program evaluators identify critical indicators and then develop descriptive examples of what each critical indicator looks like when properly implemented. See page 190 of the text for an example. Once critical indicators are identified and defined, instruments can be found that assess the indicators. Examples of such instruments include but are not limited to:

· <span style="font-family: 'Times New Roman','serif'; font-size: 12pt;">Direct observation · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt;">Interviews · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt;">Questionnaire · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt;">Focus Groups · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt;">Implementation logs · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt;">Participant Portfolios · <span style="font-family: 'Times New Roman','serif'; font-size: 12pt;">Rubrics

<span style="font-family: 'Times New Roman','serif'; font-size: 12pt; line-height: normal; margin: 0in 0in 10pt;">It is often appropriate to use a comparison group to measure the validity of evaluation using these measures.

**Question 3- What is measured or assessed?**
====<span style="font-family: 'Times New Roman',Times,serif;">The major issue being measured or assessed in Level Four is whether new knowledge and skills gained effect professional practice, and to what extent. ==== <span style="font-family: 'Times New Roman',Times,serif;">Key Question: // What did participants learn through professional development that affected their professional practice? //

<span style="font-family: 'Times New Roman',Times,serif;">**Question 4- How will the information be used?**
====<span style="font-family: 'Times New Roman',Times,serif;">Information gathered in Level Four can be used both in a summative nature to evaluate overall impact or formatively to drive changes in the PD program. ==== <span style="font-family: 'Times New Roman',Times,serif; font-size: 11pt; line-height: 115%;">Analysis can be used both formative and summative. Formatively, it can be used to facilitate better and higher quality implementation of new practices. Summatively, information on participants’ new knowledge and skills which help document implementation efforts.

__**Level 5**__
Level 5 is focused on wanting to know if the professional development program or activity was a benefit to students in any way. Did it result in improvements in students achievement, attitudes, perceptions, or behaviors?

"True purpose and ultimate goals was to provide educators with the professional knowledge and craft skills they need to help all students learn at high levels". (Guskey, 1999b) -leads to changes in practice and improved results with students but must be accompanied with feedback, collaborative planning, and ongoing assistance. -allowed professional development leaders to set expectations high and make their standards rigorous -broadened perspectives on factors that influence professional development -empowered professional developers to make what they do count Did all students acquire the intended knowledge, skills, attitudes, beliefs, and behaviors? What were the unintended consequences and their effects?
 * Question 1- Why is it important to assess student learning outcomes?**

//Cognitive Outcomes//-student academic achievements and accomplishments that students might be expected to acquire //Affective Outcomes//-student beliefs, feelings, dispositions, and attitudes that students might be expected to develop //Psychomotor Outcomes//- student behaviors, actions, or practices that we want students to acquire
 * Question 2-What types of Learning Are Assessed?**

//Cognitive Outcome Information// - standardized tests, teacher developed assessments, standardized performance assessments, grades, questionnaires, group tasks or activities, school records, and/or portfolios //Affective Outcome Information// - questionnaires and interviews //Psychomotor Outcome Information// - observations, questionnaires, interviews, and school records
 * Question 3- How will the information be gathered?**

//Validity// -- especially construct validity //Reliability// //Sampling// --simple random and stratified random commonly used //Importance of multiple measures --// assessments of change or improvement require multiple measure; formative evaluations require assessments of interim and final goals; evaluations must consider intended and unintended outcomes; and complex interactions among outcome measures necessitate the use of multiple measures. //Disaggregation of data// -- consider separately the evidence from different student subgroups //Use of pre- and posttest to measure student learning outcomes// //Use of comparison groups to measure student learning outcomes// //Importance of timing// -- when to collect information
 * Question 4-What other assessment issues should be considered?**

Knowing who wants the information, what information they want, and how they plan to use it to help guide evaluation decisions. Decisions based on accurate and meaningful information, appropriately analyzed and clearly presented, is the goal of every evaluation event.
 * Question 5 - How will this information be used?**