Understanding and Using Assessment Results.
Anuncio
San Juan, Puerto Rico August 2008 Understanding and Using Assessment Results Linda Suskie, Vice President Middle States Commission on Higher Education 3624 Market Street, Street Philadelphia PA 19104 Web: www.msche.org E-mail: [email protected] Traducido par Ray Petty y Grace Nazario Universidad Interamericana de Puerto Rico, Recinto de Ponce 1. What at is s Assessment ssess e t of Student Learning? z Deciding what we want our students to learn z Making sure they learn it! --Jane Wolfson, Director, Environmental Science & Studies Program, g Towson University y © Linda Suskie, MSCHE Student Learning Assessment as a Four-Step Cycle El avalúo del aprendizaje estudiantil como un ciclo de cuatro pasos 1. Learning Goals – metas de aprendizaje 22. Learning Opportunities – oportunidades para aprendizaje 44. Using Results – aplicando resultados 3 Assessment - avalúo 3. © Linda Suskie, MSCHE What is Institutional Effectiveness? ¿Qué es la efectividad institucional? 7. Mission & Goals Misión y metas Community Service Servicio a la comunidad 14. Student Learning Aprendizaje Estudiantil Productivity/ Efficiency Productividad/ eficiencia Revenue Generation Producción de ingresos © Linda Suskie, MSCHE Scholarship Erudición Access Acceso Diversity Diversidad 1. Mission & Goals Misión y metas 2 Planning 2. Planificación 8. Admissions Admisiones 3. Resources Recursos 9. Student Support Services Servicios de apoyo estud. 4. Leadership/Governance 4 Liderazgo/gobernación 10. Faculty 10 Facultad 5. Administration Administración 11. Educational Offerings Ofrecimientos educativos 6. Integrity Integridad 12. General Education Educación general 13. Related Educ. Activities Activid. educ. relacionadas 7 Institutional Assessment 7. Avalúo institucional 14. Asmt. of Student Learning MSCHE Avalúo© Linda de Suskie, aprend. estud. 2. What is “Good” Assessment? ¿Qué Q é es un avalúo ú “bueno”? “ ? Five Dimensions of Good Assessment Used útiles •Focus on clear and important goals •Results communicated widelyy and transparentlyy © Linda Suskie, MSCHE Five Dimensions of Good Assessment •Keep things simple. •Mantener las cosas sencillas. •Focus on a few (3-6) important goals in each program, unit, curriculum. •Enfocar en algunas metas importantes (3 á 6) en cada programa, unidad o currículo. •Stagger gg assessments. •Efectuar los avalúos en tiempos diferentes. •Use samples. •Utilizar muestras. U d Used Cost Effective Costo efectivos © Linda Suskie, MSCHE Five Dimensions of Good Assessment U d Used Reasonably accurate & truthful results Bastante acertados y confiables Cost effective •Flow from clear and important goals •Múltiple Múltiple tipos de instrumentos •Student learning: Include direct evidence Aprendizaje estudiantil con evidencia directa •Aprendizaje •Developed thoughtfully; perpetual works in progress © Linda Suskie, MSCHE Five Dimensions of Good Assessment U d Used Cost effective Val ed Valued Reasonably accurate & truthful results •Results Results inform important decisions on important goals •Ongoing, not once-and-done • Continuo, no de golpe •Overall guidance & coordination • Orientación y coordinación total © Linda Suskie, MSCHE Five Dimensions of Good Assessment •Flow from & focus on clear & important goals •Inform Inform important decisions on important goals Reasonably U Used d •Have clear, appropriate standards for accurate & acceptable & exemplary student performance truthful results Val Valued ed Cost effective Clear & important goals © Linda Suskie, MSCHE 3. Setting Benchmarks or Standards: How Good is Good Enough? Mi h l earned Michael d 55 points i t on th the midterm. idt z Did he do well on the midterm? © Linda Suskie, MSCHE To o dec decide de if Michael c ae “did d d well,” e , we e must us compare his 55 against something else. z z z z z z z z Benchmark Standard Target Frame of reference Lens Criterion “B i htli ” “Brightline” The “something else” depends on what we want the test to tell us us. © Linda Suskie, MSCHE Suppose 35 is passing and d 80 iis a perfect f t score. z Local standards z z z Question answered: z z aka competency-based competency based aka criterion-referenced Are our students meeting our standards? Challenge: z Establishing sound performance standards © Linda Suskie, MSCHE Suppose 35 is passing and 80 is a perfect score on a published exam. z External standards z z z Question answered: z z aka competency-based aka criterion-referenced Are our students meeting external standards? Challenge: z Do the standards match what we think is i important? t t? © Linda Suskie, MSCHE Suppose the class average is 65. z Peer benchmark z aka norm-referenced z Question answered: z z How do our students compare to peers? Challenge: z Identifying appropriate peers & collecting info from them © Linda Suskie, MSCHE S Suppose Mi Michael h l scored d 25 a year ago. z Value-added benchmark z z Question answered: z z aka growth, change, pre-post Are our students improving? Ch ll Challenges: z z z z z Transfers in or out Motivating students on pre pre-test test Is growth due to us? Imprecise assessments mask growth Is this question relevant? © Linda Suskie, MSCHE Suppose cclass ass a average e age is s 65 now o and 40 three years ago. z Historical trends benchmark z z z Questions answered: Q z z z aka improvement p aka (incorrectly) longitudinal Are our teaching & curricula improving? Are we g getting g better? Challenge: z Using g the same assessment © Linda Suskie, MSCHE Suppose Michael scored a 65 for knowledge and a 45 for real-world applications. z z Strengths and weaknesses benchmark Question answered: z z What are our students’ students relative strengths and areas for improvement? Challenge: z Getting “sub-scores” that are truly comparable © Linda Suskie, MSCHE Suppose Michael’s Michael s 55 cost $400 and Melissa’s 55 cost $300. z z Productivity benchmark Question answered: z z Are we getting the most for our investment? Challenge: z z Calculating cost and benefit accurately Keeping the focus on effectiveness as well as efficiency © Linda Suskie, MSCHE Which benchmark or standard should you use? z Each has advantages and disadvantages. z Each gives a somewhat incomplete picture. z Multiple perspectives give the most balanced picture of student learning learning. © Linda Suskie, MSCHE 4. Sharing Assessment Results Why are you assessing Wh i your program or curriculum? Validate it to others (accountability) z Improve p it z Make sure it isn’t slipping z © Linda Suskie, MSCHE Keep assessment summaries useful to you and your colleagues. z z z Who on campus needs to see the results? Why? What decisions will they make? What do they need to see to make those decisions? © Linda Suskie, MSCHE What decisions might your assessment help with? z Learning goals z z Curriculum z z z z z Is online instruction as effective as traditional instruction? Is collaborative learning more effective than traditional lectures? Are we developing a community of scholars? A Assessments t z z What is the value of service learning? Should our courses have more uniformity across sections? Teaching methods z z Are our learning goals sufficiently clear and focused? Have our assessments been useful? Resource allocations z Where should we commit our resources first? © Linda Suskie, MSCHE Keep assessment summaries short and simple. z Fast and easy to read and understand z Use short, simple charts, graphs, and lists. z z z z z z First aggregate (sum up) data, then drill down into details as needed. needed Round results. Sort results from highest to lowest lowest. Percentages may be more meaningful than averages. z z Use PowerPoint presentations. p Avoid narrative text. Avoid complex statistics. As you collect results over time, show trends. © Linda Suskie, MSCHE Tell a story. z Key questions to address: z z z What have you learned about your students’ learning? What are you going to do about what you have learned? When, where, and how are you going to do it? z Focus on “big news.” z z Doug g Eder Have a statistician identifyy meaningful vs. insignificant differences. Find someone skilled at finding the stories in reams off data. d © Linda Suskie, MSCHE 5. Using g Assessment Results to Improve Teaching & Learning When Assessment Results Are Good C l b Celebrate! ! Publicize! When assessment results are disappointing… Goals z Curriculum z Pedagogy z Assessments z Look at your learning goals. z z z Are your goals inappropriate or overly ambitious? Do your goals need to be clarified? Do you have too many goals? © Linda Suskie, MSCHE Look at your curriculum. z z Including placement and developmental education education. Does the curriculum adequately address each learning goal? © Linda Suskie, MSCHE Look at your teaching methods. z How do student learn best? © Linda Suskie, MSCHE Look at your assessments. z z z Are they poorly written and misinterpreted? Do they match your key learning goals? Are they too difficult for most responsible students? © Linda Suskie, MSCHE Isn’t poor performance the student’s fault? z z Sometimes, but usually a minority Suskie’s Suskie s “50% 50% rule” rule © Linda Suskie, MSCHE 6. Telling Your Story to Middle States What should institutions document? ¿Qué deben documentar las instituciones? z z z z Clear statements of goals - Metas claras Organized, sustained assessment process (“assessment plan”) Proceso de avalúo organizado y sostenido (“plan de avalúo”) z Principals, Principals guidelines guidelines, support - Principios, Principios guías guías, apoyo z What assessments are already underway - Cuáles avalúos están en proceso z What assessments are planned, when, & how - Cuáles avalúos son planificados, cuando, cómo Assessment results documenting progress toward accomplishing goals - Documentando el progreso de los resultados de avalúo para lograr las metas How results have been used for improvement p - Cómo los resultados han sido utilizados para el mejoramiento © Linda Suskie, MSCHE How might institutions document this? ¿Cómo las instituciones p ¿ pudieran documentar esto? z Need not be a fancy bound document! No tiene que ser un documento de lujo ni encuadernado z An overview in the report Una resumen en el documento z A chart or “roadmap” for assessment documentation Una guía í o “mapa” “ ” para documentar d t ell avaluó l ó z Where are you with each institutional goal, academic program and Gen Ed requirement? ¿En qué etapa se program, encuentra con cada meta institucional, programa académica y requisito de educación general? © Linda Suskie, MSCHE z Assessment documentation for each institutional goal goal, program, and Gen Ed requirement Documentación de los avalúos p para cada meta institucional,, programa académica y requisito de educación general z For large/complex institutions, a representative sample Para instituciones grandes/complejos, una muestra representativo z In the report, an appendix, online, or burned onto a CD Dentro de documento, en un apéndice, o grabado en un CD © Linda Suskie, MSCHE Questions that a reviewer might ask Preguntas que un lector pudiera presentar Are the right things happening? ¿Están ocurriendo las cosas correctas? Does the report focus on past and present, not just future? Are the characteristics of “good” assessment being met? ¿Están logrando las características de un “buen”avaluó? Do institutional leaders support and value a culture of assessment? ¿Apoyan y valorizan los líderes institucionales una cultura de avalúo? z Is there adequate support for assessment? z ¿Hay apoyo adecuado para el avalúo? z Overall guidance & coordination Orientación y coordinación total Are assessment efforts recognized & valued? Los esfuerzos de avalúo,¿son avalúo ¿son reconocidos y valorados? z Are efforts to improve teaching recognized & valued? Los esfuerzos de mejorar la enseñanza enseñanza, ¿son reconocidos y valorados? © Linda Suskie, MSCHE How much has been implemented, and have appropriate assessment processes been planned to plug holes? ¿Qué cantidad ha sido implementada ¿ p y si los procesos de avalúo apropiados han sido planificados para responder a deficiencias? z Are timelines appropriate? ¿Es el calendario apropiado? z Are the plans feasible? ¿Son factibles los planes? z Simple - sencillo z Practical - práctico z Detailed - detallado z Ownership - comprometido © Linda Suskie, MSCHE What do assessment results tell us? ¿Qué nos comunican los resultado de los avalúos? z Do results demonstrate demonstrate… Demuestran los resultados… z Achievement of mission and goals? ¿Logro de la misión y las metas? z Sufficient academic rigor? ¿Rigor académico suficiente? © Linda Suskie, MSCHE Have assessment results been used? ¿Los resultados de los avalúos han sido utilizados? z Have they been appropriately shared & discussed? ¿Fueron compartidos ¿ p y discutidos apropiadamente? p p © Linda Suskie, MSCHE Have assessment results been used? ¿Los resultados de los avalúos han sido utilizados? z Have they led to appropriate decisions? ¿Conducirían C d ií h hacia i llas decisiones d i i apropiadas? i d ? z Curricula and pedagogy – currículo y pedagogía z Programs and services – programas y servicios z Resource allocation – presupuesto y recursos z Institutional goals and plans – metas y planes institucionales © Linda Suskie, MSCHE Where do we see the institution going with assessment? ¿Donde veremos la institución en el futuro con relación a avalúo? z Good assessments are organized, systematized, and sustained. Un buen avalúo es organizado, sistemático, y sostenido. z Does the institution have sufficient engagement & momentum on its own? ¿Tiene suficiente compromiso y velocidad propia? z Will momentum slow after this review? ¿Si esta velocidad desminuirá después de este repaso? z Are there anyy significant g g gaps p in assessment p processes? ¿ ¿Hayy lagunas significativas en el proceso de avalúo? z What Commission action will most help the institution keep moving? ¿Cuál acción de la Comisión seria más beneficiosa para ayudar a la institución a moverse en el proceso? © Linda Suskie, MSCHE
