Boston After School & Beyond offers program partners a common suite of assessment and observational tools to provide detailed information on individual student-level skill development and measures of program quality. Available tools include third-party observations, youth surveys, teacher/program provider surveys, and digital badging (see the descriptions below for more information).
Each program can tailor their involvement in measurement work to their readiness and capacity, and will receive a Program Report for Improvement and System Measurement (PRISM). PRISMs allow programs to compare themselves to a citywide group of their peers and to research-informed benchmarks to determine areas of relative strength and challenge, which in turn enables targeted, data-driven continuous improvement. School year partners receive a mid-year and end-of-year PRISM, and summer partners receive a post-program PRISM in the fall.
Our central database, Cityspan, provides the infrastructure for collecting, analyzing, and reporting on these data. This interactive platform lets us document on a citywide level the number of programs providing expanded learning opportunities and participating in a coordinated system of measurement and evaluation. Similarly, this database enables the tracking of indicators of program quality, student enrollment, attendance, demographic data, and academic and social-emotional outcomes, in addition to supporting digital badging.
Benefits to getting involved in this work include:
- Joining a learning community of fellow program providers with whom to share best practices and collaborate on collective areas of improvement.
- Gaining access to experts in the fields of expanded learning, youth development, and social-emotional learning.
- Receiving dashboard reports that benchmark performance against the previous year and the community as a whole.
- Certified observers conduct day-long visits of programs using the Assessment of Program Practices Tool (APT) developed by the National Institute on Out-of-School Time (NIOST) in partnership with the Massachusetts Department of Elementary and Secondary Education 21st Century Community Learning Center Program. The APT measures research-based components of program quality such as program organization and structure, affective environment, engagement in activities and learning, and social-emotional skill building.
- STEM-focused programs are observed using the Dimensions of Success (DoS) tool designed by researchers at the PEAR Institute instead of the APT. The DoS includes ratings on features of the learning environment, youth development in STEM, activity engagement, and STEM knowledge and practices.
- The available youth surveys allow programs to measure participants’ growth on an array of critical social-emotional skills:
- Using NIOST’s Survey of Academic and Youth Outcomes – Youth Version (SAYO Y), students in grades four and above reflect upon their program experiences and opportunities to develop social-emotional skills such as teamwork, self-regulation, and problem-solving.
- Participants in STEM-focused programs also rate their interest and engagement in STEM at the end of the program by taking the PEAR Institute’s Common Instrument (CI) survey.
- Programs can also administer the Holistic Student Assessment (HSA), also developed by the PEAR Institute, to participants in grades five and up. The diagnostic HSA, used at the beginning of programming, allows students to self-assess their social-emotional strengths and challenges so programs can tailor their content, delivery, and supports accordingly. The retrospective HSA, administered at the end of programming, enables students to self-report their growth in the same social-emotional skills.
Teacher/Program Provider Surveys
- Teachers or program providers assess individual participants’ skills using the Survey of Academic and Youth Outcomes – Teacher Version (SAYO T). This survey measures students’ critical thinking, self-regulation, perseverance, communication, leadership, and academic skills, as well as their relationships with peers and adults. Administering both a pre and post SAYO T allows programs to individualize programming and monitor student growth and achievement.
- Programs can recognize student growth and achievement through our innovative digital badging program. Using For All Rubrics, an online grading and badge-awarding system, program providers can review attendance and SAYO T data to award digital badges for the social-emotional skills measured by the SAYO T. Students can then share these badges in college and career platforms such as the Common Application or LinkedIn. Additionally, program providers may develop program-specific rubrics to evaluate participants’ achievement in the social-emotional skill categories described above and consider these data when issuing badges.
Program partners must participate at the Program Quality Membership Level for at least one program cycle in order to be eligible to participate at the Badging Membership Level.