Student Achievements’ Monitoring

SAM is developed for the following group of stakeholders:

• Teachers: teachers of the primary school, subject teachers, math teachers.
• School administration
• Educational authorities at different levels: city, region, republic

Teachers: teachers of the primary school, subject teachers, math teachers.

SAM indicators that can be particulary interesting for teachers are:

• Test score for student or class
• Tier achievements for student
• The distribution of students by levels
• Achievements profile for student or class
• Matrix of primary estimates for each task for student or class.

Primary school teacher gets big amount of useful information to work with. Familiarization with SAM provides teacher with guidance to understanding of  the main points of the learning process: the technological matrix of the test, which defines the basic units and topics of subject content, three-level scheme modeling the process of assignment, type of task, structuring an array of educational material. SAM allows to assess the outcomes of their activities, to analyse the successes and failures and outline a program to work with.

For teachers of mathematics  the main interest in SAM results is the distribution of children by stages of development in primary school. This index allows teachers to get an idea of what part of the class is likely to experience difficulties in mastering new concepts. It’s the distribution, gives  grounds for defining the pedagogical strategy, namely, the distribution of academic content in time, forms of content presentation, organization of interaction in the classroom, etc.

School administration

For the school administration the interest will be in :

• Test score
• The distribution of students by proficiency levels for student or class
• Test score for each section of the program
• Distribution by proficiency levels for each section of the program

SAM results allow school administrators to compare the classes between themselves in terms of educational success, and to compare the level of urban indicators and the socio-cultural norm. Possession of this picture allows the administration to evaluate the objective position of their school in the educational field of the city and region, to represent the likely prospects of their students in mastering the high school program and enrolling in higher education.

Educational authorities at different levels: city, region, republic

For the educational authorities there might me interested the following test indicators:

• Test score
• Distribution by proficiency levels
• Test score for each section of the program
• Distribution by proficiency levels for each section of the program

SAM results in the city/region describe the general level of educational outcomes of the school network in the region.  A comparison with the average indicators in the region is allowed, as well as similar indicators from other cities/regions that allow general comparative evaluation of training of students of the local school system.

The average test scores of schools and classes  allow  to assess the success for the urban background data, as well as in comparison with other schools and classes, both locally and in other cities. Within the objectives of management of the educational system it is important to identify the classes with success, appropriate social and cultural norm in order to identify effective teachers who are able to share useful experiences and bring them to the various educational activities in a city.

The project on SAM development was supervised by the Expert Council that included Dr. Victor Bolotov (former vice-president of the Russian Academy of Education), Dr. Galina Kovaleva (the head of the Center for educational quality assessment of the Russian Academy of Education), Dr. Tony Orgee (Cambridge council chairman), Dr. Henk Moelands (director of international department of the CITO institute in Holland), Prof. of the Manchester University William Boyle, and also leading educational consultants of the World Bank.  The Expert Council meeting met once a year to evaluate the process of SAM development and validation.

After  SAM was validated, the Expert Council made a decision to conduct it’s international expertise. The results of the international evaluation of SAM were received in November 2013. The experts who conducted the expertise included:

• Howard Everson ( professor of the psychology of the Center for educational research of the NYU).
• Clancy Blair (professor of applied psychology of he Steinhardt school of the culture, education and human development of the NYU).
• Dr. Bas Hemker (leading researcher of the National institute of testology of Netherlands)

Experts evaluated following technical documentation for the SAM in English:

• Technical manual
• User guide
• Test specification
• Math test items
• Validity study

The overall results of the evaluation of the approach to SAM development were good. Experts highlighted the impressive volume of the conducted research  and necessity of the continuation  of data collection. Experts also noted the importance of the research for SAM scaling.

In overall the work on SAM was  evaluated highly. The sampling of  12000  students allows  accurate claiming of the psychometrical quality of the test to be good.

In the course of 2011 – 2013 developers conducted research on test validity. It was conducted according to Dutch system  COTAN. The research was conducted on content and construct validity.

The validity of the construct consisted of two components – rationale of measurement properties of tests and the justification of how the test adequate to the object and purpose of the testing. In the SAM model, the main construct, which was the subject to verification, is a three-tier taxonomy of learning content that is built into in the age context.

In the study, it was found that tasks within the same block, theoretically meet the criteria of the three levels, show the corresponding hierarchy of constraints, i.e., display logic functional genesis in accordance with the accepted taxonomy, which is evidence in favor of the validity of this construct.

The results of the study also allow you to trace the age dynamics of acquisition of educational content. By the end of primary school (4th grade) the second stage of  academic content acquisition  is dominant.  At the end of primary school  the 3rd level dominates. This supports the theoretical model of the SAM, i.e. the evidence in favor of its validity.

The main method of the study of the validity of the test content was the expert evaluation of the content of each job separately and the content of the test in general. Math test includes basic areas of mathematics represented in primary school curricula. Test on the Russian language includes the basic sections of Russian grammar presented in the primary school curricula. Development and testing of test tasks also included consultation with primary school teachers and subject teachers on test content. At the final stage, an expert evaluation of the content of both tests by external experts was conducted.

Thus, an independent examination of the SAM, allows us to draw a conclusion about the readiness of the  tool for mass use, its accuracy and reliability.