“The evaluation of educational programmes is of key importance to determine their impact and to ensure good educational practices”

INTERVIEW

Research is a process of creating hypotheses and of their objective verification based on the design of an intervention and the collection and analysis of data. The evidence that is generated allows the hypotheses to be contrasted, reviewed and incorporated into the design of educational policies.

The goal is not only to understand what impact a programme has on the participating children and adolescents, but also to understand the class group. With the evaluation, we can obtain a very accurate knowledge of the impact of an educational measure or policy, discovering everything that is going on, and we can document it, appraising its positive points and limitations so that it can be polished and improved. What’s more, the knowledge obtained in this way can be communicated in Catalonia and around the world in order to spread our findings about what can be done to reduce educational inequalities.

All this is what we want to achieve with the “Educational support” project, Fundació Bofill’s impact evaluation project. We spoke with Caterina Calsamiglia, evaluation director of “Educational support” and ICREA researcher at the Institute of Political Economy and Governance.

Why is educational research so important for contrasting successful models?

Educational research usually makes hypotheses like “if teachers do A, this improves the capacity of boys and girls to do B”. For example, “providing systematic individualised support to students allows the mathematical and reading skills of the whole class group to be increased”. How can this hypothesis be verified? We need to compare very similar groups of pupils which only differ with respect to whether their teachers have carried out such individualised treatment. Subsequently, it is necessary to verify whether the skills of one group are greater overall than those of the other group at the end of the school year. In order to be able to attribute the difference to the fact that the teachers have done A, both the groups of students and the teachers, respectively, should be very similar. The preferred method for ensuring that they are indeed similar is the controlled randomisation method. That is to say, the teachers who do A should be chosen at random from among the group of available teachers.

“The ‘Educational support’ project seeks to provide educational protection for children and adolescents at risk of social exclusion through maximum impact interventions.”

There is a whole part of the research, however, that takes advantage of events that appear to be almost random in nature and that cause two similar groups to undergo different interventions. For example, being born on 31st December or on 1st January may determine whether a student attends secondary school or primary school, or whether a pupil is the youngest or the oldest in his or her class. Obtaining a score of 8.199 or 8.2 may mean the difference between being admitted to one course of study or another, or to one university or another, even though the people who obtained one score or the other could well argue that the two are very similar. The study of the differences between two such groups who will follow different paths owing to a difference of very little significance can help us to understand the impact of certain interventions. Nevertheless, it is almost impossible to come up with quasi- (less rigorous) experiments and answers for all the questions that we pose ourselves.

In Catalonia and in a large part of Europe at large we have had to conduct almost all our educational research on the basis of such quasi-experiments because the performance of controlled randomised experiments had not been within our reach. This year, however, with the financing available to us, we have the chance and the responsibility to do this and to do it well. Now, in order to achieve this goal, we need everyone’s cooperation.

Why is it so important that the programmes should be based on data?

There are two reasons that make research on people particularly complicated. The first one lies precisely in the fact that we are dealing with two very similar groups in different ways, and this is not always easy to accept. The other reason is that we have to be able to follow up the two groups but people are not usually inclined, at first, to grant access to their data. Both these factors, however, different treatment and access to data, are of fundamental importance for identifying the impact of a specific policy.

“Without data we can’t validate the programmes and we won’t be able to achieve the positive impact that we’re seeking for vulnerable children and young people.”

The management of the data for our research is much more secure and controlled than it is in the business sphere. The research institutions legitimated by the EU have to comply with certain procedures, undertake commitments and offer assurances of strict use in relation to the research questions which are requested, and these conditions provide a great deal of security to the individuals involved. Ethics committees and data protection delegates review each of these procedures. For this reason, it is precisely in the context of research that people should feel safest about granting access their data. This is why the team of the “Educational support” project requests people to trust the research institutions and researchers and to cooperate in providing this access, and is open to answer any questions they may have. Without data we can’t validate the programmes and we won’t be able to achieve the positive impact that we’re seeking for vulnerable children and young people.

What were the key elements involved when defining the methodology at “Educational support”? Are there any differences between the programmes?

The most important criterion was that they should be programmes that could have a high impact on the reduction of educational inequalities. Aside from this, the programmes had to be carefully developed in all their details, they had to be apt for implementation within a very tight time frame, and it had to be easy to delimit the treated groups (the ones participating in the programme) and the control groups (the ones allowing us to determine what happens when students don’t take part in the programme). And last but not least, it was important to be able to carry out a follow-up through data reflecting the impact of the intervention. The individualised educational support programmes have proven to be of key importance for reducing the learning gap in the post-Covid recovery process around the world. The development of social and personal skills has been a pending challenge in the educational sector for years.

“The individualised educational support programmes have proven to be of key importance for reducing the learning gap in the post-Covid recovery process around the world.”

With respect to the differences between the three programmes, PENTABILITIES is the result of a European research project (ERC) carried out between 2015 and 2022, which will be validated for the first time with a large-scale controlled randomised experiment. LECXIT, on the other hand, is a programme that Fundació Bofill has been carrying out for over 10 years, while MATH TUTORING is the result of a new design built on an international prospective of evidence-based high-impact educational support programmes.

How has the evaluation been designed to ensure that all the schools obtain some intervention and have access to one of the three programmes?

In the case of primary schools, the evaluation has been built by requesting the schools to volunteer to implement either LECXIT in the 4th year or MATH TUTORING in the 6th year. Once all the candidates had applied and been preselected, we defined at random two groups of schools: one group does MATH TUTORING and the other does LECXIT.

The 4th-year groups of the schools that do MATH TUTORING (which do not receive intervention) act as controls for the 4th-year groups that do LECXIT, while the 6th-year groups of the schools that do LECXIT (which do not receive intervention either) act as controls for the 6th-year groups that do MATH TUTORING.

In this way all the schools take part in some programme but with treated and control groups for the different year levels. By following the progress in reading comprehension of the 4th-year groups and in mathematics in the case of the 6th-year groups of the two groups of schools, we will be able to compare the results of those that have, at random, done either MATH TUTORING or LECXIT.

By following the progress in reading comprehension of the 4th-year groups and in mathematics in the case of the 6th-year groups of the two groups of schools, we will be able to compare the results of those that have, at random, done either MATH TUTORING or LECXIT.

For PENTABILITIES, on the other hand, we have followed a similar procedure while requesting the secondary schools to volunteer for the implementation of PENTABILITIES in two different year levels of Compulsory Secondary Education (ESO), choosing at random the year level in which it is implemented. This randomisation and follow-up of similar groups which receive, however, different interventions, is what will allow us to evaluate the impact of our proposals. The cooperation of the students’ families in consenting to grant access to their data is a matter of fundamental importance for both the groups that receive the intervention and those that do not, in order to obtain a knowledge of the differences produced by the interventions.

What will the analysis of the results be like and what will they show us?

In all cases we will make a short-term evaluation to determine what has happened during the implementation year, but later we will carry out a follow-up of the two groups, based on administrative data, in order to see how the short-term impact affects the course of the education of the two groups in the subsequent years.

Then, in MATH TUTORING and LECXIT, we will evaluate the impact of the intervention on the receiver group of each programme as well as the impact on the class group, since unblocking the learning process of a few students may enhance the relational and learning capacities in the class. The analysis will be made by comparing the reading and mathematical skills of the treated and control groups. Some questionnaires will be used to facilitate the collection of data on how the atmosphere and relations in the classroom change. In the long term we will see how the improvement in these aspects affects the adhesion to and ties with the educational system over the course of secondary school and compulsory education.

In the case of PENTABILITIES, we will be carrying out a great deal of observation in the classroom with a view to determining whether learning is manifested in the way of interacting and working in active learning spaces. Likewise, we will determine whether the improvement in these behaviours is reflected in ties with and adhesion to the educational and working systems and in a lower incidence of risk factors.

What references have been used to give shape to this whole evaluation system?

In the field of medicine and of the experimental sciences in general, the golden rule is to carry out controlled randomised experimentation. In the field of the social sciences, we observe the first experiments of the 1960s in the United States within the framework of the Perry School Project and the STAR Project. These were the earliest experiments to be conducted and their long-term consequences are still studied today.

Ester Duflo and Abhijit Banerjee of MIT, who shared the 2019 Nobel Prize in Economic Sciences, have been among the foremost promoters of the use of randomised experiments in the research that informs public policies in the most widespread way. Their expansion in the last twenty years has been enormous. In 2003 they created J-PAL (Abdul Latif Jameel Poverty Action Lab) to promote this type of research in developing countries. Today they are represented around the world, promoting and advising on this type of research. J-PAL, and specifically Abhijit Banerjee, advised the Ministry for Inclusion, Social Security and Migrations on starting up this initiative of a pioneering nature in Spain and Europe at large.

What will happen when the project and the evaluation period come to an end? What will be the use of the results?

Once we have the results, we will share them with interested schools, families, public administrations, and national and international researchers and institutions. They will have a very big impact since the performance of an experiment on this scale in Europe is quite exceptional.

The data will confirm some aspects that we have intuited, and they will surely provide us with new knowledge and nuances that will be of fundamental importance for determining whether and, if appropriate, how and when to extend the proposals which, according to the data, hold the greatest promise. Above all, however, we hope that these programmes will give us a knowledge of the benefits and difficulties of doing research in Catalonia, informing our own policies and those of the rest of the world in addition to inducing highly qualified researchers to conduct studies on the matters that we are concerned about. Indeed, it is a fact of no little importance that, through these programmes, we have drawn some of the foremost national and international talents to work within our context.

It is a fact of no little importance that, through these programmes, we have drawn some of the foremost national and international talents to work within our context.

If we are capable of facilitating the processes and the access to the data, we will find that the questions that international research seeks to answer are our questions, those which we need to answer in order to develop our educational policy. For this reason, we hope that this will be the start of a path of no return that will allow us to continue to be found deserving of European funds and to draw national and international talent to make Catalonia a leading centre of educational policy and research.
Would you like to find out more?

• LECXIT. Seeing to it that no children finish their education without understanding what they read
• MATH TUTORING. Multiplying the chances of educational success
• PENTABILITIES. For the integration of socioemotional skills into the classroom, giving rise to important life skills
• EDUCATIONAL SUPPORT. Support to speed up educational opportunities
• INTERVIEW Lídia Climent: “Educational support programmes can effectively reduce learning inequalities”
• INTERVIEW Brent Roberts: “We invest a decade in educating children in technical skills. Social and personal skills deserve to be devoted the same amount of time”