The IDEA framework - impact-orientated planning, monitoring and evaluation of complex technology projects in educational contexts.
A digital education space for Germany
One of the largest government programmes is the Digital Education Initiative of the Federal Ministry of Education and Research (BMBF). The initiative aims to enable all generations to find their way in the dynamic, digitalised world. Digital learning is to be established in such a way that it can be fully integrated into everyday education. In addition to the development of digital learning tools, the development of the necessary infrastructure is essential for the realisation of the project. A digital education space is to be created, which will be networked with the help of a national education platform.
Digital platforms are playing an increasingly central role. They network products, processes and services and facilitate access to them for the respective target groups. They are ecosystems, so to speak, that can achieve more than the sum of their individual parts. One such platform is the BMBF's planned national education platform. In a time characterised by the platform economy, the development and design of platform projects is therefore increasingly becoming a governmental task, on the one hand through the regulation of standards and legal framework conditions, but also through direct funding, commissioning or participation in platform projects.
At the same time, there is automatically a particularly high responsibility towards the public to account for the funds used. In order to fulfil this responsibility, early impact orientation, which starts in the planning, tendering and awarding phases of public projects, would be an effective means, which is not yet used across the board in Germany. The situation is different in Anglo-Saxon countries in particular, such as the UK or Canada, where impact analyses are an integral part of the monitoring and evaluation of government-initiated projects. In addition, these countries often provide accompanying resources, such as impact evaluation centres or corresponding guidelines and frameworks. These serve as orientation for practice. The Canadian government, for example, publishes standards for the theory-based impact evaluation of public projects and provides supporting resources.
Complex, technology-supported projects, including the BMBF's education platform currently under development, face the challenge of being able to demonstrate their success transparently. In order to be able to evaluate this impact in the educational context both concurrently and conclusively, an evaluation and monitoring concept that is as flexible and agile as possible is required. This allows changes and corrections to be made at an early stage if necessary. Intended and unintended effects should be anticipated as far as possible and checked for feasibility. However, many projects are only orientated towards simple key figures and specifications that have little informative value regarding the impact mechanisms and social effects. For open-ended impact monitoring, it is not possible to fall back on less relevant methodological instruments of evaluation research, nor are there any general blueprints for the impact evaluation of such complex educational technology projects. Existing impact evaluation frameworks used in Germany have a number of shortcomings when it comes to evaluating such projects with different target groups, numerous perspectives and sometimes conflicting impact objectives.
Planning, monitoring, evaluation - structured & formalised
In view of these unresolved challenges, the previously unutilised potential and the exemplary approach of other countries, STAT-UP set itself the task of searching for a solution as part of a research and development project. There is a need for a way in which a modern impact evaluation can be utilised for complex technology projects. The so-called "IDEA framework" thus emerged from the research and development process. The basic idea behind the "IDEA Framework" is to provide the user with a formalised and structured procedure for planning, monitoring and evaluating the impact objectives of complex technological projects in the education sector.
IDEA
Acceptance: What results should be achieved within a certain period of time so that the project can achieve acceptance?
Engagement: Which results should invite you to take a closer look at the project?
Pervasion: What results should be achieved within a certain period of time in order to establish the project as a broadly effective standard?
Impact: What far-reaching social effects are to be achieved?
The reverse order of the initial letters emphasises the approach of systematically thinking about impact "from the end": projects such as the one described as an example must be linked from the outset with an impact idea that formulates the - possibly conflicting - needs and demands of different target groups in impact objectives.
The framework can be used both before and after the start of a project and is based on good practice approaches of contemporary impact analyses. The starting point is scientifically established, theory-based approaches that have been further developed with regard to platform projects and the educational context. Theory-based approaches are characterised by the development of a theory (Theory of Change), which describes within a logic model how social impacts are to be achieved through the project. The theory of change describes the transition from one impact stage to another, i.e. the impact mechanism. Logic of change models make it possible to adequately formalise complex technological projects in terms of formulating and recording their impact. To this end, a model was designed that addresses and corrects the deficits of existing models.
PROCEED IN THE FOLLOWING ORDER:
Formulation of impacts and outcomes with regard to different focus groups
Formulate the derivation of activities (e.g. actions and services) and inputs (e.g. physical and mental resources to realise a project)
Formulation of the Theory of Change, consisting of impact narrative, assumptions, risks and dependencies, which summarises in words how the transition from one impact stage to another takes place.
Finally, the model serves as a starting point for monitoring and evaluating the impact objectives of complex platforms in the education sector. Reflection is used to work out which data should be collected and analysed. Monitoring takes place throughout the entire process. This includes checking the progress of the project and recording inputs and easily measurable outputs, such as the number of participants. Evaluation, on the other hand, takes place at a fixed point in time and measures longer-term effects that are more difficult to record. Collecting data is a time-consuming process. In order to reflect as much as possible, different sources and data should be included, which is known as triangulation. Quantitative information can be supplemented by qualitative information to obtain a clearer picture. In order to determine whether the expected effects have actually materialised, an analysis of the impact contribution is carried out. The aim is to verify a theory of change, confirm stated transitions or revise the theory if necessary.
As part of our R&D project, a framework has been developed that offers a formalised and structured procedure for planning, monitoring and evaluating complex technologies in the education sector and is based on current and recognised methods of impact evaluation.