The CCRI worked with ADAS (lead partner), Fera and and Pareto Consulting on this project, as part of a team which had extensive experience of RDP evaluation, key expertise in quantitative and qualitative research methods and in-depth knowledge of the agricultural sector, agri-environment schemes and rural economic development. The research was commissioned by Defra and commenced in February 2016. Janet Dwyer, John Powell and Katarina Kubinakova were the CCRI team that worked on this project and the final report was submitted in November 2016.
Overview of research
The Rural Development Programme for England (RDPE) helps contribute to the delivery of Defra’s objectives of a cleaner, healthier environment, a world leading food and farming industry, a thriving rural economy and a nation protected against natural threats and hazards.
The Ex-post Evaluation of the RDPE 2007- 201, reported on the impacts, effectiveness and efficiency of RDP expenditure and to demonstrate what has worked well and represented a major source of policy learning for subsequent programmes.
The evaluation process drew on evidence from a number of sources to consider each of the evaluation questions. To aid this process, judgement criteria and summary templates were devised to help structure each answer, supplemented by consultation and feedback from policy leads, delivery bodies and wider stakeholders.
Key data sources were:
- Scheme data: Axis and measure-level data for expenditure, outputs and results, with project-level data for socioeconomic measures.
- Evaluations and other research: Previous evaluation reports for the RDPE and its component measures and schemes (as available).
- Primary research: A face-to-face beneficiary survey, with a sample of 200 beneficiaries, to assess impact and calculate Social Return on Investment (SROI). Modelling work was also undertaken to quantify environmental impacts of agri-environments schemes.
- Stakeholder consultations: Interviews with national and regional stakeholders including policy leads, delivery agencies, industry bodies and NGOs to provide a perspective on programme balance, effectiveness and efficiency.
Evaluation findings
The evaluation of programme, axis and measure level elements of the RDPE were captured in answers to Programme evaluation questions, detailed in the final report. These are presented in overview here, under the key evaluation themes set out in the guidelines for the ex post evaluation of 2007-2013 RDPs.
Relevance
Measures deployed under the RDPE encompassed most (but not all) of the range available under the Rural Development Regulation; stakeholders generally regarded the measures deployed as consistent with the programme objectives and hence the identified and prioritised needs. Flexibility within the programme allowed adjustments to measures to meet changing needs and address emerging issues.
The RDPE represented a major source of funding to deliver land use change and management to deliver on priorities for biodiversity, soil and water quality and climate change, in support of wider policy, for example, Biodiversity 2020 and the Natural Environment White Paper.
In contrast, the contribution of RDPE to the growth of the rural economy and employment was limited by the scale of actions and the fact that socioeconomic measures were not solely directed towards economic objectives. In particular, there was a focus in Axis 1 on resource use efficiency, improving animal health and welfare and nutrient management, in order to deliver improved competitiveness in agriculture. The main employment impact was from the Axis 3 measures 311, 312 and 313. However, RDPE support is likely to have increased the variety of employment opportunities through economic diversification of rural areas and improved the quality of existing jobs through investment in technology. Increases in part-time, casual and seasonal jobs were greater than in full-time jobs.
In terms of addressing needs, the majority of RDPE measures were relevant to, and addressed the needs of, direct programme beneficiaries. Feedback from beneficiaries was consistently positive with project objectives seen as fully or mainly met. A significant number of modifications were made throughout the programme period, some to address changing or emerging needs of the farming and forestry sector and the rural population while others were made in response to wider institutional change arising from political processes. Some modifications were in response to specific events, notably the Farming Recovery Fund, which assisted businesses to recover from flooding and the Dairy Fund which supported cooperative action to address the dairy crisis. Funding for flood recovery offered short-term relief but there was limited ability to increase resilience, while the Dairy Fund had more resonance with strategic RDP goals. For both issues, wider support, largely outside of RDPE, was also needed.
Effectiveness
RDPE objectives were set in terms of protecting and enhancing the environment, establishing competitive and sustainable rural businesses and fostering thriving rural communities.
In terms of the environmental objectives, progress has been made across all areas, biodiversity, high nature value farming, water quality and climate change mitigation, but this is not fully reflected in the CMEF impact indicators. This is due partly to external factors and the scale of environmental challenges, but also the difficulties in evidencing cause and effect in a national scale open system.
Nevertheless, an ongoing programme of monitoring and evaluation provides valuable evidence on how effectiveness can be improved and the Programme has been responsive in this respect. Key issues previously highlighted include a high element of deadweight in the Entry Level Stewardship (ELS) scheme and a need for greater spatial targeting of options which were addressed in the RDPE 2007-2013 and in the following Programme.
In terms of rural businesses and communities, the three impact indicators related explicitly to the competitiveness objective: economic growth; employment creation; and labour productivity. While baseline indicators for all these metrics have risen over the programming period and there is good evidence of impact at beneficiary level, the absolute magnitude of the effects for Axis 1 (growth and employment, improving competitiveness of farming and forestry) and Axis 3 (rural communities, rural quality of life and growth and employment in the rural economy) is modest relative to the size of the rural economy and of the agricultural and forestry sectors. This largely reflects the comparatively low share of RDPE expenditure devoted to these objectives but also reflects the influence of external factors, such as the economic recession and wider Government funding for basic services (transport, broadband, etc.).
As with all market interventions, there was an element of deadweight associated with business support, particularly with grant aid. This reflects the fact that business grants require match-funding so tend to attract projects which are financially viable in their own right and which therefore could have been achieved – though more slowly or less completely – without the grant. Conversely, grants to community organisations tend to be at higher rates and in situations where there are few alternatives to fundraising to enable these things to happen. While deadweight reduces the net impact in terms of GVA (Gross value added) and job creation at beneficiary level, the evidence indicates that there are significant multiplier effects. For community actions, deadweight was generally low, in view of the dependence on public funding, and the high social return on investment (SROI) also reflects a broad beneficiary base.
Efficiency
In terms of funding allocation, most (revised) targets for spend were largely achieved or exceeded. Targets were also achieved for most result indicators with the resources allocated. Achievement of impacts and contribution to community priorities were less obvious due to limitations of data availability, measurement methodologies and attribution issues and measurement. Efficiency measurement is further limited by lack of benchmark figures generally, more so for Axis 2 impacts. The ex-post beneficiary survey for socioeconomic measures suggested that M312 delivered good value for money in terms of job creation while the SROI analysis highlighted the wider social impacts of M313 and M121 in particular. Projects delivered through LEADER also generated high SROI ratios of around 3:1.
There is scope for future support programmes to improve efficiency and reduce cost through better targeting and minimising deadweight (including displacement) of programme-supported actions. Evidence from the evaluation suggests that partial deadweight exists across all measures but is most evident for business investment grants, those training programmes where there are high levels of alternative courses, and Axis 2 measures (particularly for 214, 221, 223 and 225).
Cost-efficiency of different delivery mechanisms varied across different measures and schemes within RDPE. A comparison of cost per result (Table 11) shows a wide range of values, often reflecting multiple priorities and approaches rather than efficiency per se. Evidence from the regional stakeholder consultations suggests that there was scope to improve efficiency and reduce administrative costs by simplifying project approval processes to ensure proportionality between the significance of funding and level of scrutiny. While transaction costs (both for applicants and administrators) can be mitigated by simplifying application processes, and fewer, larger projects offer better process cost efficiency, this is not the primary consideration. Rather, the relative effectiveness of small and large grants at securing desired outcomes is a more appropriate guide. Given the wide range in scale and type of rural firms and of investment needs/opportunities, it is likely that a mixed approach is suitable, encompassing large, transformational (infrastructure) type investments as well as more modest, operational type investments to support individual business competitiveness.
A range of factors influenced cost-efficiency of the programme delivery, which include: engagement with the programme (uptake); changing external conditions (policy, regulation, macro socio-economic and environmental conditions etc.); delivery arrangements and mechanisms; decision-making process; pre-existing levels of human and social capital including knowledge, capacity, mutual understanding, trust and credibility between different parties.
Results and impacts
The performance of RDPE in terms of the CMEF (economic) impact indicators was as follows:
1. Economic Growth: Drawing on EKOS (2015) evaluation findings, it is estimated that the RDPE contribution to rural economic growth was an estimated €1bn to €1.2bn (c. £0.8bn to £1.0bn). This represents c.3-5% of the overall real GVA growth in rural areas (depending on how they are defined) over the period 2010-2015, after the economy moved out of the recession following the financial crash. These figures do not include all Programme activities, notably no Axis 2 measures, and hence may under-estimate the overall GVA contribution.
2. Employment creation: Ex post analysis of RDPE scheme data indicates (where recorded) that over 11,000 Gross jobs were created through Axis 1 and 3 interventions whilst EKOS’ (adjusted) estimate is higher at around 19,000 Gross jobs created. However, employment creation arising from training effects or from Axis 2 activities is not recorded – hence total Gross job creation may be higher. EKOS estimates further show that deadweight and displacement effects lead to lower Net job creation figures amongst direct beneficiaries, but that multiplier effects compensate for this to give overall higher Net job creation estimates (adjusted) of around 20,000 to 23,000.
3. Labour productivity: While EKOS results suggested a slightly negative (-3%) Gross impact on GVA per full time employee (FTE), the ex post beneficiary survey suggests a strongly positive effect, averaging a Gross GVA gain of around £13k (25%), split more-or-less evenly between increased output and lower costs. The main impact has been from investment in new technologies, processes or products.
Reflecting the prominence given to the environmental objective, four of the seven impact indicators relate to environmental issues. These are considered as follows:
4. Farmland Bird Index: The CMEF indicator for reversal in biodiversity decline remains in decline. However, reference to a wider set of biodiversity indicators shows that the Programme has supported biodiversity effectively to some extent, with short term “stability” for widespread bats, woodland birds and butterflies. Evidencing cause and effect is very difficult at a national scale but there is good evidence of impact at a more local scale.
5. High nature value (HNV) farmland: The area of HNV farming in England has yet to be defined, so progress cannot be formally calculated. However, indicators ‘% SSSIs in favourable condition’ and ‘Area of new native woodland created’ show that positive progress has been made to this end.
6. Water quality: Water quality indicators are stable but do not necessarily reflect the impact of RDPE due to wider influences. Modelling the effect of agri-environment effects demonstrated modest reductions in pollutants, while a ‘difference in difference’ analysis of nutrient balances found that the gross nutrient balance may also be lower on in-scheme farms. Other modelling work indicates that Axis 1 measures (through Catchment Sensitive Farming) are contributing to decreasing Phosphate and faecal matter in water bodies.
7. Increase in production of renewable energy: This impact indicator for combatting climate change is at best a partial measure of mitigation efforts. Targets for production of renewable energy under RDPE have not been fully met and the Energy Crops grant was stopped after the mid-term. However, other Measures have contributed to reduced greenhouse gas emissions, notably through reduced livestock numbers and nutrient use under agri environment schemes, but also through Axis 1 actions to improve input use efficiency and promote animal health and welfare. Adaptation actions such as water storage have been limited but helpful.
The impact of RDPE interventions is subject to a range of external influences, including weather conditions, on-going technological change, market forces and government policies beyond the RDPE. In particular, the programming period coincided with the aftermath of the financial crisis, the imposition of government austerity measures and prolonged poor macroeconomic conditions. Consequently, attributing changes in, for example, overall economic output or environmental change to the RDPE rather than other influences is difficult. Nevertheless, expenditure under the RDPE represents substantial funding to rural England.
Janet previously worked with some of this research team to deliver a successful Ex Post Evaluation of the ERDP 2000-2006 .
