Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania The programme evaluation according to the European approach Carmelina Bevilacqua Regione Campania Nucleo di Valutazione e Verifica degli Investimenti Pubblici

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania The new programming period The EU evaluation approach from to The role of indicators update of the mid term evaluation of POR Campania

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania An evaluation project example: the mid term update evaluation of Campania OP  art. 42 Reg. (CE) 1260/1999 provides for each Local Authority carries on mid term evaluation by its updating within 31 December 2005 in agreement with member state and European commission  The objective of update of Mid Term Evaluation is twofold: first, to give recommendations and punctual suggestions for the future programming period, second, to build up a correct logical framework to implement the ex post.

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania The new programming period

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania The basic principles of cohesion reform  From a legislative point of view  Five new regulations:  Three are about how each structural fund (ESRF, ESF, Cohesion Fund) should work within the new regional programmes  One is the general regulation of the European conceal in substitution of the previous regulation (1260/99) in order to define the common dispositions about the three structural funds  The last is the juridical base to use a new tool for territorial cooperation

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania  The new legislative framework drives towards a strong strategic approach:  There are three levels of action programme: 1. the European Commission will fix strategic objectives of socio- economic and territorial cohesion within a framework of interventions of structural funds. 2. Each Member State will translate the European strategic objectives in a National Strategic Reference Framework (NSRF) 3. each Region will construct, on the basis of the previous lines (European and National) the Operational Programme

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania simplification  Great attention on simplification principle:  The co-financing of the cohesion policy is set for the three structural funds instead of six (as well as the previous programming period) with the same administration rules of the Cohesion fund;  The administration of the funds is activate at priority level instead of at measures level  The centralisation policy about eligibility expenditure rule is replaced with national policy  Each Operational Programme must be one fund based (in this way each Region must do three Operational Programmes)  The financial management and control will be more efficient with a stronger distinction between national and European responsability.

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania  The main innovations of the Cohesion policy Reform of the period are:  Strategic approach  Financial system  Administration and control system  Evaluation

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Strategic Approach  The new architecture of structural funds does not distinguish anymore Operational Programme from single programming document but defines an approach more strategic.  Taking into account the European strategic lines, the National Strategic Reference Framework will describe in a very synthetic way the National Strategy and the main interventions and actions in a such a way each region can elaborate its Operational Programme.

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania What ‘s up  Three objectives: Convergence, Regional Competitiveness and Employment, European Cooperation.  The Convergence Objective: collects the European regions less developed is financed by ERSF ESF Cohesion Fund has substituted the previous objective 1  The new version of the Convergence Objective involves a several regions belong to new Member States with different pattern of development, even though all are less developed.

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Priority Objective 1ConvergenceInnovation, Infrastructure, Human Resources, Environmental Risk, Capacity Building, Sustainable Transport Objective 1 Objective 2 Competitivene ss Innovation, accessibility, European strategy for employment, Environmental Risk, InterregCooperationInnovation, Nature, Culture, Education.

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania The National Strategic Reference Frameworks (NSRF) The NSRF represents the opportunity to unify the programming activity of regional policies at European level in order to make a stronger harmony with national policies. The NSRF: ensures the programming coherence with European strategic lines; puts in evidence the links among European, national and regional priorities is the basic orientation to elaborate regional operative programmes. Strategic contents : thematic priority with quantified indicators action to improve the effectiveness of public administration the list of regions involved with respect the two objectives (Convergence and Competitiveness) the executive and operative action to implement the strategic priorities and orientations. The actions to verify the additional request The Member States propose their NSRF to the Europeans Commission after five months to the adoption of European strategic orientations.

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania The EU evaluation approach from to

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Evaluation Art. 45: The strategic guidelines of the Community, the national strategic reference framework, and the operational programmes shall be subject of evaluation. Evaluations shall aim to improve the quality, effectiveness and consistency of Fund assistance and the implementation of operational programmes. They shall also appraise their impact with respect to the strategic objectives of the Community, to article 158 of the Treaty and to the specific structural problems affecting the Member States and regions concerned, while taking account of needs of sustainable development and of the relevant Community legislation concerning environmental impact and strategic environmental assessment.

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania The new draft papers of EU Commission

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania The new draft papers of EU Commission

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania  The rationale of evaluation programme can be focused on two peculiarities:  Strategic: when it is relevant to support programming activities in the entire process according to Community and national priorities.  Operational: when it is relevant to support the monitoring of an operational programme in order to make it be successfull

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania  The Commission estabilishes the evaluation methods and standards to be applied (…), while the Member States shall provide the human and financial resources necessary for carrying out evaluations, organise the production and gathering of the necessary data and use the various types of information provided by the monitoring system.  Evaluation is carried out BEFORE, DURING and AFTER the programming period

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania  Phases of the evaluation ex ante Evaluation : it shall aim to optimise the allocation of budgetary resources under operational programmes and improve programming quality. It shall identify and appraise medium- and long-term needs, the goals to be achieved, the results expected, the quantified targets, the coherence, if necessary, of the strategy proposed for the region, the Community value-added, the extent to which the Community’s priorities have been taken into account, the lessons drawn from previous programming and the quality of the procedures for implementation, monitoring, evaluation and financial management.

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania  Phases of the evaluation Mid Term Evaluation: During the implementation of plans and programmes, it is realised to control the achievement level of mid term objectives (set by indicators of impacts, results and outcomes) and to elaborate mixed adjustment to drive the programme to success. ex post Evaluation : at the end of the implementation of plans and programmes, it is realised to comprehend if the objectives and goals have been reached or not with respect the amount of financial resources used.

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania SF and assessment  Normative sourches: (Evaluation in Reg. CE 1260/99) Evaluation Reg  UE Methological framework: (The MEANS) Evalsed Guide The working papers

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania  Methodology: Working Papers on indicators Working paper 1: Vademecum for Structural Funds Plans and Programming Documents Working paper 3: Indicators for Monitoring and Evaluation: an indicative methodology Working paper 9: The update of the Mid Term Evaluation of Structural Funds Interventions New draft (X) on programming

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Indicators for programmes assessment

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania  Evaluation process elements: Evaluation project Indicators system

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania

Oggetto dell’analisi valutativa Domande valutativeIndicatore per ciascuna domanda o sub- domanda Metodologia per la raccolt a dei dati FontiGli elementi conoscitivi che l’analisi deve individuare b.5: Politiche di inclusione sociale Esaminare i risultati prodotti dai servizi e dalle infrastrutture dirette a promuovere l’inclusione sociale e la conciliazione tra vita familiare e esigenze di lavoro. Verificare l’efficienza e l’efficacia delle iniziative programmate contro la povertà e l’esclusione sociale: Individuare possibili meccanismi di coordinamento tra le attuali procedure di attuazione delle politiche di inclusione e lo strumento del reddito di cittadinanza Contribuire alla definizione delle policies di inclusione/protezione per la prossima fase di programmazione. Indicazioni di cambiamenti rilevanti nei livelli di benessere, nei comportame nti, aspettative, preferenze dei destinatari finali distinti –ove possibile- per genere. Cambiamenti negli indicatori di  Accesso  Utilizzo  Condizioni di utilizzo  Qualità dei servizi/prodo tti finanziati dal POR Eventuali altri indicatori specifici Grado di diffusione e aderenza alle procedure VISPO Analisi cartolar e dati second ari Analisi dati primari da focu group Interviste a testimo ni privileg iati Studi e ricerche di settore Documenti ufficiali regionali (strategie, valutazioni, monitorag gi) Documenti relativi a progetti finanziati Cosa funziona/non funziona a livello di singolo intervento di inclusione sociale? Dove? In quali contesti –territoriali/ istituzionali, le politiche di inclusione funzionano meglio/peggio? Perché? cosa giustifica le diverse performance/impatto di una stessa tipologia di intervento? Chi? quali categorie di soggetti traggono maggiore/minore beneficio? (per classi di età, estrazione sociale, genere) Come? quali meccanismi attuativi delle misure FSE concorrono al successo/insuccesso delle politiche sociali? Es.: piani di zona sociali, criteri di selezione. ecc.

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Indicator measures An objective to be reached The activation of resource The achievement of outputs The effects reached The variable of a peculiar phenomenon of the territorial context. The indicator system

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania The information given by an indicator is a quantitative measure of variable that represents an important phenomena to control (e.g. percentage of assisted SME by public interventions, percentage of educated people that are satisfied by educational programme, etc..)

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania An indicator must give simple information that should be easy friendly in order to be correctly used. The indicator should help public administration in the decision making process, therefore it must be linked to a success criterion (e.g. the criterion improving quality of life)

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania The indicator and its measure unitb must be sensitive, that is the quantified measures must change in a significant way when the change of phenomenon they represent occurs.

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania output indicators : variables that directly check the achievement level of physical objectives of measures (e.g. km of road, number of attended people to educational programme, kmq of equipped areas) result indicators: variables that check if the necessary condition to reach final objectives have been implemented. Impact indicator : variables that check the achievement of final objective (the improvement of welfare, the reached level of development) There are no direct link with each measure of the Operational Programme. They occur with the realisation of many output and result objectives.

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania

“CONVERGENCE” AND “COMPETITIVENESS AND EMPLOYMENT” OBJECTIVES Programme level (1) Jobs created (full time equivalents) (2) for men; (3) for women Thematic fields Research and knowledge economy ( ) (4) Number of RTD projects (5) Number of cooperation projects enterprises – research institutions (6) Research jobs created (preferably 5 years after project start) Direct investment aid to SME (08) (7) Number of projects (8) of it: number of start-ups supported (first two years after start-up) (9) Jobs created (gross, full time equivalent) (10) Investment induced (million €) EC core indicators

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Information society ( ) (11) Number of projects (12) Number of additional population covered by broadband access Transport investment (16 – 32) (13) Number of projects (14) km of new roads, (15) of which TEN (16) km of reconstructed roads (17) km of new railroads (18) of which TEN (19) km of reconstructed railroads (20) Value for timesavings in Euro / year from new and reconstructed roads for passengers and freight (21) Value for timesavings in Euro / year from new and reconstructed railroads for passengers and freight (22) Additional population served with improved urban transport

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Renewable energy (39-42) (23) Number of projects (24) Additional capacity of renewable energy production (MWh) Environment (44-47, 49) (25) Additional population served by water projects (26) Additional population served by waste water projects (27) Number of waste projects (28) Number of projects on improvement of air quality (29) Area rehabilitated (km2) Climate change (16-19, 39-43, 48, 51, 52, 62) (30) Reduction greenhouse emissions (CO2 and equivalents, kt)

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Prevention of risks (53) (31) Number of projects (32) Number of people benefiting from flood protection measures (33) Number of people benefiting from forest fire protection and other protection measures Tourism (55-57) (34) Number of projects (35) Number of jobs created Education (74) (36) Number of projects (37) Number of benefiting students Health (75) (38) Number of projects

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Urban issues Physical and environmental regeneration (39) Number of projects ensuring sustainability and improving the attractiveness of towns and cities Competitiveness (40) Number of projects seeking to promote businesses, entrepreneurship, new technology Social inclusion (41) Number of projects offering services to promote equal opportunities and social inclusion for minorities and young people

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Cross-border co-operation and Transnational co-operation degree of co-operation (42) Number of projects respecting two of the following criteria: joint development, joint implementation, joint staffing, joint financing (43) Number of projects respecting three of the following criteria: joint development, joint implementation, joint staffing, joint financing (44) Number of projects respecting four of the following criteria: joint development, joint implementation, joint staffing, joint financing

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Cross-border co-operation Number of projects (45) encouraging the development of cross-border trade (46) developing joint use of infrastructure (47) developing collaboration in the field of public services (48) reducing isolation through improved access to transport, ICT networks and services (49) encouraging and improving the joint protection and management of the environment (50) Number of people participating in joint education or training activities (51) Number of people getting employment on the other side of the border as a result of CBC project

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Transnational co-operation Number of projects (52) on water management (53) improving accessibility (54) on risk prevention (55) developing RTD and innovation networks Inter-regional co-operation (56) Number of projects

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Case Study: The Update of Mid Term Evaluation of POR Campania

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania

Thematic appraisals Programme evaluation Recommendations for the future programming Objectives and structure

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania The Report is structured in two parts “Programme Evaluation and Recommandation for the future programming period with four appendixes Appendix I – The POR Measures Appendix II – The indicators of impacts, results and outcomes with respect to each typology of interventions Appendix III – GIS Appendix IV – Financial analysis by Fund Objectives and Structure

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Objectives and Structure The Upadate of Mid Term Evaluation - The expenditure capacity reached at 31 October 2005 with respect also the expenditure realised at mid term evaluation deadline -The capacity to put together the expenditure control and monitoring activities with assessment procedures of investment quality by using the indicator system (impact result and output) PROGRAMME EVALUATION puts in evidence the actual performances of the programme with respect its implementation RECOMMENDATION FOR THE FUTURE PROGRAMMING Strategic reconstruction of regional orientations RECOMMENDATION FOR THE FUTURE PROGRAMMING Strategic reconstruction of regional orientations Individuation of the weight of POR action in order to achieve the fixed development objectives and the level of complementary of the POR with respect to the other programmes Defining strength and weakness for each sector according to the fixed objectives during the implementation phase Analysing the sectors in the light of new orientation documents for the future programming period The capacity of the programme to respond to the development priorities demanded by regional context. -The capacity of problem analysis e problem solving of public administration level with respect to expenditure trends. -The capacity of vertical and horizontal integration inside different level of governance

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania The approach of evaluation process takes into account that it is not possible to modify or to adjust in a structural way the programme in this phase of its implementation. In other words, it is not possible to suggest policy change or strategy modification. That kind of changes can be suggested only for the future programming period. Programme Evaluation

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Priority Areas of interventio n of Structural Funds General Features Specific features Financial performance Success Factors Foreseen results at the end of the programme Infrastructure development Business development Human resources development 1 Environment 2 Transport 3 ICT 1 SME 2 R&D 1. Job creation 2. Social inclusion 3.Gender mainstreaming 1. Job creation 2. Social inclusion 3.Gender mainstreaming Target of the OP Measures Decrease of unemployment rate about of 10% Increase of GDP rate about of 66-68% of the national average Increase of the infrastructures almost equal to the average of South of Italy Increase of the Work Force Units about of 3% in business enterprises Improvement of environment quality Increase of women participation in the labour market Financial output indicators Physical output and result indicators Input Output Matrix / Multiobjectives model SP = Sustainability of the residual programmed expenditure SOf = Sustainability of the residual programmed outputs to be reached SOr =Sustainability of the residual programmed results to be reached SOf = Sustainability of the residual programmed outputs to be reached SOr =Sustainability of the residual programmed results to be reached How success factors affect the expenditure capacity Level of attention Expenditure multiplier SPo = Sustainability of the residual programmed impacts to be reached Financial prospect Principal Feature Self Assessment 2005 Main Interventions Recommendations and suggestions to perform the expenditure objectives together with outputs and results

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Priority: Sustainability of the residual programmed expenditure  The programmed cash flow trend shows a strong slope at the end of 2004, mainly due to the procedure of Integrated Projects that has required a hard programming effort at the beginning

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania The evaluation has been structured by structural funds intervention areas in order to understand how the program has invested its resources in terms of physical outputs and expected results by classical investment categories of development policies. That is, infrastructure, business SME, human resources. The Working Paper N°9, focusing on the importance of these investment categories, defines an essential set of core indicators for each of them: 1.Infrastructure Development Transports/ Roads; Transports/ Railways; Environment; 2. Information Society 3. Business Development Development of SME Research and Development 4.Human Resources Development

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania OP measures have been aggregated by intervention areas. The expenditure data have been collected from information system MONIT and concern both programmed payments and real payments. The level of attention has been assessed for each measure taking in account the residual capacity to perform the programmed outputs and results (2008). The level of attention has been graduated in a range of percentages as following: 0-10% High level of attention  10% - 25% Medium level of attention  More than 25% Low level of attention 

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania

The following table shows a synthetic measure of the level of attention for each intervention area (according to the Working Paper n°9). The example refers to the case of Development Infrastructure

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania The average financial size of interventions shows a general dispersion of resources except for transports.

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania The following table shows an example of assessment report of level of attention for some measures concerning infrastructure development area

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Obiettivi- target del POR Decrease of unemployment rate about of 10% Increase of the Work Force Units about of 3% in business enterprises Increase of GDP rate about of 66-68% of the national average Improvement of environment quality Increase of the infrastructures almost equal to the average of South of Italy Increase of women participation in the labour market Impacts: Sustainability of the residual programmed impacts to be reached

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Recommendations for the future programming period Building up of the logical framework of regional strategies EU and national documents concerning the future programming period Lessons from the past and suggestions

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Recommendations for the future programming period

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Recommendations for the future programming period

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Assessment of policies integration Assessment of policies concentration Assessment of location suitability GESTIONE DELLA CONTINUITA’ DEL PROCESSO DI VALUTAZIONE EX ANTE – IN ITINERE – EX POST Appendix III: GIS

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Appendix III: GIS

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Appendix III: GIS

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania Appendix III: GIS

Carmelina Bevilacqua – Nucleo di Valutazione e Verifica degli Investimenti Pubblici della Regione Campania