By Dana Cristina Repede
Member, Young and Emerging Evaluators group of the European Evaluation Society (yEES!)
This blog is part of the Eval4Action ‘Walk the Talk’ blog series. The series details six nominated actions for influential evaluation that were contributed during the Walk the Talk drive, held in October 2021. These lessons and reflections inspire greater action for influential evaluation in the Decade of Action.
Dana Cristina Repede interviewed Virgil Pamfil, the acting President of RoSE (Romanian Society of Evaluators) to unpack the VOPE’s efforts to professionalize evaluation in Romania, notably through developing occupational standards and implementing a code of conduct for evaluators.
Professionalizing evaluation
Dana: In your video for Eval4Action’s Walk the Talk you speak about RoSE’s endeavor to professionalize evaluation in Romania. What does professionalization of evaluation mean and what are the few steps that RoSE took to that end?
Virgil: On one hand, the discussion about the professionalization of evaluation should be put in terms of developing an appropriate enabling environment. This may involve designing, adopting and implementing legislation and/or policies to institutionalize national evaluation systems, but also institutionalizing the profession through different elements such as developing occupational standards, codes of conduct etc. In Romania, the occupational standards for evaluators were thought to make a major contribution to the development of the profession because they participate in the design of a large array of activities within evaluation: performance, workforce design, etc. through standardizing them. Occupational standards have implications not only on productivity, but also on the development of human capital. Therefore, by enabling the basis for the development of educational schemes, such as initial and continuous learning for evaluators, the standards ensure that there is a link between human capital investment and productivity, the market’s needs, etc. In line with this, RoSE has been certified as a training provider since 2013 and delivers training and mentoring to evaluators.
On the other hand, professionalization has to be developed at individual level through strengthening the capacities of individuals’ knowledge and skills. In other words, the personal drive of an individual to continuously improve, and to be responsible or accountable for actions taken, is especially significant in the evaluation profession.
RoSE considers itself as an active actor that participates in building an evaluation culture because of its two-tier strategy of taking action at institutional and individual level. RoSE has been a catalyst for action through its activity of knowledge sharing and networking. In this respect, the three projects implemented in partnership with VOPEs from North Macedonia, Albania, and Kosovo under the peer to peer programme supported by International Organization for Cooperation in Evaluation (IOCE) is a key milestone in RoSE interventions.
Dana: What is RoSE’s roadmap for continuing to professionalize evaluation in the future?
Virgil: This is difficult to say because it is still a long way from building an evaluation culture in Romania as the National Policy, Strategy, and Action Plan are still pending. However, as a EU Member State and a beneficiary of EU Funding Instruments, Romania has to comply with a complex regulatory framework that defines requirements to evaluate impacts of the EU programmes. While on one hand, the evaluation is driven by the EU level law, on the other hand, the Member States can make use of their national approaches. It is therefore up to each country to support the evaluators’ job market and profession in accordance with market needs. In Romania, there is an increasing need for evaluators to cover various areas of interventions in sustainable development to measure the achievement and performance of policies, programmes and projects. However, not having specific evaluators profiles that cover these areas included in the Code of Occupations prevents Romanian evaluators from participating in certain calls for experts as they would not comply with the requirements.
Dana: Is evaluation in Romania recognized as an established profession (shared purpose, common identity, common agreement of the practitioners on the responsibilities and characteristics of the profession)?
Virgil: More or less! In Romania evaluation is very much considered as a cross-cutting occupation. The evaluators come from different fields of expertise. Therefore, they are assigned in most cases on a short-term basis to do a job that complies with their previous work records. It is like the problem with the chicken and the egg – who was first? Without having specific previous working experience an applicant evaluator won’t win the assignment, and if they don’t get assignments an evaluator cannot gain working experience. The academic and continuing education opportunities for evaluators are leading to higher competition and fewer roles.
A framework for defining occupational standards
Dana: How do you do an Occupational Standard framework for the field of evaluation? Please share your experience and recommendations.
Virgil: In Romania, the process of establishing the Occupational Standards was rather long because of the many stages that had to be followed in accordance with the law and the standard procedure enforced by the Romanian Authority of Qualifications. It took about one year, and fortunately it was fast-tracked thanks to RoSE’s internal expertise for drafting occupational standards.
The first step was to collect data on the occupation and process them under an occupational analysis in order to convince the national authorities of the importance of our action. Once the occupational analysis was endorsed, the next step was to identify the necessary key competencies structured by elements of competence, knowledge, skills, and behaviours, working contexts, and the range of variables. Afterwards, the Draft Occupational Standard was subject to audit and assessment by several authorities, such as the Government's Sectoral Committee and the National Authority of Qualifications. Finally, after about 12 months of work, in 2012, the Occupational Standard was published online by the National Register of Qualifications, as a reference for education and certification of competencies for project evaluators.
Dana: Which are the evaluator competencies included in the Occupational Standard for Project Evaluators (OSPE)?
Virgil: These were structured into three categories, namely: key competencies, competencies that cover related occupations and specific competencies. There are five specific competence units as follows: individual preparation for evaluation, organization of evaluation process, project evaluation, validation of evaluation results and provision of technical assistance for selecting the projects. Each competence unit has set the level of responsibility and autonomy, and it is structured by elements of competence, criteria for achievement of the associated results and tasks. They also provide specific information on the working conditions, range of variables and compulsory knowledge.
It is worth highlighting that the OSPE designed by RoSE refers to evaluators of projects, due to limitations in the Code of Occupations of Romania (COR), which does not include the positions of Evaluator of Programmes and Evaluator of Policies. Even though it is the Government’s responsibility to include these occupations in the COR, there is a lack of awareness about the different profiles of evaluation professionals. With effective lobbying, the occupational framework could be diversified with a larger array of evaluators profiles.
Code of professional conduct
Dana: Why is a Code of Professional Conduct important for the evaluation profession?
Virgil: In Romania, when selecting a person to provide evaluation services, only a statement of no conflict is requested. Unfortunately, it does not include rules about behaviours, values, and decision-making skills. Therefore, RoSE as a professional organization has decided to define its own Code of Professional Conduct to set standards and expectations for its members. We consider that the Code of Conduct of evaluators is a mandatory addition to the competency framework.
Dana: How difficult is it in Romania for an evaluator to adhere to the Code of Professional Conduct?
Virgil: There is no obligation in Romania for an evaluator to adhere to a Code of Professional Conduct. As far as I know, no other organization formally requires the evaluators to adhere to a Code of Conduct, but only to sign a statement of no conflict of interest.
As a professional VOPE, RoSE’s priority was to define its professional culture quickly, to set standards and expectations to enable its customers and partners to know its values,and to create a level of transparency for a healthy business relationship.
Evaluation of Sustainable Development Goals (SDGs)
Dana: How can the professionalization of evaluation be beneficial in the national review system of SDGs?
Virgil: It is indisputable that professionalization of evaluation is beneficial and it is a must for the national review system of SDGs. Two elements are necessary for an effective SDG implementation and monitoring: good coordination at the level of government because the process is expected to be country-led evaluation work, and a multi-level perspective with functional interlinkages and coordination between all actors.
The importance of evaluation in the process is that it generates evidence and evidence can contribute to strategies to operationalize the SDGs and inform policy and management decisions. However, adequate evaluation capacity is needed to ensure the quality of measuring SDG targets. This means that each country, and Romania in particular, must develop an effective professional framework for evaluators with clear profiles that cover the entire Policy Intervention Level (policy, programme, project, activity) at the government and civil society levels.
Dana Cristina Repede has a Masters in Monitoring and Evaluation from the University of Saarland and Masters in Sociology from the Université Libré de Bruxelles. Currently she works as an M&E consultant with Ann-Murray Brown consultancy. Dana has worked for the European Commission holding various positions such as Programme Manager, Policy Officer and External Auditor. Follow Dana on LinkedIn and contact her via derepede@gmail.com.
Virgil Pamfil has 30+ years of experience in the area of public administration and civil service reform, socio-economic development, education, democracy and rule of law, capacity building, decentralization and governance programmes. With a master’s degree in economics, postgraduate studies and certifications in project management, he has been assigned as a key expert in projects in various countries. In 2015 Virgil received an IOCE Regional Award for his contributions to evaluation in Europe. Follow Virgil on LinkedIn and contact him via vpamfil@evaluatoriasociati.ro.
