Consultation cum workshop on Strategies for Evaluating SDGs – equity and equality

9^th March 2016

National Institute of Labor Economics Research & Development (NILERD) under NITI Aayog, Government of India in association with the Institute of Social Studies Trust (ISST) and the Evaluation Community of India (ECOI) organized a consultation cum workshop on various issues relating to SDGs and gender and equity focused evaluation in context of India on March 9, 2016 at NILERD, Narela Institutional Area, Delhi. 

The themes on which consultation was convened are as follow. These are the issues that are being discussed and debated internationally at various forums.

1.  Relevance of “new metrics” (measurement tools and indicators) for the evaluation of SDGs from an equity-focused & gender-responsive perspective

2. Evaluation & complexity – Dealing with the increasing complexity of development & interconnectedness of SDGs to ensure that “no one is left behind”

3. Towards equity-focused & gender-responsive national evaluation systems – Multi-stakeholder partnerships to strengthen national evaluation capacities

4. Demand for & use of evidence from equity-focused and gender responsive evaluation to inform equitable development

The consultation cum workshop was represented from various types of national, international organizations including Government, academia, researchers and independent evaluators, NGOs, UN agencies etc.. List of participants is annexed.

A summary of deliberations:

The consultation cum workshop was declared open by traditional lighting of lamp by Dr. Pitam Singh (retired as Joint Advisor from NITI Aayog) who was joined by guests and other participants. Dr. Rashmi Agrawal, Director NILERD and core member of ECOI welcomed all the participants and apprised them about the purpose of the workshop. Dr. Rajib Nandi the co host discussed the background and current focus areas of functioning of ISST. He also spoke about the newly formed Evaluation Community of India (ECOI) and its objectives. Ms. Rituu Nanda elaborated on the process of the formation and purpose of ECOI. Information on recent developments, including formulation of action groups to generate knowledge products on various themes, was shared by Dr. Rashmi Agrawal. Mr. Prasun Kumar from KODE System then launched the website of ECOI (www.ecoionline.com) and took the participants through its features.

Dr. Pitam Singh while inaugurating the workshop discussed the structure and mandate of NITI Aayog ( Earlier National Planning Commission) and stated that NITI Aayog is placing lot of emphasis on monitoring and evaluation of development programmes. It is also in the process of establishing regional centers for M&E. The inaugural session of the workshop was followed by presentations by Mr. Manas Bhattacharyya representing Association for Stimulating Know How (ASK), Ms. Subhalakshmi Nandi representing UN Women, Dr. Paramita Majumdar, Ministry of Women and Child Development added government view point on various issues.

Manas spoke on the role of participatory evaluation in achievement of the SDGs by ensuring independent assessment of critical aspects of relevance, effectiveness, efficiency, results, impacts, sustainability, and community participation in specific development projects & programs; by drawing learning, both in terms of strengths and weaknesses, and by helping and supporting organizations to develop, change, modify and strengthen their strategies, approaches, methodologies, nature of interventions, staffing & mechanisms; by fostering an environment of enhanced ownership among stakeholders and the community; by integrating “Gender Inclusion” and “Equity” as a cross cutting aspect/theme in the development programs; and by focus on participation and utilization taking inclusion and equity as a cross cutting theme. A need was expressed to maintain cultural sensitivity while dealing with different communities, without any compromise with the objectivity. Evaluations need to be empathetic towards the implementing organization and its staff with the ability to understand their situations, problems, constraints & challenges while simultaneously not losing the objectivity to be truly participatory. The process of participatory evaluation should not be treated as a fault finding exercise rather a process of enabling an organization to overcome constraints which pose as hurdles in achieving the desired development results. Encouraging continuous engagement, reflections & open discussions can lead to ownership use-focused evaluations.

Subhalakshmi explained the role of UN Women which is coordinating across ministries, across different organizations and key stakeholders, to contribute to global norm setting and contextualizing and localizing norms. Goals of SDGs cannot be achieved without achieving the goal 5 of SDGs dealing with gender equality and empowerment of women and girls. UN is committed to develop a roadmap for women’s status. It was mentioned that the SDGs are ambitious and transformative. There is a need to develop a road map for survey on country specific status of women – the existing state of data and the gaps therein. In particular, a need was expressed to have data on women’s asset ownership, unpaid work and their time use and crime against women. Data on gender in the context of evaluation of SDGs is also required.

 Paramita mentioned that the principles of gender budgeting has been accepted in India and is in vogue for over ten years.  However, data are collected mechanically at present and more work needs to be done at the level of development of indicators. While the financial audits conducted by Comptroller and Auditor General have sanctity attached to them and are taken due note of, the same is not true of social audit with gender perspective. Institutional mechanisms are required to give teeth to social Audits. At present, only some programs are picked up for performance audit. Internal audit division of Ministry of Finance has been taken on board for mainstreaming gender. Existing data is weak and various sources provide different data on the same aspects. For example, Mid -day Meal Scheme was evaluated 42 times by various agencies with contradictory data and reporting. It is to be seen as to how to reconcile with various sets of data. The National Policy on Empowerment of Women, focusing on addressing more gender sensitive issues, is under preparation. Dr. Majumdar expressed the need to provide inputs in finalizing the policy. Ministries should also come forward to prepare a checklist for gender guidelines where inputs from various stakeholders are needed.

Rashmi indicated that SDGs are not in silos and government has mapped nodal Ministries, schemes and interventions to be made for each of the SDG targets. She also stated that mere government formulations would not go a long way; attitudinal changes at all levels in all sections of the communities can accelerate the process of reaching gender equality and social equity. Gender equality should not be mystified and problems of both men and women should be addressed.

Dr. Sunita Palita commented that it is a cause of concern that girls disappear suddenly from education front after certain years of schooling. They are abused from both sides, for not being able to do household chores because of school attendance and in the school not performing well because of domestic tasks. Data are required to understand this sudden disappearance of girl students.

Ms. Savina Claudia Ammassari mentioned that a lot of work has been done on studying the vulnerable population. Attention should also be paid to assess outcomes for India's Third Gender (transgender people).We have to monitor and evaluate outcomes for vulnerable and often marginalised people, Transegender and other vulnerable populations who are at high risk of HIV and suffer from significant stigma and discrimination. TG, sex worker, Homosexuals, drug addicts - all are important groups to keep in mind and involve in design, implementation and use of evaluations.  The most important part of the evaluation is the USE of data and strategic information to accelerate progress towards achievement of the SDGs in India and beyond. It is found that there is a lack of connect between the scientific community and the community they are serving, that is, the vulnerable population.   There needs to be prioritization of data to change the mindset, to educate the vulnerable, and to disseminate information. The government needs to be brought on board, and better coordination needs to happen between the researchers and the decision makers.  Rituu added that if we are looking at power relations, let us not forget those with other sexual identities as said by Savina. In April 2014, The Supreme Court of India passed a judgement on issues of transgender, giving them the right to choose their gender identity and a whole range of development measures including status under OBC. A recent study in Punjab indicates that school dropout rate is high in transgender children.

Following the presentations, Ms. Beryl Leach facilitated group formation and group presentations on four themes identified to be addressed in the M&E systems to assess SDGs in the Indian context. The group observations on the various themes are collated as follows.

Theme 1: The relevance of “new metrics” (measurement tools and indicators) for the evaluation of SDGs from an equity-focused and gender-responsive perspective

Issues raised and outcome of discussions:

It was mentioned that limitations of current data collection methods include lack of availability of data in public domain/open data, no checklist or cross checking, concern with regard to data authenticity/reliability, discrepancy, lack of mainstreaming of data collection methods into organizational information systems and lack of institutional learning, lack of harmonization and neutralization of data collected across government and non-government organizations, time and resource constraints. Observations on the limitations of indicators produced included the following - sex based indicators do not allow capture of gender dynamics of respondents, lack of sensitivity to gender and within group vulnerability, inaccessibility to most vulnerable, not focused on qualitative & non-economic & non-monetary & non-measurable aspects, lack of  checking beyond project period raising concerns on sustainability, and lack of participatory design. The most difficult issues in measuring social equity and gender equality and ensuring their sustainability were identified were - data dissemination, attitudinal and behavioral changes, challenging essential practices and norms, need for community based plans and stakeholder ownership, capturing hard to measure discrimination, the need for a methodology sensitive to cover most vulnerable among the vulnerable (check at design stage) and need for awareness creation. The new metrics that the group was aware of and which showed the most promise included Focus Group Discussions (FGDs) and observation based techniques for field research.

During the discussion it was pointed out that there is need for more innovative metrics suiting to varying cultures and traditions. Only FGDs and observation techniques were not enough for capturing the change. It was also mentioned that in general people do not believe in qualitative methods, Some standardization is needed to make them acceptable. Data analysis plan needs to be established for evaluation and synchronization between quant-qual is required.  Participatory approaches are of extreme importance. Capacity development to employ qualitative methods is essential at various levels.

Theme 2: Evaluation and complexity – Dealing with the increasing complexity of development and interconnectedness of SDGs to ensure “no one is left behind”

Core issues raised in group presentation: The group noted that the challenges to cope with complexity in evaluation processes include diversity cutting across inter-state differences, social (caste, class), cultural, power dynamics implying an inability to rely on a uniform framework and exclusion of people on multiple levels such as disability and HIV/AIDS. Secondly, environmental dynamism and constant change causes problems when it comes to developing a concept/logic model. Thirdly, differences in stakeholder interests and agendas may not be covered – the stronger and more vocal stakeholders may try to influence the process and outcomes. Solutions proposed cut across evaluation design and participation, and are listed as follows:

• Participatory design should incorporate flexibility to proceed in an interactive way - who needs to be involved at what stage, what methods and tools must be used (informed from reviews of existing information from multiple sources)
• Evaluation design needs to take into account in-depth understanding of local contexts, needs to be fully-customized and context specific, cookie-cutter approach should not be there.
• Multi-layer, multi-dimensional, multi-method approach needed for evaluations specifically to address the issues of E and E.
• Systems approach which identifies “interconnectedness”  across global, national, and local levels and stakeholder mapping
• Participatory monitoring with different stakeholders involved is particularly important in evaluations of complex programmes
• Strength-based approach can create a space where different stakeholders can share openly and value each other’s perspective and learn from each other. Composition of the evaluation team- sensitive to gender and equity lens, has a good mix of thematic and evaluation experience
• Evaluators need to take the role of  facilitators to create an environment to learn and for evaluation not to be seen as an audit or fault-finding exercise
• Participatory statistics involving community in collecting quantitative data especially on sensitive issues

Theme 3: Towards equity-focused and gender-responsive national evaluation systems – Multi-stakeholder partnerships to strengthen national evaluation capacities

Core issues raised in group presentation: An evaluation system is lacking. There is need for a nodal agency, which might be NITI Aayog, for evaluation with a steering group to address the specific sectoral, regional and community needs. The steering group must comprise of state representatives, community representatives and other stakeholders to look into their specific requirements.  This will provide guidelines and standardized evaluation criteria. An exhaustive checklist needs to be prepared. Flexible guidelines need to be developed across evaluations. Capacity building for gender sensitive and equity focused evaluations is needed. Peer learning and tours must be integral to capacity building to help learn from good systems. Embedded cells with expert teams will facilitate technical support for successful partnership in India to strengthen M&E systems in general, and equity-focused and gender-responsive systems in particular. Every state in the country has a statistical unit/Planning Department, M&E system should dove tail with them. The training in M&E system must be made mandatory.  The challenges are that we need to have evolving National Evaluation Policy which will lay down standardized guidelines and checklist for cross-cutting sectors.

Theme 4: Demand for and use of evidence from equity-focused and gender responsive evaluation to inform equitable development

Core issues raised in group presentation: The group took a holistic approach to discuss the factors affecting the demand for and use of evaluations, factors for demand and utilization of social equity and gender equality evaluations, and the types of evaluation presentations than can be proposed to make evaluation more accessible to stakeholders. It was felt that within government a number of factors affected demand for evaluation including lack of incentives, complacency, high staff turnover, lack of capacities, lack of understanding so on and so forth. This could be dealt with providing performance and career incentives, training to improve performance of project and replication, standardized guidelines and a less rigid framework. It was also felt that at the level of the implementing agency, conception of the program should take into account a credible and capable agency which would be gender responsive. At the level of the evaluators/researchers, the factors included theory based, mixed method, multi-disciplinary evaluations, time and budget issues.

At present, not much emphasis is given to formative, mid- term and concurrent evaluations which can be more useful for utilization purposes. All the schemes and programmes should have a provision for these evaluations.

There is hardly any communication plan for the evaluation reports nor is any specific responsibility given to individuals/organisations for disseminating recommendations. Feedback loops for follow up are also missing which shows that evaluations are conducted without intent for utilization. Even when some reports are placed on web site for better outreach, a large community cannot access these due to lack of internet facilities, illiteracy, lack of understanding due to language etc. The communication of results and best practices should be aligned with the traditions and understanding of the community. For example at rural level story- telling, role play , visual media etc can be useful. The community will then raise the demand for use of recommendations.

Private sector in India is coming in big way in developmental activities. There should be a  demand for quality evaluations for their interventions to serve as evidence for quality decision making. It is important to involve decision makers in the scheme of things of M&E which is a difficult task. Sensitisation workshops should be mandatory for them.

The use of evaluations can be enhanced if stakeholders are involved at various stages of planning of intervention and evaluation questions are a part of intervention and implementation.

To summarise, program design itself should be gender and equity focused. A movement for socio psychological empowerment of the vulnerable is a pre-requisite for development and evaluation, evaluations need to be internalized and focus should be on behavior change. Evaluations should be context related beating the stereotypes, most under privileged to be included and sensitization of government and parliament is needed which will take place with the community pressure. Gender focus lense should be a way of working with a collective effort. Evaluations are needed both with micro and macro focus. Interpretation of evaluations should also be without bias. Outreach of evaluation results should be enhanced using various communication strategies translating them in the language and understanding of various users. If we are looking for inclusiveness there is a need to catch the last person which is very difficult. Strategy should be evolved for how to reach to the last person.

 

 

Workshop on Evaluation Strategies for SDGs with E&E- Some Glimpses-9th March 2016