2.   A detailed discussion of the rationale for enhancing the Fund’s capacity to undertake independent evaluation through the establishment of an independent office is contained in “Review of Experience with Evaluation in the Fund” and will not be repeated here at length. What was made clear in that paper was that the motivation to develop the role of independent evaluation at the Fund stemmed from a desire to enhance the Fund’s internal learning culture, foster a more broadly-based understanding of its mandate, and bolster the credibility of its work outside the institution. It by no means represented a lack of confidence in management on the part of the Board or a desire by the Board to micro-manage the day-to-day operations of the institution.

3.   With these comments in mind, this paper takes the next step in the effort to operationalize the decision to establish an independent evaluation office. It will therefore clarify a number of issues raised during the previous Executive Board discussion and address many of the concerns expressed by Directors and management with potential pitfalls in the establishment of EVO. Draft Terms of Reference for the Director of EVO and other documentation related to the establishment of EVO are also presented.

4.   EVO is to assist the Executive Board in providing effective governance and oversight of the Fund. This implies that its Director will need to be independent of Fund management and operate at arms length from the Executive Board. According to the Fund’s Articles of Agreement, the Executive Board is “responsible for conducting the business of the Fund and for this purpose shall exercise all the powers delegated to it by the Board of Governors²”. The Managing Director conducts the ordinary business of the Fund “under the direction of the Executive Board”. EVO will also assist the Board of Governors in fulfilling its responsibilities, as the Executive Board will make regular reports on the activities of EVO to the Board of Governors either directly or through the IMFC.

5.   The Director of EVO would report regularly to the Executive Board on the activities of EVO, including through the preparation of an Annual Report. In consultation with Executive Directors, management, and staff, the EVO Director will be responsible for preparing a work program for EVO which could be brought to the Executive Board for discussion and possible approval as is the case in other IFIs, including the IBRD and the IaDB³.

6.   Whether the Board should give formal approval to EVO’s Work Program is the subject of some debate. Some view the exercise of such discretionary power as potentially undermining EVO’s perceived independence. Others believe that Board approval is consistent with (and indeed required by) Board responsibility, and that the risk of inappropriate use of this provision would be mitigated by a requirement to disclose publicly if any proposal by the Director of EVO was rejected by the Board. Regardless of which approach is adopted, the Director would be expected to provide staff and management with adequate and appropriate opportunity to respond to EVO’s proposed work program and the analysis and recommendations arising from its evaluation activities.

7.   The Evaluation Group of Executive Directors will continue in its present role of supporting the Executive Board in its oversight of independent and external evaluation at the Fund. However, given the need to ensure the independence of the Board’s evaluation function, the selection of its members should be placed in the hands of Executive Directors themselves. After EVO becomes operational, the role of the EG may need to be re-examined.

8.   The Executive Board would retain the right to initiate external evaluations itself. EVO would provide administrative and technical support for these projects except in cases when the Board deems such a role to be inappropriate.

9.   During the April 2000 discussion, management and Directors emphasized the importance of coordinating the work of EVO, the Office of Internal Audit and Inspection (OIA) and staff self-evaluation so as to minimize overlap and duplication and ensure that resources allocated to EVO were used efficiently.

10.   The view was expressed that the scope for independent evaluation be clearly circumscribed to ex post evaluation to avoid interference with ongoing operations. Such is the practice for independent evaluation units at a number of MDBs. However, while the concept of ex post is clearly understood in the context of project evaluation, it is less well-defined for policies and programs, which would be a major focus of independent evaluation in the Fund. For example, the concept of “ex post” is not well defined when considering policy adequacy or implementation, policy advice in the context of surveillance, or when reviewing the experience of a country with a history of Fund programs.

11.   It should also be acknowledged that some blurring of scope is both inevitable and appropriate when considering independent evaluation. An effort to pursue an overly precise delineation of responsibilities, without the benefit of perfect foresight, run the risk of undermining the independence and external credibility of EVO.

12.   Nonetheless, Directors are cognizant of the need to ensure that EVO does not interfere with ongoing operations and are determined to avoid micro-management. A number of proposals have been advanced, including to limit the scope of evaluation to policy issues and cross-country analyses and/or to completed programs.

13.   However, many of these may be seen as overly restrictive and may not be practical. Others would prefer no strict constraints, preferring instead the articulation of principles that reinforced the complementary nature of EVO to existing Fund activities. They have argued that these would provide a more enduring and constructive approach. Such an approach would not unduly undermine external perceptions of the degree of independence being afforded EVO. That being said, the success of any such arrangement will hinge critically on the degree of professionalism, mutual respect, and trust which management and the Director of EVO bring to bear.

14.   Therefore, while there would be no strictly and precisely defined restrictions on what EVO could evaluate, its Director could instead be bound by three guiding principles. First, evaluations undertaken by EVO should complement efforts underway elsewhere in the Fund. To ensure that this happens, EVO must make explicit in each of the evaluations it undertakes what its objectives are and in what way its efforts complement the work of staff. Second, EVO should avoid pre-empting efforts by management to enhance the efficacy of the Fund’s work and manage the institution. In particular, any decision to launch an evaluation should be taken with reference to staff and management’s willingness and ability to undertake the work themselves and in a timely manner. Finally, to help ensure that EVO evaluations of surveillance and country programs do not interfere with ongoing operations, they should be “time-frame” specific.

15.   With respect to the work of OIA specifically, blurring of mandates with EVO could be minimized through a more clearly defined mandate for OIA. The parameters of a revised OIA mandate would need to take into account the fact that, partly in response to the launching of the pilot project on external evaluation in 1996, the activities and resources of OIA were expanded to allow it to conduct more reviews of the Fund’s organizational structure and work practices and assist the Board and management in developing and facilitating the agreed external evaluation projects. That being said, the task of redefining the scope of OIA’s mandate remains a management prerogative.

16.   In light of the above, Annex I illustrates the likely division of labour between EVO, OIA, External Audit, and staff self-evaluation.

17.   Access to Information: EVO, to be effective, would require unrestricted access to interview staff and to all relevant Fund documents, minutes, and internal staff memoranda needed to carry out its Work Program. This would not include personal information on staff members to which access is normally restricted. Staff would be expected to cooperate fully in the process of accessing information. EVO would also be able to request information from Executive Directors, government officials, or any other persons.

18.   The Director should be a person of proven competence and impartial judgement with broad experience and background in the management and evaluation of policy, economic, and financial matters. This person must be held in high regard by his or her peers and by Fund management. He or she should be familiar with Fund policies and operations, but also have established professional credibility outside of the IMF.

19.   In order to attract superior candidates to the position and reflecting its importance to the Fund, the Director should hold the equivalent rank of Department Director at the B5 level.

20.   To achieve an appropriate balance of continuity and experience against the need to bring fresh perspectives to bear on the leadership of the EVO, the Director’s term of appointment would be for a period of 4 years, renewable by the Board for a second term of up to three years. Given the need to balance the Director’s operational independence against his or her ultimate accountability to the Executive Board, the Board would be able to terminate the Director’s appointment for unsatisfactory performance. To further ensure the independence of the Director, at the completion or termination of his or her appointment, the Director would not be eligible to join or rejoin the regular staff of the Fund.

21.   The EG recommends that consideration be given to enlisting an international executive search firm to conduct the identification of candidates, subject to Executive Board endorsed criteria⁴. The search firm would help “de-politicize” the selection process and broaden the pool of potential candidates. It will also ensure that all names put forward for consideration represent individuals who meet the qualifications for the post. The search firm would identify and interview the candidates and present a list of four to six names for consideration by Directors.

22.   Recognizing that the cooperation of management will be essential for an effective independent evaluation office — particularly with respect to the incorporation of evaluation recommendations into Fund operations — the suitability of the recommended candidate for Director of EVO would be discussed with the Managing Director. Once Executive Directors have finalized their choice of candidate, the nomination will be presented to the Executive Board which would be asked to appoint the individual to the position of Director of EVO.

23.   This selection procedure could be expected to be completed within six months so that the Director could join the Fund by the first quarter of 2001 and the work of the EVO could begin by the start of FY 2002. Proposed Terms of Reference for the Director of EVO are contained in Annex II.

24.   The Executive Board should agree on an indicative size and composition of the staff of the EVO. At the April 2000 discussion, many Executive Directors showed interest in management’s proposal⁵ for a complement of 11 full-time positions, although a few Executive Directors suggested a smaller office of 6–7 positions. Directors supported the proposal that EVO be provided with a budget for the hiring of external consultants on a contract basis to augment its own internal expertise and capacity for specific evaluations. The budgetary implications of an office of 11 positions (not all of which would be additional to overall Fund staffing) are discussed in more detail below.

25.   Academic qualification and adequate experience in economics should be regarded as a necessary, but not a sufficient, condition for full-time professional employment in EVO. These skills should be augmented by a background in related disciplines and a genuine interest in, and enthusiasm for, improving the Fund’s effectiveness through independent evaluation. To ensure there is an adequate base of knowledge of Fund operations within EVO, a portion of EVO’s full-time professional staff could be seconded from the regular Fund staff. To ensure both a real and perceived inflow of fresh perspectives, a portion of the full-time professional staff should come from outside the institution. The status of EVO staff will be determined by decisions of the Executive Board adopted in consultation with the Managing Director and the Director of EVO.

26.   It is neither expected nor desirable that individuals make a career out of EVO evaluation. Rather, EVO’s effectiveness will be a function of its ability to mobilize strong motivated performers who can bring new ideas and up-to-date knowledge to independent evaluation. To that end, the terms of employment in EVO will need to be such that, in the case of Fund staff considering working for EVO, that experience is seen as particularly beneficial to career progression in the Fund. The Fund would also benefit from their return to the operational side of Fund work since this would help to spread the lessons learned from evaluation and heighten the appreciation and awareness of independent evaluation throughout the Fund. It is therefore recommended that:

• Professional staff in EVO should be composed of a combination of secondments from staff and external recruits, with external recruits making up at least one half of the total, excluding the Director.

• While there would be no presumption that staff hired from outside the Fund would continue on to regular Fund employment, they should nevertheless possess the qualifications for regular Fund employment.

• To attract the highest quality staff from within the Fund, and to ensure the maximum benefit of the EVO to the Fund, the Director of EVO should be given discretion to recommend promotion of EVO staff after at least one year working for EVO.

• Management will need to commit to ensuring that EVO staff who return to regular Fund staff are offered desirable assignments upon their return, and are in no way discriminated against because of authorship of reports that are critical of potential receiving departments.

• In all other respects, conditions of employment for staff in EVO should be the same as those of the regular staff of the Fund. To maintain the independence of evaluations, the Director of EVO should exercise prudence in determining if, and to what extent, staff seconded to EVO should be assigned to the evaluation of operations with which they have been previously involved.

27.   In order to ensure that the factual content of EVO reports is accurate, a procedure for staff and management to comment on the reports before their finalization should be established. EVO would also be expected to provide staff and management and, when appropriate, the relevant country authorities, with the opportunity to prepare a written response to the evaluation report for circulation to Executive Directors at the same time as the evaluation report. Staff and management, and again, when appropriate, the relevant country authorities, would also be given the option of providing a statement for publication along with the final report. In all cases, EVO would be expected to provide staff and management with a reasonable period of time in which to consider, and provide comments on, the results of independent evaluations.

28.   EVO, in consultation with the EG and with the assistance of EXR, would be responsible for the manner in which EVO findings are disseminated outside the Fund. The publication of EVO reports would occur together with the Board’s Summing Up, staff response, management statement (if any) and, where appropriate, a statement from the relevant country authorities. Confidential material would first be removed from these documents.

29.   With an indicative complement of 11 positions, a possible staffing structure could comprise a Director, a Deputy Director, five evaluators, one economist, and three staff/administrative assistants. Using internal estimates of the cost of salary, benefits, and office overhead (in FY 2001 prices), the direct staffing cost of the EVO would be about $2.5 million. To this should be added an administrative budget of about $45,000, travel costs of the order of $80,000,⁶ and a consultant budget in the range of $300,000 to $400,000, with the precise amount to be calibrated to the demands of the Work Program.^{7 8}

30.   While EVO would be expected to absorb the costs of supporting any fully external evaluation initiated by the EG, the cost of hiring external evaluators and their travel would not come from EVO’s budget. The Executive Board would therefore have to make a special appropriation for the associated costs. The net cost of EVO would be reduced by any transfer of positions from the other departments/offices to the EVO that is determined by management and the Executive Board. Management should identify the extent to which positions from elsewhere within the Fund can be reallocated to EVO. The funds now provided to OIA (approximately $500,000) to support the EG’s external evaluation efforts would be removed from OIA’s budget.

31.   It may take some time for the EVO to be fully and satisfactorily staffed. It is therefore unlikely that EVO will require, or be in a position to make use of, its full staffing complement in its first year of operation.

32.   The indirect cost of EVO in the form of time spent by other departments in assembling information and being interviewed for the evaluations cannot be estimated at present. Nevertheless, the Director of EVO should be sensitive to the demands placed on regular staff in this regard.

33.   In the previous discussion, many Executive Directors endorsed management’s proposal that after the first year of operation at full capacity, the resources devoted to independent evaluation could be increased approximately in proportion to the growth of the Fund’s overall administrative budget. This would permit an increase in the size of the EVO commensurately with the growth of the Fund. While this notional principle could be maintained for the near term, given the uncertainties surrounding the actual costs of an efficient and effective operation, the EG will want to review the financing issue at an early opportunity.

34.   The EVO’s annual budget would be prepared by the Director. It would be discussed and approved by the Executive Board separately from the Fund’s administrative and capital budgets and, in the interests of transparency, be appended to the budget of the Executive Board.

35.   In carrying out its mandate, EVO will be free to consult with whomever and whichever groups it deems necessary, both within and outside of the Fund. As a matter of professional courtesy, and to facilitate effective coordination with outreach efforts being undertaken elsewhere in the Fund, EVO should be in frequent contact with EXR.

36.   The Executive Board would have ultimate authority to determine whether or not the results of a particular EVO evaluation would be published. However, given the mandate of EVO and the importance of transparency to the credibility of an independent evaluation office, there would be an expectation that EVO’s Work Programs would be made public and a strong presumption that the work of EVO (including its Annual Reports) would be published (within the constraints imposed by the need to respect the confidentiality of information provided to the Fund by its members). The Executive Board would not interfere in the drafting of EVO documents (including the Annual Report) press releases and other public statements. In the case of the Annual Report, the Director would be expected to provide information on topics put forward for independent evaluation by EVO which were not accepted by the Executive Board for inclusion in the Work Program. EVO would work with EXR in determining the appropriate modalities to disseminate information to the public on its activities.

37.   Given the need to ensure that EVO is adequately fulfilling the mandate given to it by the Executive Board, the Executive Board should undertake an independent external evaluation of EVO within 3 years of its launching.

38.   If Directors agree with the basic thrust of this paper, the search process for a Director for EVO should begin shortly with the intent of having a Director appointed before the end of the year. Ideally, the Director would be in a position to assume the duties of the office early in 2001, at which time he or she would begin the process of staff recruitment and prepare the ground work to have EVO up and running no later than May 2001.

39.   If this timetable is accepted, the Evaluation Group of Executive Directors recommends the following next steps:

• With assistance from staff, the EG should begin the tendering process to engage the services of an executive search firm;

• This background paper should be posted on the Fund’s website immediately to permit outside individuals and groups to consider its content in advance of the Annual Meetings in Prague;

• The Executive Board should seek the endorsement of the IMFC in Prague for the establishment of EVO along the lines described in this paper.

40.   The EVO would be expected to be fully operational before the 2001 spring meeting of the IMFC.

The International Monetary Fund is committed to a program of evaluation and review as an instrument for the systematic assessment of the results of its strategies, policies, programs, and processes. The Executive Board, management, and staff of the Fund recognize that independent evaluation is essential for institutional improvement and the lessons learned from independent evaluation are necessary for the improvement of ongoing activities and the design of new operations. In the conducting independent evaluations, the Director of EVO will be responsible to the Executive Board and independent of the Fund’s management.

The Director is responsible to the Executive Board for:

• managing a credible program of independent evaluations of the Fund’s policies and activities;

• assessing whether the Fund’s strategies, policies, and programs are producing the expected results and making recommendations which will help improve the efficiency and effectiveness of these operations;

• assisting the Executive Board in monitoring the extent to which lessons from evaluation are being integrated into the work of the Fund and assisting operational departments to maintain the Fund’s institutional memory on lessons learned from evaluation studies;

• seeking internal and external input into the formation of EVO’s work program and into particular evaluations, where appropriate;

• working with the External Relations Department to disseminate evaluation findings outside the Fund;

• providing secretarial and logistical support for external evaluations directly initiated by the Executive Board; and

• reporting regularly to the Executive Board, including through the preparation of an Annual Report, on EVO’s activities during the year and the findings of its various evaluations and assessments. The Annual Report should also provide information on evaluation topics that EVO had proposed but which have been rejected for inclusion into the Work Program.

3.   In conducting a particular evaluation, the Director will have unrestricted access to the staff, and to all relevant Fund documents, minutes, and internal staff memoranda except for personal information on individual staff members to which access is normally restricted. To further the internalization of the work of EVO throughout the Fund, the Director will be invited to attend Executive Board meetings and relevant Board Committee meetings.

4.   The Director will be appointed by the Executive Board, after consultation with the Managing Director. The Director will be a person of proven competence and achievement with experience in the evaluation of strategy, policy, economic, and financial matters, held in high regard by his or her peers, and of impartial judgement. He or she will also be expected to be knowledgeable about Fund policies and operations but have experience that goes beyond the Fund to encompass a number of relevant areas.

5.   The Director will have the equivalent rank of a Department Director. The successful candidate will serve for a period of four years, renewable for a second term of up to three years, if recommended by the EG, and approved by the Executive Board. At the end of the term of service, the Director will not be eligible for appointment or reappointment to the regular staff of the Fund. The Director’s appointment may be terminated at any time with the approval of the Executive Board.

6.   The Director will be responsible for recruiting the staff of the EVO — either from within the Fund or from external sources. To ensure there is sufficient knowledge of Fund operations within the EVO, up to one half of the professional staff could be either current or former Fund staff members. The status of EVO staff will be determined by decisions of the Executive Board adopted in consultation with the Managing Director and the Director of EVO.

7.   Following consultations which would include Executive Directors, management, staff, and external sources, the Director will prepare a work program outlining the evaluations to be pursued by EVO. The proposed Work Program would be presented to Executive Directors for discussion and possible approval. The Director will report regularly to Executive Directors on progress in implementing the work program.

8.   The Director, in consultation with the Executive Directors, will prepare a budget for EVO and present it to the Executive Board for discussion and approval.

9.   Without compromising the independence of the Office, the Director will endeavor at all times to ensure that EVO and its staff maintain close and continuing contact with Fund departments and their staff, so that — within the constraints of independent evaluation — the views of the Fund staff are appropriately considered in the reports of the Office.

¹ “Review of Experience with Evaluation in the Fund” (EBAP/00/29) and “Independent Evaluation in the Fund and in Other International Institutions” (EBAP/00/29, Supplement 1).

² Article XII, Section 3(a).

³Given the importance of maintaining EVO as mechanism which is independent of management but which assists the Executive Board in its oversight of the Fund, Directors may want to consider if, and when, it would be appropriate to conduct discussions of EVO-related issues in the context of a meeting of the Committee of the Whole or of the Executive Board.

⁴ This recommendation is based partly on the IDB’s experience in selecting a Director for their Office of Evaluation and Oversight.

⁵ See BUFF/00/40 (3/24/00).

⁶ The travel cost for the Resident Representatives Review (1996-97) was $27,000, and for the Technical Assistance Review (1998-99) $45,000. In addition to travel to member countries required by the evaluations, the Fund should participate in relevant conferences, including international evaluation conferences convened by the DAC Working Party on Aid Evaluation. It should also become a member of the Evaluation Cooperation Group of the Multilateral Development Banks.

⁷ The consultant budget will enable EVO to enlist external support and augment its expertise for particular evaluations without having to more permanently expand its staffing. As such, it will be instrumental in preventing the sort of unwarranted expansion of which bureaucracies are frequently at risk.

⁸ The cost of consultants for the Resident Representatives Review was $61,000 and for the Technical Assistance Review $151,000.