SAFE believes that projects should be rooted in data–it is ethical to evaluate our work so that we understand what works and what doesn’t, because that is how we can have the most impact for moms and babies and be as responsible as possible with the funds we have available. We know that this isn’t just a struggle for us, but for many smaller organizations whose budget just doesn’t allow for huge M&E expenses.
That is why for the past year SAFE has worked hard to develop an effective, evidence-based, robust monitoring and evaluation (M&E) strategy in order to better quantify the amazing impact SAFE has in the communities where we work. Our strategy was intentionally developed to gather high quality, mixed methods data at a fraction of the cost and in a much quicker time-frame than other programs we have seen. We developed the method primarily through our work with the ACT for Child Health Initiative we recently launched, supported by the Children’s Prize, in which we surveyed 4,499 women in a comprehensive quantitative health survey, and interviewed more than 60 people as part of our baseline assessment (these included maternal death social autopsy, exit interviews, and in-depth interviews).
Yet, we really wanted to develop a methodology that could be useful in the long-term, not just to SAFE but also to other nonprofits wanting to incorporate more data collection on smaller budgets than originally anticipate as being required. That is one of the reasons we are so thrilled with a new partnership!
In January, SAFE was invited by the Center for Health Market Innovations (CHMI) to participate in their Primary Care Adaptation Partnership. This program allowed us the opportunity to learn more about a wonderful organization called Healthy Entrepreneurs (HE) that addresses health commodity security and supply chains in a sustainable, community-based business model. As part of a new partnership created with HE (more on this later!), SAFE offered its expertise in M&E in order to support the design and implementation of a survey in an area in which HE already works in Uganda. With a very modest 4 figure budget and less than 2 months to design and implement the tool, the SAFE and HE teams were able to successfully survey 913 participants in the Kibaale District of Western Uganda and will complete about 2000 added surveys by the end of April in an area where SAFE currently works in preparation for launching a new joint SAFE-HE project in that area later this year.
Now having been able to work with another organization to adapt our M&E model to their own context, SAFE wants to encourage other small organizations to prioritize monitoring and evaluation of their programs in order to better inform their interventions and ensure they are having the largest impact. The remainder of this post will be dedicated to sharing some ideas about M&E budgets in hopes of helping others (and we welcome you to reach out to us for more specific information if helpful). (Additional posts in coming weeks will focus on other aspects of M&E development, and our growing partnership with HE, so stay tuned.)
The first step in any M&E project is to draft an accurate budget to see if the work you want to accomplish can be completed with the budget at hand. We know that monitoring and evaluation can be a bit overwhelming if you have never done them before, so we wanted to offer some thoughts as to how to structure a budget and plan.
Truthfully, when we first designed an M&E budget, we didn’t really know much about what other people paid, and honestly that was probably a good thing since most M&E projects had a lot more capital to work with than we did. We just looked at the overall categories of other budget templates, and adapted them to our context. They included:
TRANSLATION: Your survey cannot be in English if the local language isn’t usually English, and it will severely compromise data quality if you have interviewers translating on the spot. For the survey to really be valid, your survey needs to be translated into the local language BEFORE you conduct any training or data collection. If you have staff who are really good at both spoken *and* written translation, and understand the differences between them, you can have an existing staff member work on it, but of course, you will have to pay them for that service. If your staff isn’t accustomed to these types of translation, you would be better to pay for translation. Remember that translation isn’t supposed to be verbatim translation, but rather it’s looking to translate the true meaning. For example, the term for “antenatal care” in our area literally translates to “drinking medicine”. So having someone translate “did you receive antenatal care during this pregnancy?” shouldn’t be translated verbatim; it should actually say “did you receive [drinking medicine] during this pregnancy?” Making sure you budget for a good translator will save you a lot of headaches later in the survey process. Don’t chintz on translation, and make sure you have the survey translated into ALL major languages (if you are in an area that has a lot of refugees, you will likely need more than one translation; not being able to survey someone because the survey isn’t in the correct language will introduce bias because the results will not longer be representative of the population on the whole; different people really do have different health indicators, especially those who don’t speak the predominate language in a particular region).
STAFF: Enumerators (quantitative surveys) and interviewers (qualitative) should be local people who are fluent in the major languages; this is very important, as having foreigners conduct the surveys, even with a translator, would introduce a tremendous amount of bias and render the results invalid. We chose to use relatively young (20s – 30s) enumerators/interviewers with at least a high school diploma and fluency in multiple languages, who were really looking to “prove” themselves, and budgeted time (and a little money) to train them. This saved us significant money, as more experienced interviewers would have been far more expensive. We budgeted to pay them a fair daily wage plus a small stipend for food and water during field work (which was very important for their health and satisfaction). We priced out compensation by asking partner organizations what they paid people of a similar training background. Additionally, as we have used these interviewers multiple times for surveys, so our invest in their original training has paid off.
Also, don’t forget about supervisors. If you’ll have yourself, your more experienced staff, or an outsider coming to monitor the data collection (which ideally you should have), make sure that you budget for their time, food, water, and lodging.
TRAINING: Do NOT forget to budget for training: training space with electricity (how are you going to train all of these people in a small office room?), refreshments during training (as much as you might not want to pay this, it’s really crucial if you want this to work), renting a projector, and materials like notebooks, pens, a roll of big paper and markers (or a whiteboard and dry erase markers with an eraser). You should not chintz on this, as the quality of your training (and the length of your training) is directly related to the quality of the work your people will do, particularly if you’re using less-experienced interviewers.
SURVEY MATERIALS: What do you need to actually do the survey? If you have a large sample size, we found that using a tablet-based survey (rather than paper surveys) was much less expensive. But that means that you need Android-based tablets (and extra, because some will break or get stolen or malfunction or you will find you need one so you can follow along or you’ll need one more interviewer than you thought, etc.). You also need CASES to put the tablets in and a backpack for the enumerators/interviewers to carry the tablets and supporting materials. **DO NOT omit these items because they seem less important. If there aren’t cases, your tablets *will* break or not last as long because of water or dust getting inside them (we say this from experience). If you don’t have backpacks, staff will lose things (also from experience). A little upfront costs for these items can go a long ways in saving money later. Ways to cut costs include buying online, getting things donated (have a “Donate your old Android tablet party”, or a “donate your gently used backpack party”) when appropriate, going through suppliers like Amazon when you don’t have to pay shipping, getting local when appropriate (like the cloth sewn backpacks instead of commercial ones which are cheaper and have the added benefit of supporting local business), etc. Some other questions:
OTHER EQUIPMENT: If you will be doing population-based surveys, do not forget to get adequate charging equipment. We recommend getting a multi-USB charging device that you can plug into a cigarette lighter (so you can charge in the car), and/or staying somewhere that has a back-up generator or solar electricity. If you’re working in a low-middle income country where power outages are frequent, you WILL most likely have many times where power is out precisely when you want to charge.
MUAC TAPES: Will you focus on nutrition? If so, you might need tape measures, MUAC tapes, or scales. *Think about this ahead of time and budget for it.* Also remember that partner organizations might have things available you could borrow or pay a nominal fee to rent.
AUDIO RECORDERS: Will you be recording an interview (if you need to do that)? If so, you need to either have them record with a tablet or get an audio recorder and ample batteries.
AIRTIME: Your enumerators and administrating staff will need cell phone airtime to communicate about data collection issues. We didn’t budget for this, but should have. A good rule for us was to budget about $0.50/day/enumerator and $5/day/supervisor. Encourage them to communicate by text whenever appropriate, and have the enumerator “beep” (call and hang up quickly) the supervisor to get a return call. This will save you money.
MAPPING AND ENUMERATION CENSUS: Most types of quantitative survey sampling methods will require that you generally know the number of households in each village and generally know the boundaries of each village. We found that there were no estimates available in some of our areas, and that those that were available were sometimes old and inaccurate. We used what are called “shape files” of the parishes in our area (each parish have multiple villages), and either Google Maps or QGIS (both free) to generate general maps of the area. You can also create your own map using a handheld GPS (which we did with no prior technical knowledge, and you can ask us if you want more info—we’re developing a manual!). You can then go work with village chiefs or elders to identify boundaries and number of homes. You can either ASK for the number of homes (knowing that there will be some bias here) or physically count them yourself. It’s even better if you can actually visit every home in the entire region and get a head count by gender and age, as that will be useful during sampling frame develop and survey administration… sometimes that’s possible and sometimes it’s not (for us, each “enumerator census” cost about $1,300 on average). BUT remember that you need to budget for it, the personnel, the technology (or printing), getting out to the village chiefs, often compensating them for their time in taking you around their village, etc.
FUEL AND CAR RENTAL: How will you get around the survey area? You will likely need a large vehicle that can accommodate all people. If your organization owns one, cool. If not, you’ll need to hire one with a driver, and then pay for fuel. Have your staff negotiate the price, in consideration that you will rent it for multiple days. Don’t just accept the first price that you get, and it’s even better if you develop relationships with particular drivers over time, because then you will consistently get better prices.
TERRAIN AND OTHER TRANSPORT MECHANISMS: If your region is particularly mountainous or treacherous, you may need to budget for your enumerators to get out of the hired vehicle and take local transport means (motorcycles, etc.) to get to certain areas that are inaccessible by car. Make sure you think about this and budget for it if you need to.
A WORD OF ADVICE: If you need help developing strategies, don’t be afraid to find a really dedicated MPH (or other professional) student. We use highly qualified and well-trained students in a lot our work; it saves us a lot of money, gives them really good experience, and gets us much better data than we would have if we relied only on our own devices. (If you want to use a recent MPH graduate who will do a good job and might not cost as much as a senior professional, let us know! We have some great people to recommend!!)
Now go out there and get surveying! If you or someone you know is interesting in learning more about conducting M&E on a budget or would like SAFE to consult on a project, feel free to email us at email@example.com.