Measurement advice

In a national survey conducted in 2014 by Wilder Research and the Federal Reserve Bank of Minneapolis, community development and health practitioners offered the following advice to others who want to measure the impacts of initiatives intended to improve community health and well-being.

 

Benefits of Measurement

"A comprehensive data system, while initially resource-intensive in terms of and funds, will, in the long-run, be worth the investment. It will yield returns through use of the data to inform process and program improvement and to measure and report the impacts of investments through changes in access, awareness, knowledge and behavior."
Child-Focused Community Foundation, Southeast U.S.

"If you start slowly with a few key metrics, you will learn a lot, find like-minded investors and partners, and over time be able to better focus resources in areas that can have the greatest impact and leverage the combined resources of your partners. Resources continue to shrink, so the ability to attract investments and make changes in the communities we serve will only happen if we can demonstrate that the investments are having positive and desired outcomes."
Credit Union, Northeast U.S.

"Adding a health focus to the work of 'non-health' efforts and organizations often brings a new energy, additional ways to assess impact, and often a new focus on data."
City Health Department, Midwest U.S.

"Document and measure, and review your data frequently in order to make necessary adjustments. This is even more important when working in collaborative projects. It keeps all partners engaged and invested in their common goals and objectives."
Nonprofit Hospital, California

"Measurement drives change, either in terms of program adjustments or in terms of finding funding and political will/support for continuing successful programs."
Not-for-profit Healthcare System, Midwest U.S.

"Impact measurement is difficult, time consuming, and expensive. However, the knowledge gained and the ability to translate outcomes to stakeholders and investors are invaluable."
Community Development Financial Institution, Pennsylvania

"Measurement is very important, both to note change and more basically to keep a community goal in mind. It is surprising how many sources of metrics are currently available. Moving away from a total focus on clinical data to consider and measure the conditions for living will be critical for creating healthy contexts for human life in this century."
National Health Association, East Coast U.S.

"Evaluation does not have to be fancy. What you learn from the evaluations will help to shape programs and initiatives, and will ultimately allow you to see the impact your organization is having on the individuals who participate in the programming."
Cancer Research Hospital, California 

"Our key piece of advice is to 'start measuring something'. The measures don't have to be perfect and can be refined over time. Creating common definitions across sectors is also critical to monitoring community health. By growing demand for community-level data, we can compel institutions to improve data quality and expand data access."
Health Improvement Foundation, California

 

Cross-Sector Collaboration and Partnering

"Collaboration is key. Partners must be willing to share data and align measurement tools."
Non-profit Public Health Corporation, Southeast U.S.

"Community partners are essential to sustainability. Build relationships in the community. Do things with the neighborhood, not to the neighborhood."
Not-for-Profit, Community-owned Health System, Midwest U.S.

"Consider a collective impact model for population health as you begin to address social determinants of health. Form collaborations with multi-sectoral partners to address health and the factors that impact health. Implement evidence-based strategies with metrics to measure outcomes and review these outcomes. Be realistic in your time frame to affect population health, as many population health outcomes need years to change while the community health improvement strategies are being implemented."
Integrated Health System and Clinic Group, New York

"Engage local health departments and their community partners in the development of new initiatives and projects as experts of the jurisdiction that they serve."
National Health Department Association, East Coast U.S.

"Engage the community and identify both needs and barriers. Once gaps are identified, develop strategic long-term partnerships to bring capacity into the service community and collaborate. Population health demands a multi-sectoral approach; no single organization can do it all. Adapt best practices to meet local needs. Ensure sustainability by empowering people with knowledge and tools to maintain a healthy lifestyle. Develop an infrastructure of health to support the creation of a 'culture of health'."
Neighborhood Revitalization Organization, Midwest U.S.

"Find local partners and understand what they can bring to the table, particularly in terms of data. This work is difficult to do in a silo!"
Health-focused Community Investment Group, East Coast U.S.

"In addition to developing measures of success, also be mindful of the capacity of partner organizations to conduct program evaluation. One of the implicit goals in our community work is to develop the skills and ability of municipal health to conduct evaluation. Some organizations are further along that path than others. It's important to us to be realistic in our expectations of community partners and to help them develop methods to measure outputs and outcomes rather than just assume they can do it."
Acute Care Community Hospital, East Coast U.S.

"It is crucial to shift the focus from outputs to outcomes, and to also look for ways to create layered strategies that include policy, organizational changes, coalitions and networks, provider education, community education, and individual knowledge and skills to affect change in a community or population condition.  By partnering with universities and other researchers, people working at the community level can bring the knowledge and skills for effective outcomes measurement and impact without having to have all of those skills themselves."
Health Advocacy Organization, Texas

"It is important that when organizations come together to address a health issue, they recognize that inputs are not the same as activities, outputs, short-term outcomes, long-term outcomes and impact. They are more likely to create realistic expectations if everyone is on the same page about what successful activities look like vs. successful outputs vs. successful short and long-term outcomes vs. successful impact."
Financial Institution, Southwest U.S.

"Reach out and partner with universities who have the infrastructure in place to collect scientific data and report on it."
County Park System, Southeast U.S.

"Try to differentiate between outcomes and output. Be prepared to wait 10+ years to see measureable results. Educate investors to have reasonable expectations. Understand which financial partners are available for co-investment and partnering. Reach out to others who have alignment with the mission for collaboration and collective impact."
Investment Initiative, California

"One organization cannot do it alone. Develop metrics collaboratively so that all relevant entities are engaged and agree upon the measurement and important metrics. Take a multi-level approach. For example, it's not enough to assess how many people in an area are obese, but make sure to look at all levels of the systems in which they live and work and measure potential contributors and keep the focus on all of the levels."
Research and Evaluation Group, Texas

 

Selection of Measures

"Our biggest piece of advice would be that it's okay to start small, with just a few metrics."
Community Development Financial Institution, East Coast U.S.

"Create clear and detailed monitoring plans. Be realistic about what can be measured and do not overpromise. Identify needed resources for measuring impacts. Review various metrics and adapt them to your needs. Don't reinvent the wheel. Set up different measures depending upon what you are evaluating. Finally, conduct 'impact evaluation', giving enough time after the project has been implemented."
Nonprofit Health Policy and Research Organization, Midwest U.S.

"Always ask 'How will this information be used?' before starting to include new or revised metrics."
University Medical Center, Northeast U.S.

"Always begin with the end in mind. Know what you want to achieve and what success will look like, then figure out a way to track, assess, and evaluate the progress and outcomes that will answer the questions."
County Public Health Provider, Midwest U.S.

"Be focused. Identify what really matters. Do not try to be all things to all audiences."
Financial Investment Group, East Coast U.S.

"Begin your program with a clear idea of what you want to track and how you'll use it to reflect change in the community. However, make sure that program analysis and evaluation is part of your plan, so that you'll be able to adapt if you find that you either weren't achieving the outcomes you'd hoped for or were in actuality achieving other unintended, but meaningful, outcomes."
Community-focused Alternative Lender, Northeast U.S.

"Our biggest piece of advice would be that it's okay to start small, with just a few metrics. Part of the value of our work will be to have data over time that shows change as a result of our interventions."
Community Development Lender, East Coast U.S.

"Figure out how to measure outcomes. We know how many people increased their income and by how much. It helps to tell a story to have measurable outcomes. We have started tracking health outcomes in our wellness coaching now such as reduced tobacco use, weight loss, feeling of well-being, and other outcomes."
Poverty-prevention Community Agency, Midwest, U.S.

"In choosing performance measures, relate short-term and intermediate measures with long-term measures. Balance measure validity with practicality when possible. Stratify measures to identify sub-populations who may be under-served or may inequitably benefit from current programs, systems or policies. Develop broad coalitions and partnerships that will support reporting to build transparency and accountability."
Public Health Agency, East Coast U.S.

"Identify realistic outcomes and measures that are meaningful to the community as well as your initiative. This can be done by creating logic models with all involved. This process creates buy-in for collecting data and keeps all on track. Healthy People 2020 can help identify measures and how to collect them. Also, a community health needs assessment is a great way to collect baseline and follow-up data to measure progress on initiatives. We have had success in doing this by engaging local partners and community members and using mixed methods to collect data. Relying on secondary data is not enough as this data is often out of date and does not measure the need for services. Primary data can help measure the impact of initiatives and the health of the community."
Community Health Improvement Program, East Coast U.S.

"Regular reporting and feedback to the community at large on progress to date is important and builds trust with partners and community members.  Creating a mechanism to receive feedback is also important for quality improvement and quality assurance purposes."
Community Health Improvement Program, East Coast U.S.

"It's very challenging to measure changes in county, zip code or census tracks over short periods of time, although that is our ultimate goal. We attempt to select measures that match more focused efforts such as access to healthy food by food share recipients at farmers' markets in areas known to lack food choices or to measure improvement of screenings recommended for certain populations."
Academic Medical Center, Midwest U.S.

"Keep it simple. Identify indicators and metrics that are readily available or easy to collect. We use many indicators from the American Community Survey at the County and census tract level, but these may not be as precise as what our work groups need. We also collect information via surveys and focus groups, but this takes a lot of time and planning. We are working as a community to identify those indicators that best indicate progress towards our goals and objectives, which are routinely collected and can be analyzed and shared with our team.
Public Health Department, Northwest U.S.

"Pick metrics that are actionable, both near term and long term. Also, select performance measures around process to keep initiatives moving."
Public Health Department, Northwest U.S.

"Start somewhere. Pick any outcome measure to overcome the challenge of measuring small changes within the community over time, then keep asking questions."
Integrated Healthcare Delivery System, Midwest U.S.

 

Developing Measurement Systems

"It's important to build outcome measurement in at the outset of the program, setting expectations and identifying key metrics in advance."
Community Development Financial Institution, New York

"At the time we started our programs there was little talk of how they should be measured. Had we thought about and elicited the assistance of someone with expertise in program evaluation, our program design would have been somewhat different from the outset. So, think about what you are trying to accomplish and what meaningful measurements can be achieved in tandem from the beginning."
Acute Care Hospital, Texas

"We have developed a strategic planning system to guide our community health improvement process. It leads from our vision, mission, and core values. Every three years we complete a Community Health Snapshot, which identifies measurable objectives and reports the current state of health and well-being in our community as a "report card" (as compared with national baselines and targets)."
Preventative Health Agency, California

"First, take a look at what your borrowers, constituents, and others are already measuring. Find the overlaps, and leverage, and add to them to avoid duplication and redundancy."
Community Development Financial Institution, Southeast U.S.

"Be clear on what you are measuring at the outset, but don't make the process too complex or onerous with respect to unique data collection requirements. For example, try to work with standard, routinely available health status surveillance indicators as much as possible. Be patient and flexible in working with community partners."
Public Health Institute, Northeast U.S.

"Clearly, community economic development is beneficial to community health improvement. However, it is not always possible to create a logic model that can show perfect causation. The inability to create that logic model should not prevent organizations from trying to tackle the root causes of poor health. As a result, with work that is further upstream, the focus tends to be on outputs rather than outcomes. This is not inherently a negative, although it is important to recognize that what does get measured, is what often gets done. A broad set of outputs can be useful in ensuring that an organization's work doesn't become too parochial."
Economic Development Agency, Midwest U.S.

"Don't forget the qualitative. It may be easy to do a survey, but the numbers themselves alone don't move people to action."
County Public Health Department, California

"Effective measurement is difficult. We recommend planning ahead to measure rather than trying to capture results after the fact."
Community Service Agency, Midwest U.S.

 "Every program to improve the health of our community that we initiate has explicit measures for inputs, outputs, and for derived outcomes. We will not take on a program without metrics. Our programs range from disease prevention to disease management, especially chronic diseases such as diabetes, heart failure and the like.  We track fetal, infant, and maternal mortality and childhood morbidities from pre-kindergarten to high school graduation.  These time-series of rates and related summary measures are essential in evaluating what interventions work and what do not."
Health Improvement Outreach Program, Midwest U.S.

"Every project must have a clearly defined health outcome measure. Furthermore, for complex issues, it is more effective to rally people around a common population health goal, than a common activity."
Multi-sector Health Coalition, Southeast U.S.

"A few questions to ask when new initiatives are started: What does the current data tell us about the problem? How can we use data to guide the development of the work? What systems are in place that affect problems such as education, healthcare, and transportation? What can be done on a systems level for long-term change? Often the idea is to move towards immediate action rather than the development of complete data sources, setting objectives, tactics and metrics. These are the things that will confirm if interventions are effective."
Social Services Organization, Southeast U.S.

"Impact measurement is difficult, time consuming, and expensive. However, the knowledge gained and the ability to translate outcomes to stakeholders and investors is invaluable. Reach out to specialists and partners so that impact measurement is a collaborative and informed process."
Nonprofit Community Development Financial Institution, East Coast U.S.

"Impact measures need to be done in a collective fashion. Multiple organizations and sectors have an impact on improving the health of the community, but measures are often not well coordinated."
Public Health Department, East Coast U.S.

"Measure, measure, measure - involve community members from the beginning in the planning and evaluation process, train people early to allow for sustainability, celebrate any success, no matter how small, and keep outcome measures simple but consistent."
Integrated Healthcare Delivery System, Midwest U.S.

"We worked with the State Department of Health on developing SMART objectives for our Community Health Improvement Plan. Public Health is an excellent partner in lending their expertise is measuring performance. My advice is to find your local Public Health District and work closely with them to develop your CHNA, CHIP, and performance measurements."
County Health Improvement Group, Northwest U.S.

"There should be a discussion and consensus of the use of metrics with the target community or population during the development of initiatives to assure trust, relevance, and overall utility."
Health Improvement Foundation, Texas

 

Data Collection and Reporting

"It is very important to clearly define the goals of your program and identify your target audience. Be sure to include everything you may need in the data collection from the beginning of the program versus having to go back and add it in later."
Healthcare Provider, Midwest U.S.

"Use existing data where possible to reduce the burden on staff, as well as validated indicators used by your targeted audience so they are valid, meaningful and comparable. Having said that, think about how those indicators apply to what your initiative is."
Nonprofit Housing Advocacy Group, East Cost U.S.

"At the implementation level, initiatives that require additional data collection efforts by staff are most successful if tied to existing priorities, initiatives, or funding requirements faced by staff and organizations."
Nonprofit Healthcare Providers' Association, Northwest U.S.

"Be aware that you need to make time and space for conversations about data to occur.  Often times the 'data people' from different disciplines never get to meet each other and don't get a chance to explore how they may be able to tell a more complete story if they plan for data collection and data sharing where appropriate."
Housing Finance Agency, Midwest U.S.

"Develop a clear and detailed evaluation plan. Consider a limited set of key data points to collect that will help you prove the impact and effect of your program. Collecting tons of data won't necessarily tell the story you want and may hinder the data collection process."
Nonprofit Food Access Organization, East Coast U.S.

"Gathering data and determining its impact is a long term process. You must be diligent in your efforts and get the community involved to make a wider impact."
Nonprofit Healthcare Provider, Midwest U.S.

"Get organized with data very early! Root your strategies with data and develop a strong logic model. Do not look at the data as only good or bad, but rather, use data to spark conversation and innovate for improvement. Share data and information with the community and ask what they want to know and learn about themselves and their community. Lastly, in your data collection, be sure to invest in someone to clean and present data in a digestible format. Use infographics to tell a story about your data and community. This helps folks understand the data better and is simply beautiful."
Nonprofit Community Development Organization, California

"Hire staff who know how to determine what data to collect and how to analyze it and sell its value to other staff. Work with other agencies. If we shared more data, we might be able to track community-level success and we might even be able to understand the impact of social service programs on individuals over time and multiple providers. Sell funders on the idea that it's worth funding performance measurement explicitly."
Day Center for Homeless Families, Midwest U.S.

"Track data longitudinally; it takes time to measure long term change. This isn't conducive to partnerships that want to remain active and have instant results, therefore short term wins like workshops, events, and reports must be planned as a primary focus to achieve the long term outcomes."
Public University, Texas

"You have to be willing to implement changes, track the impacts, and continue to refine based on the results. If you are afraid to try or want perfect results you will likely never implement meaningful programs because it is hard to directly apply what other communities have done due to the geographic nuances that make each place unique."
Affordable Housing and Community Development Organization, California