Evaluating the Soundness of Research
In an age of information overload, it can be hard to know whether the research we encounter is trustworthy. Sound research can deepen our understanding of what is happening and why. Unsound research may instead misinform, waste time and money, and mislead decision makers.
Whether you are a student, a curious reader, or a leader hoping to commission research of your own, below are four questions you can ask to help assess the quality of social science research.
-
Who conducted the research?
Start by identifying who conducted the research. They could be scholars affiliated with institutions such as universities, think tanks, nonprofit organizations, or government agencies. Different types of institutions engage in research for different reasons, and they may operate under different norms and standards. Some may fund their own research, while others may accept funding from outside sources. Sound research firms will communicate this information openly and transparently.
When looking to hire research services, consider asking these questions:
- Who will lead the project and what are their qualifications?
- Has this research firm worked with organizations like yours before?
- How do the researchers ensure integrity throughout the process?
-
How are data collected?
Social scientists use several different methods to collect data, and the method should match the research question. If you have a “what” question (“What percentage of young people have at least one trusted adult in their lives?”) a survey may be most suitable. For “why” or “how” questions (“How do adults become trusted adults in the lives of young people?”) interviews or focus groups may be your best bet. Good research doesn’t just pick the right method—it also executes it well.
When discussing methods with potential research partners, ask about:
- What method or methods are best suited for this research question?
- What are the strengths and limitations of these methods?
- Will study instruments (in other words, survey or interview questions) be tested for clarity and resonance before launch?
-
Is the sample appropriate?
A sample is a group of people meant to represent a larger group that you are trying to learn about. If such a sample is collected carefully and systematically, the findings from that smaller group can tell you something meaningful about the larger one. Say, for example, you are interested in understanding how teenagers at a high school feel about the mental health resources available to them. You might not have the time or resources to talk to every single student, so you gather a smaller group with the goal of generalizing to the larger student body. Talking only to honors students would be interesting, but they may not necessarily represent the broader school experience. To get a better sense of the whole community, you need a sampling strategy that captures students from different grade levels, backgrounds, social groups, and academic tracks.
When commissioning your own study, ask the researchers:
- How many study participants should be included and how will they be recruited?
- Will the sample size be large enough to draw meaningful conclusions?
- Will it reflect the diversity of the broader population of interest?
-
Is the research responsible?
Responsible research, or research conducted with integrity, respects the dignity, rights, and privacy of participants. Committing to responsible research methods is especially important when working with vulnerable or marginalized populations. Conducting responsible research includes obtaining informed consent, protecting confidentiality, and clearly communicating the purpose and potential risks of the study. Integrity also requires researchers to report findings honestly and to avoid exaggerating or manipulating them to fit a preferred narrative. A trustworthy research partner will treat research ethics as a priority rather than a formality.
Consider asking these questions about research integrity before hiring a research partner:
- Has your staff received training in research ethics and research participant protection?
- Will this study be submitted to an Institutional Review Board or ethics review?
- Are there any risks to participants in this study, and, if so, how will they be minimized?
Research quality and why it matters
Assessing the quality of research is essential for making informed decisions. Whether you are interpreting findings from a published report or commissioning a study of your own, knowing how to assess the soundness of research empowers you to consume, understand, and apply information effectively and responsibly.
Interested in commissioning research of your own? Follow the links to learn about Springtide’s services and methods.
