The PRO-RES Framework for Ethical Evidence

The PRO-RES Project has produced a guidance FRAMEWORK that encourages policymakers and their advisors to seek evidence for their decisions from research that has been conducted ethically, responsibly and with integrity. This normative framework includes the following elements:

This framework is based upon principles of normative ethics which refers to the grounds of meaning or acceptance of decisions about why certain behaviour is right or wrong. The varieties of normative ethics are typified as primarily deontological (see deontological ethics) or teleological (see teleological ethics). this is about the way one ought to behave as a researcher. Normative principles have been established over time from considering moral choices at an abstract level in ethical theorising, from the codes of behaviour established by professional institutions and from the observation in practice of what happens when research is done for the ‘wrong’ reasons or in ‘incorrect’ ways as well as observing the benefits from doing research the ‘right’ way. Thus theorising about ethics can never be divorced from the application of principles in practice (La Follette 2002: 8). (La Follette, H. (ed.) (2002) Ethics in Practice: An Anthology, Oxford: Blackwell.)The central point about normative ethics is that it entails value judgements and how one chooses to behave as a researcher can never be proven right or wrong by appealing to empirical facts. The role of normative ethics is not to recommend any particular course of action but to set out possibilities, help to assess values and assist in the making of informed, thoughtful choices (Thompson 2000: 30–2).Thompson, M. (2000) Ethics (Teach Yourself Series), London: Hodder Headline.

  1. A statement – The  Accord  – which lays out the principles for ethical research which we hope all stakeholders can sign up to and endorse. The statement is for all who are concerned to ensure policies are based upon ethical evidence.
  2. A Toolbox  to supplement the Accord for policy makers and advisors to help them identify ethical evidence for their decision-making processes
  3. Additional supportive Resources  that complement the Accord and the Toolbox.

Together these three ‘pillars’ comprise a normative FRAMEWORK that will support policies constructed on ethical evidence, cover the wide spectrum of non-medical research and offer practical solutions for all evidence-seeking stakeholders, that will comply with the highest standards of research ethics and integrity. The Framework has been explored and tested with the appropriate constituencies and across the full range of stakeholders. These include the producers of research, disseminators and intermediaries, influencers, policy advisers, decision-makers and implementers.

To answer the “Who is this for?” question: it is for policymakers keen to use ethical evidence; it is for policy advisors seeking to offer advice based on responsible sources; it is for researchers and their funders wishing to make sure that policies will be based upon their ethically produced evidence; it is for think tanks wishing to enhance their legitimacy by demonstrating that their reports have been produced with integrity and it is also available for citizens to make their own assessment about the evidential sources of the policies that directly affect them. Learn more about us in youtube.

GOOD RESEARCH: BETTER POLICIES

THE ACCORD

(on ethical evidence in research)

OUR RATIONALE: Good research will help produce better policies. Decision takers and policymakers should be seeking evidence to support their work from the range of expertise on offer. Sound, reliable, transparent research, not driven by ideology or subservient to it and undeclared vested interests, produces robust evidence that can benefit social wellbeing and societal progress. It is in the interests of the scientific community to ensure the evidence produced is reliable and trustworthy and ethically generated. It is in the interests of those who make policy to be able to assure the decision takers (and the general public) that evidence has been generated in the best possible way.

As signatories to the Accord.

  • We recognize that an underpinning by high quality research, analysis and evidence, including policy appraisals and evaluations, is a pre-condition for evidence-based policy-/decision-making, and hence rational policy actions and effective outcomes.
  • As individuals and institutions involved in commissioning, funding, sponsoring or conducting research, collecting or using evidence for policymaking, we aim to be as transparent as possible on how the high quality of that evidence is assured and will flag up any potential conflicts of interest.
  • We agree that to a reasonable degree the independence and integrity of individuals responsible for the conducting and/or gathering of research evidence and its use in policymaking must be respected and supported in ways that ensure the evidence they produce is neither biased nor misleading.
  • We will communicate, employ and/or apply only high quality evidence, research or enquiry, in other words evidence that has been undertaken, gathered, collated and analyzed using sound, robust and ethical methods appropriate to the task.
  • We will ensure that the commissioning, funding, management, conduct, dissemination and governance of research meet high standards of ethics and integrity.

The following points explain the rationale behind the Accord and how it fits into the normative framework – the rationale for which is explained here:

Most of the codes and guidelines for research ethics and integrity are constructed on the basis of a normative prescription or a ‘duty-based’ as opposed to a ‘rights-based’ morality. Although one could conceive of an alternate guidance structure based on rights, it would be much harder to apply since rights are more difficult to define and operationalise and, in practice, are more inclined to conflict with each other. It would certainly be confusing (as it is with the current mix of rights and duties in European law) to try reconcile a ‘rights’ approach with ‘duties’ under the law, to ethics and to the research profession. The comparison of rights-based and duty-based moralities brings out the problem of all ethical principles being in tension. The writers of codes and guidelines are constantly trying to reconcile such tensions – any right to be informed will always be contradicted by a right for data not to be disclosed. It would only take one respondent in many datasets to seek anonymity for the rest to have to be anonymised – thus restricting its availability – even perhaps for tests of validity and reliability to be conducted by other researchers. The PRO-RES approach was always based on resourcing reflective practice rather than on formal bureaucratic compliance. And this approach was vindicated in all the work done with stakeholders. The framework was constructed by drawing upon the views and ideas of the full range of key stakeholders in an iterative process. It has built upon previous foundational research ethics codes, guidelines and frameworks in an assessment of what elements of these foundations have ‘worked’ in influencing and informing policymaking in the past.We advocated from the outset that the framework should come in the form of advice and ‘guidelines’ rather than a prescriptive or sanctionable code together with a practical toolbox both to identify best practices (checklists) and to provide models in order to measure the impact of non-deployment of such best practices. This must be promoted as a ‘what works’ approach. Experience suggests that the more regulatory a code, the more malpractice is encouraged if othe institutional and/or infrastructural pressures not to behave ‘well’ remain unaddressed. This represents a pragmatically-oriented ‘virtue ethics’ approach; one that encourages and rewards responsible conduct in researchers and their employing and/or funding institutions. This too was endorsed in our interactions with the full range of stakeholders. It is always possible to construct prescriptive codes if the power to apply sanctions, such as restricting access to funds or delegitimising the ability to offer research services – we offer one such suggestion here.
Links are supplied to supportive resources that will help in seeking to promote ethics and integrity in the evidence produced in all non-medical research.

Here the Accord statements are repeated with explanations to ensure that the meanings behind the statement is clear.

  • We recognize that an underpinning by high quality research, analysis and evidence, including policy appraisals and evaluations, is a pre-condition for evidence-based policy-/decision-making, and hence rational policy actions and effective outcomes.

What this means:

  • Under a commitment to evidence-based policy, all evidence should be based as far as possible on ethically sound research and analysis. This applies to all who are paying for the research to be conducted, those who do the research and those who make use of the research findings.
  • There are many forms of research and evidence. They include not just formal research projects and programmes, but a range of actions relating to investigation, collation, discovery, exploration, practice, and disciplinary development.
  • Research and analysis, in order to be seen as conducted ethically, should be beneficent (or at least non-maleficent) in its aims, its substantive focus, in the process of research, and its application.
  • High ethics standards in research can be found in the range of ethics codes and guidelines that are available and to the statements established as part of the PRO-RES Project.
  • Research, enquiry, analysis and policy advice should not be based on pre-formed prejudicial ideologies or biased political or financial interests.
  • Conflicts of interest should ideally be avoided in the production of evidence and in the provision of policy advice. If this is not possible, all conflicts of interest (LINK) should be openly disclosed.
  • As individuals and institutions involved in commissioning, funding, sponsoring or conducting research, collecting or using evidence for policymaking, we aim to be as transparent as possible on how the high quality of that evidence is assured and will flag up any potential conflicts of interest.

What this means:

  • In order to produce high quality evidence, research and analysis must be methodologically robust.
  • Only research and enquiry that has also been conducted ethically and with integrity can be considered ‘high quality’.
  • It is clear that there may be limits to ‘transparency’(click to glossary)which may depend upon varying commitments, say, to funders and/or to research subjects. Recognition and declaration of those limits is consistent with a commitment to transparency.
  • Similarly, all participants to the research process (stakeholders) are likely to have some ‘vested’ interests, although such interests will not necessarily be ‘in conflict’. Interests should be declared (made transparent as much as possible) and any potential conflicts of interest declared.
  • We agree that to a reasonable degree the independence and integrity of individuals responsible for the conducting and/or gathering of research evidence and its use in policymaking must be respected and supported in ways that ensure that the evidence they produce is neither biased nor misleading.

What this means:

  • Whenever possible, all sources of information used to formulate evidence should be acknowledged, with exceptions being well-justified and, if feasible, noted (for instance in the case of confidential information or views).
  • The processes and institutions involved in the selection of evidence, including research, to inform policy should be as independent, open and transparent as possible. Thus the phrase’ reasonable degree’ acknowledges the difficulty of complete independence of action – for all stakeholders. This recognises that ‘independence’ is difficult to define and that inevitably has limits. No participant in the research and evidence-gathering process can be truly considered completely independent. Where constraints on independent action exist the nature and extent of such constraints need to be declared.
  • The effectiveness and impact of all policies should be honestly and transparently assessed or evaluated using high quality research and analytic methods.
  • We will communicate, employ and/or apply only high quality evidence, research or enquiry, in other words evidence that has been undertaken, gathered, collated and analyzed using sound, robust and ethical methods appropriate to the task.

What this means:

  • Ethical issues can arise at every stage of research: conception, development, proposal, process, conclusion and dissemination. It follows that ethical consideration cannot be a single-stage process; it has to be continuous.
  • Researchers and analysts have to be aware of, and sensitive to, the ethical dimensions of their work. That awareness depends on engagement in ethical discourse as an integral aspect of engagement in research and analysis. Ethical conduct cannot adequately be guaranteed by a fixed number of pre-set rules
  • All researchers and analysts should aim to develop a culture of ethical enquiry, based on continuous discursive engagement. To achieve this, there has to be engagement of everyone responsible for the process, including researchers, analysts, stakeholders, peers and the users of research.
  • We will ensure that the commissioning, funding, management, conduct, dissemination and governance of research meet high standards of ethics and integrity.

What this means:

  • All research should be funded, managed, conducted and disseminated ethically and with integrity. All those involved in the research process – from original idea to findings and applications of results – share responsibility for ethical practice and outcomes.

To help implement the Accord and the principles behind it users need to know:

How to conduct research ethically and with integrity (for researchers, managers and funders).

How to ensure research is conducted ethically and with integrity(for reviewers in research ethics appraisal).

How to supply evidence for effective policymaking (for researchers, managers, funders).

How to select good quality research(for science/policy advisors and policymakers).

How to evaluate the ethical impact of policies(for ALL above stakeholders).

HOW TO ASSESS THAT EVIDENCE IS ETHICAL?

The key ethical questions to be asked about any research output, scientific finding, evidence-based policy advice or similar are: WHO did WHAT to WHOM, WHEN, WHERE, HOW and WHY?
AND how were the findings PUBLISHED, DISSEMINATED &/or APPLIED? AND what were the CONSEQUENCES of the use to which the findings were
put? (EVALUATION)
By seeking the answers to these questions, it is possible to make a judgment about how ethically the research/analysis was conducted and if the researchers/analysts/advisors behaved with integrity. We provide you with a Toolbox that covers all of these elements. These are questions that can be asked of any EGO to judge how ethical their work is.
We also provide some examples of the Toolbox ‘in use’.

A TOOLBOX FOR ASSESSING THE ETHICAL QUALITY OF RESEARCH EVIDENCE

This Toolbox is designed to be straightforward and easy to use. It operationalises the statements contained in the Accord. It parallels the kinds of standard questions asked of any researcher making a research proposal, or asked by any science evaluator or ethics reviewer of such a proposal. Similarly anyone wishing to ‘test’ a researcher and their work for its integrity should be able to ask these questions of them. The questions are applicable to all forms of enquiry seeking to gather data and analyse it for evidential purposes. The sets of questions could appear in any order, although the ‘WHO’ category often comes first. These questions inevitably overlap to varying egrees – but answering them all offers the most comprehensive articulation of the quality of a research engagement. The Toolbox could be operationalized in a range of ways – the first is based on a simple checklist approach. Any organization with the power to apply sanctions, such as restricting access to funds or delegitimising the ability to offer research services can turn these questions into ‘prescriptions’ – we offer one such suggestion here.

WHO were the researchers and the research agency?

For the individual researcher:

  • What are the credentials of the researcher?
  • What is/was their competence; experience; track record?
  • Who do/did they work for?
  • Do they have any vested/conflicts of interest?
  • Do they adhere to any specific professional/ethical codes and/or guidelines?
  • How was the specific project that generated the evidence in question funded?

For the research agency:

  • What are the credentials of the research agency?
  • What is their competence; experience; track record?
  • Who do/did they work for?
  • What kind of research/data-gathering agency are they?
  • How is the agency funded/by whom?
  • How is the agency governed – how was it founded and with what purpose?
  • Does the agency commit to adhere to certain codes/guidelines – does it have a ‘mission statement’?
  • Does the agency have any vested/conflicts of interests?
  • How does it manage data protection regulations?
  • How was the specific project that generated the evidence in question funded?

We present here a draft statement of principles that lie behind seeking/using ethical evidence from non-medical research to inform policy. In talking about ethical evidence, we are both tackling the principle of evidence per se and the way this evidence is built throughout the whole research process from inception through to application or use. The short, clear, succinct and actionable statement we present here is designated the ‘Accord’. This is the baseline that we intend the further consultation process to be built on. Neither its title nor content is ‘fixed’ at this point. We aim to explore its potential with the appropriate constituencies and across the range of stakeholders. These include the producers of research, disseminators and intermediaries, influencers, policy advisers, decision-makers and implementers. The section following the Accord statement draws out the elements of the brief Accord statement in terms of slightly more detailed principles together with a rationale for this approach. The draft Accord is based on the work accomplished by the first phase of the PRO-RES Project and based on declared foundational assumptions about the values, principles and standards involved in ethical research conducted with integrity. The Accord will be presented on the PRO-RES website and linked to a ‘Toolbox’ to aid stakeholders in assessing the ethics and integrity of research evidence and supportive resources to help produce such evidence across the range of non-medical research activities.

HOW was the research/data-gathering and analysis conducted?

  • What exactly was done to gather and analyse the data?
  • What research plan or analytical ‘design’ was used?
  • What specific methods were employed both to gather data and to analyse it?
  • Was there an original protocol made available publicly? (If so, did the research deviate from this? If so, was this justified?)
  • What kinds of data were gathered? (Were there checks for validity, reliability, authenticity of sourcesetc.)
  • How were data managed and analysed?
  • Is there any evidence of bias? If so, where and what?
  • Were other stakeholders (community members, research participants, general public, etc.) involved in any part of the research or data-gathering? If so, why and how?
  • Were relevant personal identities protected and, if so, how?
  • Was the process transparent? If not, why were there limitations on transparency?

There is no implicit judgment of the ‘ethical quality’ of the variety of methods that can be employed. What matters is, again, the transparency of those conducting the research, and their offering of clear justifications/rationale for any methods used. Thus covert research, deception, community/societal engagement, social engineering etc. are not to be regarded as inherently unethical – the judgement of whether they are or not might depend upon the context in which they are used and, whether a policymaker/advisor considers evidence derived from a particular method is justifiable. Neither is there any implication that only primary research is of evidential value – all forms of secondary data analysis can be subjected to these questions: from meta-analyses of controlled experimental studies to simple frequency counts of questionnaire responses. The ‘validity’ of primary research data depends upon the rigour of the research design and its accurate execution; the validity of most forms of secondary data analysis depends upon access to/availability of raw source data. Even documentary or archival analyses are valid to be tested against accurate use of source materials.

WHOM/WHAT was the prime focus of the study?

  • Who or what were the subject/objects/participants of the study?
  • Could these ‘subjects’ or ‘objects’ have been considered vulnerable in any way – or made more vulnerable by the enquiries being conducted?
  • How was the welfare of the subject/objects/participants ensured?
  • How was the welfare of the researcher(s) (if appropriate) ensured?
  • Can any risks of harm be foreseen/anticipated and mitigated as a consequence of engaging in enquiries/research about/with the ‘objects’ of this study?

These elements concern the relationship between researcher and researched and how the researcher treats the researched. The subjects/objects/participants could have been humans, animals, organisms or parts of such, material objects, ecosystems, organisations, communities, societies etc. – or any combination of the aforementioned. Thus research by economists might be a study of banking ‘systems’ without references to bankers per se. Research enquiries related to public health might be concerned with the public and not individual members of that ‘public’. Researcher welfare issues are likely to arise out of their relationships with the subjects/objects of study – so researcher health and
welfare needs to be considered and any forms of reflective practice they adopt encouraged and disclosed. Once more these questions are not just related to primary research, nor simply to research with humans or live animals – they apply equally to any form of secondary research/data gathering and to material objects or places. Thus, for example, a volcanologist is unlikely to be able to cause undue
harm to the objects of their study, but is likely to put themselves at risk when engaging with the primary objects of their attention. On the other hand, if they adopted some physical engagements with volcanoes(bombs?) – the possibility of harm to other aspects of the ecosystem and communities has to be envisaged.

WHY was the research/enquiry/analysis conducted?

  • What was the purpose of the research enquiry?
  • For what reason was it conducted?
  • Who supplied the funding
  • How was it funded?
  • What were/are the funder’s intentions?
  • What were/are the researcher’s intentions?
  • What were/are the research agency’s intentions?
  • Were participant communities involved in determining the need for this research?
  • Were potential impacts evaluated and appropriate actions planned?

Motive and intent are key ethical issues. They go to why the research was conducted in the first place and what outcomes were hoped for and by whom. Impacts could be environmental, social, psychological, political etc. Hence the question of who commissioned and funded the research/enquiry is doubly important – details on the funding agency is key to full transparency.

WHEN and WHERE was the research/analysis conducted and/or policy advice provided?

  • In what context was the research/analysis/enquiry carried out?
  • What was the nature of the research site/setting?
  • When was the research/analysis conducted?

Most ethical judgements rely upon a full understanding the context in which the action under consideration occurred – the place and the time. This requires a comprehensive understanding of place and time: geographical, institutional, organizational etc. and diurnal, annual, chronological, historical and so on. Thus there are wide variations between a laboratory site, urban settings entailing risk and threats, libraries, and high- and low-resource countries. Laboratories can vary in licensing levels, while field sites vary in the kinds of permissions required. Historical archival research varies considerably in terms of ethical risk from the study of more contemporary documentation but engaging in historical enquiry may still entail risks to the present in terms of societal or communal stigmatisation and/or reputation. For example, knowledge of how and why a particular organisation was established may ‘taint’ its current reputation.

Was the research REVIEWED in advance for its scientific or analytic ‘quality’ and its adherence to ethics?

  • Is any form of pre-project review of the approach provided for within the institutional/sectoral set-up?
  • Was there independent review/appraisal by a competent body for the ethical issues raised by this research/enquiry?
  • Who reviewed the research methodologically/scientifically for quality issues prior to its implementation?
  • What regulatory approvals were granted for the research if any?
  • What additional permissions were necessary/granted for the research?

There are many stages/steps in terms of approval and/or appraisal processes to assess the quality of and risks (ethical etc.) for research projects. In some countries/institutions these processes are absent, but the increase of multinational, interdisciplinary approaches to research implicates researchers in ensuring some formal reviews are conducted. Reviewing standards and standard operating procedures are increasingly shared internationally and across institutions. In addition novel citizen science evaluation methods are emerging such as crowd reviewing. It may be difficult for all forms of research/analytical agency to secure independent assessment for the ethics of their work. Increasingly organisations do strive to establish their own in-house system with a degree of independence provided by some external memberships. There is no ‘best’ or single way of doing this, the importance again is for transparency – clarifying if any form of assessment of quality and ethics is done prior to the commencement of research and/or enquiry.

What were the OUTCOMES of the research?

  • How were the research/analysis findings reported, shared and/or disseminated? What policy advice was derived and given?
  • If parts/all of the analysis were not published, what was the reason for this?
  • How ‘selective’ were the reporting of findings?
  • Were the research findings implemented in practice – i.e. ‘applied’ or used?
  • What were the consequences of the findings being, or not being, implemented?
  • Were there any limitations on what could be accomplished with the findings – dissemination and/or application?
  • Could any form of ‘impact assessment’ be performed?
  • Was any evaluation of the outcomes conducted or planned for?

The research findings could be disseminated in a range of different ways – in academic publications, peer-reviewed scholarly publications, in-house technical reports, commissioned reports, independent white papers, official policy documents, policy briefings, participant feedback, social media, news media and so on. What was done with the ‘outcomes’ links back to the original ‘why?’ question, or what was hoped for/intended for the research. The researchers might not be in a position to directly apply the findings, but they might be better able to guide and assist those who can – i.e. the policymakers. A decision might be made to withhold publication of findings – justifications for such an action would have to be clear and strong.

DO’s and DON’T’s

(a simplified checklist)

IF YOU ARE
…a researcher who wishes to supply policymakers with ethically sound, robust evidence; OR …a think tank keen to supply policymakers with ethically sound evidence.

Then fully completing the elements of the Toolbox will supply any interested parties with enough information to judge your ability to supply ethical research evidence.

IF YOU ARE
…a funder or commissioner of research seeking to make sure that the research evidence you are supporting is reliable and trustworthy.

Then checking if those you fund have completed the Toolbox adequately you will be able to judge their ‘worthiness’ for your support.
For those generating evidence (researchers, RPO’s and think tanks):

(NB: IF YOU HAVE COMPLETED THE PRO-RES TOOLBOX – YOU WILL BE ABLE TO SUPPLY COMPREHENSIVE DETAILS FOR ALL THESE ‘DO’s’.)

1) …clearly identify the researchers/research agency conducting the research.
2) …fully demonstrate your ability to conduct the research: your competence, credentials, education
and training.
3) …indicate your track record: outline the nature of your previous work.
4) …state all your funding sources.
5) …declare any vested interests or conflicts of interest you may have.
6) …state the ethical codes and guidelines you follow.
7) …sign up to the PRO-RES Accord.
8) …be clear and transparent about the ideological and/or theoretical bases for your work and how
they could affect your products/services.
9) …specify fully your data management procedures and legal compliance.
10) …detail the specific research methods or data gathering procedures you are able to employ.
11) …detail the methods adopted in any specific research project and/or report you conducted.
12) …explain exactly how you gathered and analysed the data in each research report delivered.
13) …outline how you deal with any potential biases to address issues of reliability, validity and trust
in your outcomes.
14) …with regard to any specific research action, study or investigation – detail the nature of the
‘subjects’ that you studied.
15) …give full details about how you mitigated the risks of any harm coming to the subjects, yourself
or any co-researchers.
16) …outline what you see to see as the benefits of your work.
In addition to all points above, for each specific research project/analysis you conduct:
17) …explain why it was done – the goals that were sought.
18) …indicate all funding sources.
19) …outline the outcomes of the work and if any impact evaluation was conducted.
20) …supply full details of the research site/setting and effects of your work on the environment
(e.g. social, psychological, economic and physical).
21) …give information on any appraisal conducted on the study in terms of ethics and/or scientific
methods. Disclose outcomes/reports on such appraisal.
22) …detail how the findings of the work were reported, communicated or disseminated.

1) …be vague about any of your responses so that there can be no ambiguities or
misunderstandings.
2) …name/claim researchers/research agency who are not directly involved in your work.
3) …claim any skills, credentials, education or training which you do not have.
4) …make unsubstantiated claims for your previous work.
5) …hide any of your funding sources.
6) …disguise vested interests or conflicts of interest you may have.
7) …state support for ethical codes and/or guidelines in a tokenist manner. (They should be codes
you are familiar with and can demonstrate your endorsement for.)
8) …claim to be a signatory to the PRO-RES Accord or any other code if you are not.
9) …hide any ideological and/or theoretical bases for your work which could affect your
products/services.
10) …claim data management procedures and legal compliance that are not demonstrable.
11) …omit any details concerning the research methods or data gathering procedures you employ
and which are stated in reports.
12) …omit details about how you gathered and analysed the data in each research report delivered.
13) …attempt to hide any potential biases to the reliability, validity and trust that could emerge in
your work.
14) …leave out any details about how you treat and relate to the ‘subjects’ that you study.
15) …leave out any details about how you minimise any harm coming to your research subjects,
yourself or any co-researchers.
16) …overclaim about what you see as the benefits, consequences and/or impact of your work.

IF YOU ARE:
…a policymaker or policy advisor who is looking to ensure that the evidence you use to support and guide your policies has been generated ethically and with integrity These are the simple DO’s and DON’T’s to help you make a judgement. Any responsible evidencegenerating organisation should be able to readily address these

1) …ask for details about the researchers/research agency conducting the research.
2) …check the researcher/agency’s ability to conduct the research: competence, credentials,
education and training.
3) …review any track record: look at their previous work.
4) …clarify their funding sources.
5) …identify any vested interests or conflicts of interest they may have.
6) …check the ethical codes and guidelines they claim to follow.
7) …check if they are signatories to the PRO-RES Accord or other statements of research integrity.
8) …explore their ideological and/or theoretical bases and how that might influence
their products/services.
9) …examine their data management procedures and legal compliance.
10) …check out the specific research methods or data gathering procedures they employ.
11) …examine their methods of data collection and analysis for any specific evidence they are
offering.
12) …discover how they dealt with potential biases.
13) …ask for detail about how they treated any subjects they studied.
14) …seek an account of what they see as the benefits and/or impact of their work.
15) …check out how they deal with any unintended consequences of their work.
16) …look at how they report or disseminate their work.

1) …allow any vagueness in their responses so that there can be no ambiguities or
misunderstandings.
2) …avoid checking on their skills, credentials, education or training.
3) …forget to look at the outcomes of their previous work.
4) …permit them to hide or not disclose funding sources.
5) …ignore vested interests or conflicts of interest you suspect they may have.
6) …neglect their stated support for ethical codes and/or guidelines.
7) …forget to check if they are a signatory to the PRO-RES Accord or any other codes of integrity.
8) …ignore their ideological and/or theoretical bases.
9) …allow vague claims to data management procedures and legal compliance.
10) …forget to check the research methods/data gathering procedures they employ.
11) …ignore any potential biases that could emerge in their work.
12) …neglect to request details about how they treat and relate to their ‘subjects’.
13) …neglect checking on the claims they make about the benefits and impact of their work.

Some examples can be found at ‘The Research Ethics Review Process’ in the PRO-RES resources: