A TOOLBOX FOR ASSESSING THE ETHICAL QUALITY OF RESEARCH EVIDENCE

This Toolbox is designed to be straightforward and easy to use. It operationalises the statements contained in the Accord. It parallels the kinds of standard questions asked of any researcher making a research proposal, or asked by any science evaluator or ethics reviewer of such a proposal. Similarly, anyone wishing to ‘test’ a researcher and their work for its integrity should be able to ask these questions of them. The questions are applicable to all forms of enquiry seeking to gather data and analyse it for evidential purposes. The sets of questions could appear in any order, although the ‘WHO’ category often comes first. These questions inevitably overlap to varying degrees – but answering them all offers the most comprehensive articulation of the quality of a research engagement. The Toolbox could be operationalized in a range of ways – the first is based on a simple checklist approach. Any organization with the power to apply sanctions, such as restricting access to funds or delegitimising the ability to offer research services can turn these questions into ‘prescriptions’ – we offer one such suggestion here.

WHO were the researchers and the research agency?

For the individual researcher:

  • What are the credentials of the researcher?
  • What is/was their competence; experience; track record?
  • Who do/did they work for?
  • Do they have any vested/conflicts of interest?
  • Do they adhere to any specific professional/ethical codes and/or guidelines?
  • How was the specific project that generated the evidence in question funded?

For the research agency:

  • What are the credentials of the research agency?
  • What is their competence; experience; track record?
  • Who do/did they work for?
  • What kind of research/data-gathering agency are they?
  • How is the agency funded/by whom?
  • How is the agency governed – how was it founded and with what purpose?
  • Does the agency commit to adhere to certain codes/guidelines – does it have a ‘mission statement’?
  • Does the agency have any vested/conflicts of interests?
  • How does it manage data protection regulations?
  • How was the specific project that generated the evidence in question funded?

It is important to note that – given the range of evidence employed in policymaking – we are adopting a very broad definition of ‘research’ – to include all forms of data gathering intended to supply evidence for policymaking. As a result, the agencies gathering the data might include academic researchers, think tanks, lobbying agencies, PR consultants, advocacy agencies, civil society organisations, early adopters/influencers (bloggers, etc.); these criteria do not ‘rule out’ novice researchers, citizen scientists, members of the public, journalists etc. (All these ‘agencies’ could be regarded as ‘Evidence Generating Organisations’ (EGOs).) There is no explicit requirement for only experienced researchers to be treated as ‘legitimate’. The key is to be transparent about exactly who the researcher/agency is and who they are working for – even if it is for themselves. It is to be expected that researcher CVs/résumés would be supplied together with any agency track records, details about the RPO/EGO/Agency’s background and its main funding sources – which could be large corporations with heavily vested commercial interests or crowd funding schemes in which the interests might be more diverse. Mission statements or adherence to codes guidelines and/or professional association memberships would be appropriate here. A key question for the evidence-gathering agency would be how does it fund itself? Does it have a diversity of funding or is it dependent on a particular stakeholder and with what contractual commitments?

HOW was the research/data-gathering and analysis conducted?

  • What exactly was done to gather and analyse the data?
  • What research plan or analytical ‘design’ was used?
  • What specific methods were employed both to gather data and to analyse it?
  • Was there an original protocol made available publicly? (If so, did the research deviate from this? If so, was this justified?)
  • What kinds of data were gathered? (Were there checks for validity, reliability, authenticity of sourcesetc.)
  • How were data managed and analysed?
  • Is there any evidence of bias? If so, where and what?
  • Were other stakeholders (community members, research participants, general public, etc.) involved in any part of the research or data-gathering? If so, why and how?
  • Were relevant personal identities protected and, if so, how?
  • Was the process transparent? If not, why were there limitations on transparency?

There is no implicit judgment of the ‘ethical quality’ of the variety of methods that can be employed. What matters is, again, the transparency of those conducting the research, and their offering of clear justifications/rationale for any methods used. Thus covert research, deception, community/societal engagement, social engineering etc. are not to be regarded as inherently unethical – the judgement of whether they are or not might depend upon the context in which they are used and, whether a policymaker/advisor considers evidence derived from a particular method is justifiable. Neither is there any implication that only primary research is of evidential value – all forms of secondary data analysis can be subjected to these questions: from meta-analyses of controlled experimental studies to simple frequency counts of questionnaire responses. The ‘validity’ of primary research data depends upon the rigour of the research design and its accurate execution; the validity of most forms of secondary data analysis depends upon access to/availability of raw source data. Even documentary or archival analyses are valid to be tested against accurate use of source materials.

WHOM/WHAT was the prime focus of the study?

  • Who or what were the subject/objects/participants of the study?
  • Could these ‘subjects’ or ‘objects’ have been considered vulnerable in any way – or made more vulnerable by the enquiries being conducted?
  • How was the welfare of the subject/objects/participants ensured?
  • How was the welfare of the researcher(s) (if appropriate) ensured?
  • Can any risks of harm be foreseen/anticipated and mitigated as a consequence of engaging in enquiries/research about/with the ‘objects’ of this study?

These elements concern the relationship between researcher and researched and how the researcher treats the researched. The subjects/objects/participants could have been humans, animals, organisms or parts of such, material objects, ecosystems, organisations, communities, societies etc. – or any combination of the aforementioned. Thus research by economists might be a study of banking ‘systems’ without references to bankers per se. Research enquiries related to public health might be concerned with the public and not individual members of that ‘public’. Researcher welfare issues are likely to arise out of their relationships with the subjects/objects of study – so researcher health and
welfare needs to be considered and any forms of reflective practice they adopt encouraged and disclosed. Once more these questions are not just related to primary research, nor simply to research with humans or live animals – they apply equally to any form of secondary research/data gathering and to material objects or places. Thus, for example, a volcanologist is unlikely to be able to cause undue
harm to the objects of their study, but is likely to put themselves at risk when engaging with the primary objects of their attention. On the other hand, if they adopted some physical engagements with volcanoes(bombs?) – the possibility of harm to other aspects of the ecosystem and communities has to be envisaged.

WHY was the research/enquiry/analysis conducted?

  • What was the purpose of the research enquiry?
  • For what reason was it conducted?
  • Who supplied the funding?
  • How was it funded?
  • What were/are the funder’s intentions?
  • What were/are the researcher’s intentions?
  • What were/are the research agency’s intentions?
  • Were participant communities involved in determining the need for this research?
  • Were potential impacts evaluated and appropriate actions planned?

Motive and intent are key ethical issues. They go to why the research was conducted in the first place and what outcomes were hoped for and by whom. Impacts could be environmental, social, psychological, political etc. Hence the question of who commissioned and funded the research/enquiry is doubly important – details on the funding agency is key to full transparency.

WHEN and WHERE was the research/analysis conducted and/or policy advice provided?

  • In what context was the research/analysis/enquiry carried out?
  • What was the nature of the research site/setting?
  • When was the research/analysis conducted?

Most ethical judgements rely upon a full understanding the context in which the action under consideration occurred – the place and the time. This requires a comprehensive understanding of place and time: geographical, institutional, organizational etc. and diurnal, annual, chronological, historical and so on. Thus there are wide variations between a laboratory site, urban settings entailing risk and threats, libraries, and high- and low-resource countries. Laboratories can vary in licensing levels, while field sites vary in the kinds of permissions required. Historical archival research varies considerably in terms of ethical risk from the study of more contemporary documentation but engaging in historical enquiry may still entail risks to the present in terms of societal or communal stigmatisation and/or reputation. For example, knowledge of how and why a particular organisation was established may ‘taint’ its current reputation.

Was the research REVIEWED in advance for its scientific or analytic ‘quality’ and its adherence to ethics?

  • Is any form of pre-project review of the approach provided for within the institutional/sectoral set-up?
  • Was there independent review/appraisal by a competent body for the ethical issues raised by this research/enquiry?
  • Who reviewed the research methodologically/scientifically for quality issues prior to its implementation?
  • What regulatory approvals were granted for the research if any?
  • What additional permissions were necessary/granted for the research?

There are many stages/steps in terms of approval and/or appraisal processes to assess the quality of and risks (ethical etc.) for research projects. In some countries/institutions these processes are absent, but the increase of multinational, interdisciplinary approaches to research implicates researchers in ensuring some formal reviews are conducted. Reviewing standards and standard operating procedures are increasingly shared internationally and across institutions. In addition novel citizen science evaluation methods are emerging such as crowd reviewing. It may be difficult for all forms of research/analytical agency to secure independent assessment for the ethics of their work. Increasingly organisations do strive to establish their own in-house system with a degree of independence provided by some external memberships. There is no ‘best’ or single way of doing this, the importance again is for transparency – clarifying if any form of assessment of quality and ethics is done prior to the commencement of research and/or enquiry.

What were the OUTCOMES of the research?

  • How were the research/analysis findings reported, shared and/or disseminated? What policy advice was derived and given?
  • If parts/all of the analysis were not published, what was the reason for this?
  • How ‘selective’ were the reporting of findings?
  • Were the research findings implemented in practice – i.e. ‘applied’ or used?
  • What were the consequences of the findings being, or not being, implemented?
  • Were there any limitations on what could be accomplished with the findings – dissemination and/or application?
  • Could any form of ‘impact assessment’ be performed?
  • Was any evaluation of the outcomes conducted or planned for?

The research findings could be disseminated in a range of different ways – in academic publications, peer-reviewed scholarly publications, in-house technical reports, commissioned reports, independent white papers, official policy documents, policy briefings, participant feedback, social media, news media and so on. What was done with the ‘outcomes’ links back to the original ‘why?’ question, or what was hoped for/intended for the research. The researchers might not be in a position to directly apply the findings, but they might be better able to guide and assist those who can – i.e. the policymakers. A decision might be made to withhold publication of findings – justifications for such an action would have to be clear and strong.

DO’s and DON’T’s

(a simplified checklist)

IF YOU ARE
…a researcher who wishes to supply policymakers with ethically sound, robust evidence; OR …a think tank keen to supply policymakers with ethically sound evidence.

Then fully completing the elements of the Toolbox will supply any interested parties with enough information to judge your ability to supply ethical research evidence.

IF YOU ARE
…a funder or commissioner of research seeking to make sure that the research evidence you are supporting is reliable and trustworthy.

Then checking if those you fund have completed the Toolbox adequately you will be able to judge their ‘worthiness’ for your support.
For those generating evidence (researchers, RPO’s and think tanks):

(NB: IF YOU HAVE COMPLETED THE PRO-RES TOOLBOX – YOU WILL BE ABLE TO SUPPLY COMPREHENSIVE DETAILS FOR ALL THESE ‘DO’s’.)

1) …clearly identify the researchers/research agency conducting the research.
2) …fully demonstrate your ability to conduct the research: your competence, credentials, education
and training.
3) …indicate your track record: outline the nature of your previous work.
4) …state all your funding sources.
5) …declare any vested interests or conflicts of interest you may have.
6) …state the ethical codes and guidelines you follow.
7) …sign up to the PRO-RES Accord.
8) …be clear and transparent about the ideological and/or theoretical bases for your work and how
they could affect your products/services.
9) …specify fully your data management procedures and legal compliance.
10) …detail the specific research methods or data gathering procedures you are able to employ.
11) …detail the methods adopted in any specific research project and/or report you conducted.
12) …explain exactly how you gathered and analysed the data in each research report delivered.
13) …outline how you deal with any potential biases to address issues of reliability, validity and trust
in your outcomes.
14) …with regard to any specific research action, study or investigation – detail the nature of the
‘subjects’ that you studied.
15) …give full details about how you mitigated the risks of any harm coming to the subjects, yourself
or any co-researchers.
16) …outline what you see to see as the benefits of your work.
In addition to all points above, for each specific research project/analysis you conduct:
17) …explain why it was done – the goals that were sought.
18) …indicate all funding sources.
19) …outline the outcomes of the work and if any impact evaluation was conducted.
20) …supply full details of the research site/setting and effects of your work on the environment
(e.g. social, psychological, economic and physical).
21) …give information on any appraisal conducted on the study in terms of ethics and/or scientific
methods. Disclose outcomes/reports on such appraisal.
22) …detail how the findings of the work were reported, communicated or disseminated.

1) …be vague about any of your responses so that there can be no ambiguities or
misunderstandings.
2) …name/claim researchers/research agency who are not directly involved in your work.
3) …claim any skills, credentials, education or training which you do not have.
4) …make unsubstantiated claims for your previous work.
5) …hide any of your funding sources.
6) …disguise vested interests or conflicts of interest you may have.
7) …state support for ethical codes and/or guidelines in a tokenist manner. (They should be codes
you are familiar with and can demonstrate your endorsement for.)
8) …claim to be a signatory to the PRO-RES Accord or any other code if you are not.
9) …hide any ideological and/or theoretical bases for your work which could affect your
products/services.
10) …claim data management procedures and legal compliance that are not demonstrable.
11) …omit any details concerning the research methods or data gathering procedures you employ
and which are stated in reports.
12) …omit details about how you gathered and analysed the data in each research report delivered.
13) …attempt to hide any potential biases to the reliability, validity and trust that could emerge in
your work.
14) …leave out any details about how you treat and relate to the ‘subjects’ that you study.
15) …leave out any details about how you minimise any harm coming to your research subjects,
yourself or any co-researchers.
16) …overclaim about what you see as the benefits, consequences and/or impact of your work.

IF YOU ARE:
…a policymaker or policy advisor who is looking to ensure that the evidence you use to support and guide your policies has been generated ethically and with integrity These are the simple DO’s and DON’T’s to help you make a judgement. Any responsible evidencegenerating organisation should be able to readily address these

1) …ask for details about the researchers/research agency conducting the research.
2) …check the researcher/agency’s ability to conduct the research: competence, credentials,
education and training.
3) …review any track record: look at their previous work.
4) …clarify their funding sources.
5) …identify any vested interests or conflicts of interest they may have.
6) …check the ethical codes and guidelines they claim to follow.
7) …check if they are signatories to the PRO-RES Accord or other statements of research integrity.
8) …explore their ideological and/or theoretical bases and how that might influence
their products/services.
9) …examine their data management procedures and legal compliance.
10) …check out the specific research methods or data gathering procedures they employ.
11) …examine their methods of data collection and analysis for any specific evidence they are
offering.
12) …discover how they dealt with potential biases.
13) …ask for detail about how they treated any subjects they studied.
14) …seek an account of what they see as the benefits and/or impact of their work.
15) …check out how they deal with any unintended consequences of their work.
16) …look at how they report or disseminate their work.

1) …allow any vagueness in their responses so that there can be no ambiguities or
misunderstandings.
2) …avoid checking on their skills, credentials, education or training.
3) …forget to look at the outcomes of their previous work.
4) …permit them to hide or not disclose funding sources.
5) …ignore vested interests or conflicts of interest you suspect they may have.
6) …neglect their stated support for ethical codes and/or guidelines.
7) …forget to check if they are a signatory to the PRO-RES Accord or any other codes of integrity.
8) …ignore their ideological and/or theoretical bases.
9) …allow vague claims to data management procedures and legal compliance.
10) …forget to check the research methods/data gathering procedures they employ.
11) …ignore any potential biases that could emerge in their work.
12) …neglect to request details about how they treat and relate to their ‘subjects’.
13) …neglect checking on the claims they make about the benefits and impact of their work.

Some examples can be found at ‘The Research Ethics Review Process’ in the PRO-RES resources

The ACCORD Toolbox Applied

ILLUSTRATIVE EXAMPLE 1 : The PRO-RES Project (long version) SwafS-21-2017 
Promoting integrity in the use of research results in evidence based policy: a focus on non-medical research

For the individual researcher:

Dr Ron Iphofen is a fellow of the UK Academy of Social Sciences, the Royal Society of Medicine and the UK Higher Education Academy as well as other professional associations. He retired as Director of Postgraduate Studies in the School of Health Sciences, Bangor University, Wales in 2008 but has been working in the field of research ethics and integrity since the late 1990s and has national and international recognition as an expert on social research ethics and governance and in maintaining professional standards in research. He has presented on this topic at over 50 national/international events for universities, government agencies, research institutes and the EC (several executive agencies) including the ERC and published widely on the topic – with books, book chapters, guidance documents, training packages and articles in the professional press. He founded and edited a journal in social gerontology for fifteen years. He was a member of the UK Universities Sector of the Association of Research Ethics Committees and is an adviser for the UK Research Integrity Office. He has acted as consultant, adviser and/or delivered training on research ethics for the Scottish Executive, UK Government Social Research Unit, National Disability Authority (Ireland), National Centre for Social Research (NatCen), Social Research Association (SRA), Audit Commission, Joseph Rowntree Foundation, Royal National Institute for Blind People, the Food Standards Agency, the Ministry of Justice, the BIG Lottery Fund, West Midlands/South East Local Authorities’ Consortium, Skills Development Scotland, ANR (French Research Funding agency), SSRC (Canada) and many others. He helped establish several codes and guidelines on research ethics – notably the SRA’s ethical guidelines and the EC-funded FP5 Project RESPECT Code of Ethics for Socioeconomic Research. He is a signatory to the PRO-RES ACCORD Statement. Further details and contact available on the researchers’ website: https://roniphofen.com/

For the research agencies:

The agencies conducting this project formed a consortium of 14 European partners considered representative of key areas of nonmedical research. The partners have lengthy experience in their respective research fields and represent professional associations, academic research departments and independent research consultancy/agencies (i.e.industry/SMEs) hence the details of their individual funding forms vary. All partners are transparent about their funding sources on their websites and all conform to the GDPR. The disciplines/research fields covered within the partners of the consortium are social science, economics and finance, engineering, technology (e.g. nanotechnology and biotechnology), environmental science, marine sciences, humanities, philosophy, policy research, data protection/management and crisis/disaster research. The consortium composition and its plurality attempted to cover the ‘non-medical’ disciplines in a representative way, with key communities with important ethical and integrity concerns. The consortium composition therefore represents four ‘knowledge areas’, with individual partners belonging to several areas to assure efficient communication. These are: project management and consultation process experts; researchers in ethics and integrity; partners with large networks covering a range of science areas; and partners with experience in policy advice and with an understanding of ‘policy advice logistics’. The consortium contains the necessary expertise to implement the project programmatically, academically but also to gather the appropriate stakeholders and maximise its impact. There are no known conflicts of interest entailed in this work and the only ‘vested’ interests are to help secure ethically generated research evidence for better policymaking. Further details on each member of the Consortium and their governance structures can be found at: https://proresproject.eu/partners/ This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 788352 and is part of the SWAFS (Science With and For Society) Programme.

The Project is a CSA (coordinating and support action) and so is a multi-partner project and therefore, unlike a Collaborative Project, research and development is not financed, rather the funding is for activities coordinating or supporting research actions and strategies, such as networking and exchange actions, cross-border access to research infrastructure, studies, conferences, preparation of studies and so on. This means the data collection might not be seen as ‘primary’ research – rather secondary and desk research with forms of networking and further development of existing work prioritised. The necessary data were information about existing materials on research ethics, integrity and policymaking, previous EU-funded related projects and networks and the views, opinions and advice drawn from the full range of stakeholders. Data were gathered from literature, websites/online media, workshops and one-to-one online engagements with targeted stakeholders. The design was to develop a user-friendly framework, cumulatively and iteratively, its utility ‘tested’ with stakeholders at key stages and with ongoing links to other ongoing projects – not solely EU-funded. Much of the data were necessarily textual though some quantitative nominal/descriptive data were also adopted for rapid reference purposes. Some data content was tabulated and summarised from more discursive sources – in the form of topic-centred points used to inform a developing framework. The original protocol required stakeholder events, including a mid-term conference, as a primary source of information, guidance and advice – due to the COVID-19 pandemic online interviews replaced a second phase of workshops. The data appear to have been richer as a consequence of spending more time with key individuals. The process has been publicly declared on the website and in deliverables and so has been fully transparent. Nearly all stakeholders were happy for their name and contact details to be shared within the consortium. Some of the individual interviews wished to remain anonymous to make it clear their views were their own and not representative of the organisations of which they were currently a member. The hardest group to gain data from were members of the public, it is hard to see how they would wish to be and could be engaged at this stage in the development of the PRO-RES Framework and resources.

The stakeholders (researchers, funders, study subjects, civil society organisations, media communicators, reviewers, evaluators, scientific advisors and policymakers) are unlikely to be considered vulnerable or made more so by a CSA project that is intended to encourage good evidence being produced to inform better policymaking. Consent for participation was sought from all engaged stakeholders. The project coordinator (ESF) kept in regular contact with all the research partners to ensure their needs were being met. No evident harms emerged as a consequence of this study and the outcomes were seen to be of benefit to society in general.

The European Commission funded this study in order to help ensure that policy makers sought ethically generated evidence to support their policies. The consortium partners had a range of interests for their involvement. For all it was seen as a means for improving the use of good evidence in support of policies, rather than a reliance on lobbying agencies whose evidence might be subject to ideological biases. For many it presented an extension of their longstanding work in promoting ethical research practices and professional integrity amongst the scientific community. It is assumed that members of the stakeholder communities would have participated in suggesting elements of the funding call. The planned impact was primarily to have policymakers endorse the application and use of the final framework developed by the project.

As a CSA/secondary research project, most of the data were gathered as desk research and via direct contact with stakeholders in workshops and online. The data were collected and analysed over a three-year period from 2018 to 2021. No primary data gathering as in ‘field’ or laboratory research was required.

All the partners in the project will have secured an internal review (within their own EGOs) to cover the quality and ethics of the proposed action prior to submission to the funding body. The project went through the standard appraisal for the design and the ethics of the action within the European Commission’s procedures. A high standard of scientific evaluation was awarded and all requirements from the ethics appraisal process were met. The only additional regulatory requirements were those that fell under GDPR and these were all managed independently by each of the consortium partners.

Some internal data collection and analysis was only shared between partners as this was part of the consenting process for stakeholders. Most final reports were made public as either deliverables, on the website or in a range of presentations, on social media and/or in print publications. The website containing the final framework was the major output and targeted mainly at policymakers, their advisors and researchers who wished their work to be used to inform policy. The use of the website was to be monitored for three years subsequent to the end of the project and any possible means to ensure its further durability and responsiveness to change in use were to be explored. Any further evaluation as to impact would be subject to additional specific funding arrangements.

ILLUSTRATIVE EXAMPLE 2 : The PRO-RES Project (short version) SwafS-21-2017 
Promoting integrity in the use of research results in evidence based policy: a focus on non-medical research

For the individual researcher:

Dr Ron Iphofen is a fellow of the UK Academy of Social Sciences, the Royal Society of Medicine and the UK Higher Education Academy as well as other professional associations. He has national and international recognition as an expert on social research ethics and governance and in maintaining professional standards in research. He helped establish several codes and guidelines on research ethics – notably the SRA’s ethical guidelines and the EC-funded FP5 Project RESPECT Code of Ethics for Socioeconomic Research. He is a signatory to the PRO-RES ACCORD Statement. Further details and contact available on the researchers’ website: https://roniphofen.com/

For the research agencies:

The agencies conducting this project formed a consortium of 14 European partners considered representative of key areas of nonmedical research. The partners have lengthy experience in their respective research fields and represent professional associations, academic research departments and independent research consultancy/agencies (i.e. industry/SMEs) hence the details of their individual funding forms vary. The consortium composition and its plurality attempted to cover the ‘non-medical’ disciplines in a representative way, with key communities with important ethical and integrity concerns. The consortium contains the necessary expertise to implement the project programmatically, academically but also to gather the appropriate stakeholders and maximise its impact. There are no known conflicts of interest entailed in this work. Further details on each member of the Consortium and their governance structures can be found at: https://proresproject.eu/partners/ This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 788352 and is part of the SWAFS (Science With and For Society) Programme.

The Project is a CSA (coordinating and support action) so the data collection entailed secondary desk research with forms of networking and development of existing work. The data were information about existing materials on research ethics, integrity and policymaking, previous EU-funded related projects and networks and the views, opinions and advice drawn from the full range of stakeholders. Data were gathered from literature, websites/online media, workshops and one-to-one online engagements with targeted stakeholders. Nearly all stakeholders were happy for their name and contact details to be shared within the consortium.

The stakeholders (researchers, funders, study subjects, civil society organisations, media communicators, reviewers, evaluators, scientific advisors and policymakers) are unlikely to be considered vulnerable or made more so by a CSA project that is intended to encourage good evidence being produced to inform better policymaking. Consent for participation was sought from all engaged stakeholders.

The European Commission funded this study in order to help ensure that policy makers sought ethically generated evidence to support their policies. The planned impact was primarily to have policymakers endorse the application and use of the final framework developed by the project.

As a CSA/secondary research project, most of the data were gathered as desk research and via direct contact with stakeholders in workshops and online. The data were collected and analysed over a three-year period from 2018 to 2021. No primary research was required.

All partners in the project secured an internal review (within their own EGOs) to cover the quality and ethics of the proposed action prior to submission. The project went through the standard appraisal for the design and the ethics of the action within the European Commission’s procedures. A high standard of scientific evaluation was awarded and all requirements from the ethics appraisal process were met. GDPR covered additional regulatory requirements.

The website containing the final framework was the major output and targeted mainly at policymakers, their advisors and researchers who wished their work to be used to inform policy. The use of the website was to be monitored for three years subsequent to the end of the project and any possible means to ensure its further durability and responsiveness to change in use were to be explored. Any further evaluation as to impact would depend upon additional funding.

ILLUSTRATIVE EXAMPLE 3 : Policy Briefing by the European Policy Centre, requested by the Budget Committee of the European Parliament
The benefits of EU membership are not measured by net operating balances (February 2020)

For the individual researcher:

Dr Fabian Zuleeg is Chief Executive of the European Policy Centre, with overall responsibility, including providing strategic direction, managing its staff and resources and representing the EPC since October 2013. He remains Chief Economist at the same time (a post he has held since January 2010). He holds a PhD on the political economy of EU accession from Edinburgh University. Before coming to the EPC, he has worked as an economic analyst in academia, the public and the private sector. His analysis focuses on EU economic policies, including economic governance and Economic and Monetary Union, EU budget, industrial policy, trade, Single Market, European labour markets and Europe’s economic & social model. He has a long-standing interest in the political economy of European integration, with a particular focus on the UK-EU relationship, analysing the impact of Brexit on the UK and the rest of the EU, as well as the process of separation.
He was appointed to the Standing Council on Europe, established by Scotland’s First Minister after the Brexit vote in June 2016, and he is an Adviser to the Scottish Parliament’s Culture, Tourism, Europe and External Affairs Committee. He is currently a Member of the Executive Board of Transparency International EU, Honorary Fellow at the Europa Institute of the University of Edinburgh and Honorary Professor at Heriot Watt University, as well as being on the Advisory Board of the Scottish Centre on European Relations (SCER) and the Advisory Board for the NI/IRL Brexit Witness project. He also chairs the Selection Committee for the European Friends of Accounts for the King Baudouin Foundation. Fabian works closely with decision-makers in the European institutions, the EPC members and partners and the wider Brussels stakeholder community. Fabian regularly comments on current political and economic issues in the EU in the media. He also chairs and contributes to a wide range of debates, conferences and seminars and has researched and published widely on European integration and on European economic and social policies. Further details and contact available on his website: https://www.epc.eu/en/analysts/DrFabian-Zuleeg~fffb4

Marta Pilati is a Policy Analyst at the European Policy Centre (EPC) and joined in October 2018. Her areas of expertise include economic and regional policy, industrial policy, and research and innovation. Before joining the EPC, she worked at the Centre for European Policy Studies focusing on EU economic convergence. Prior to that, she worked as Research Assistant at Cardno Emerging Markets. She holds a Master’s in International Economic Policy from Sciences Po in Paris, as well as a Double Bachelor’s Degree in Economics from Georgia State University in the USA and Economics and Management from Ca’ Foscari University of Venice in Italy. Further details and contact available on her website: https://www.epc.eu/en/analysts/Marta-Pilati~2ad2e4

For the research agencies:

The European Policy Centre (EPC) is an independent, not-for-profit think tank dedicated to fostering European integration through analysis and debate, supporting and challenging decision-makers at all levels to make informed decisions based on evidence and analysis, and providing a platform for engaging partners, stakeholders and citizens in EU policy-making and in the debate about the future of Europe. The EPC was launched as a think tank in 1996. In 2003, the EPC was established as an international not-for-profit organisation under Belgian law. The EPC’s articles of association and subsequent amendments are available online here: https://www.epc.eu/en/about
The EPC is committed to sound financial management and to transparency in respect of its funding sources. Any financial support received for EPC activities, including publications, does not commit the funding organisations in any way and the views expressed are solely those of the authors. Further information on the EPC’s financing can be found here: https://www.epc.eu/en/financing
The EPC is governed by a Governing Board, which is responsible for the management of the EPC. Its main function, in practice, is to provide strategic direction to the EPC and to monitor and evaluate its operations and the implementation of the strategic plan. However, powers relating to the day-to-day management of the EPC are delegated to the Chief Executive, who acts as the EPC’s Legal Representative. The Governing Board meets three to four times a year. Members are appointed by the General Assembly. The current Chair of the Board is David O’Sullivan, a former EU Ambassador. Board members also sit on the General Assembly, which meets once a year to formally approve the annual accounts and adopt the budget for the year ahead, in line with Belgian law. Further information on the EPC’s governing structure, including all board members can be found here: https://www.epc.eu/en/governing-board
The EPC signed up to the European Commission’s voluntary Register of interest representatives in January 2009 (EU Transparency Register No. 89632641000 47). By signing up to the register, the EPC and its staff commit themselves to comply with the Commission’s Code of Conduct for interest representatives. The EPC is a signatory to the PRO-RES CCORD Statement.

The analysis is based on desk research. Data was gathered from literature and quantitative data from different official sources, such as the European Environmental Agency and statistical data from the European Commission, Eurostat and others. At all stages, the authors critically assessed the reliability of sources and the data before using them. For their analysis, the authors made use of several economic models, as outlined in the footnotes of the text. Furthermore, the authors were in contact with analysts from the German economic research institute ZEW, who were commissioned by the European Parliament to conduct other connected research in the context of this project to exchange views and review each other’s work. No further stakeholders were involved in the research or datagathering.

The primary focus of the project was the EU budget and the EU’s financial architecture as such. As the research also focused on the EU’s policy approach so far, EU institutions and their officials (on a general, not individual level) can be seen as secondary research focus. In its recommendations, the project focused on the European Parliament and its Budget committee to inform their debate and decisions. All objects are not to be considered as vulnerable.

The European Parliament funded this briefing. It was one of four Briefings requested by the European Parliament’s Budget committee in order to inform their approach and decisions towards the negotiation of the next Multiannual Financial Framework. The EPC’s research interest stems from its experience in the work on the EU budget and the EU Single Market and its objective to contribute to decision-makers at all levels making informed decisions based on evidence and analysis. The planned impact was to inform and influence the decisions within the European Parliament based on the EPC’s analysis and thus to contribute to a fair and balanced EU budget.

The research was carried out in the context of the European Parliament’s preparation for the ongoing debate on the next EU budget. Most of the data were gathered as desk research. The data were collected and analysed throughout December 2019 and February 2020 and was, after input from the German economic research institute ZEW and passing the EPC’s internal peer-review process, published on 19 February 2020.

Following a call for tenders by the European Parliament, the project proposal went through the standard application process within the European Parliament’s procedures for research funding. A high standard of scientific evaluation was awarded and all requirements from the ethics appraisal process were met.

The briefing was the main output of this project. It targeted mainly the European Parliament and its Budget committee, but in a broader sense also to policymakers from other EU and national institutions, researchers, economic associations and corporations, civil society organisations and the public. The briefing was sent to the members of the European Parliament’s budget committee after its publication. It was supposed to be presented to and discussed with the committee members. However, due to the coronavirus pandemic, the presentation had to be cancelled. No evaluation was planned to take place.