• 1 Introduction

  Generally, the intention of legislation will be to address, steer or deter certain behaviour either exhibited or expected by individuals, entities or both, and depending on the legislative approach chosen, some form of coercion or incentivisation will be used to ensure compliance in order to reach the intended goal.¹x M. Weber (edited by G. Roth and C. Wittich), Economy and Society: An Outline of Interpretive Sociology (2013), at 34-5; see also C. Coglianese, ‘Measuring Regulatory Performance; Evaluating the Impact of Regulation and Regulatory Policy’, OECD Expert Paper August 2012:1, at 8. Financial legislation, as any other legislation, can be considered effective when it successfully produces the intended results, those results usually being the degree to which the intended goal of the policymakers is actually achieved.²x C.K. Haarhuis and B. Niemeijer, ‘Wetten in werking; Over interventies, werking, effectiviteit en context’, 2 Recht der Werkelijkheid 9-34, at 10-11 (2008); M. Mousmouti, ‘Operationalising Quality of Legislation through the Effectiveness Test’, 6(2) Legisprudence 191-206, at 201 (2012); H. Xanthaki, ‘An Enlightened Approach to Legislative Scrutiny: Focusing on Effectiveness’, 9(3) European Journal of Risk Regulation 431-44, at 432 (2018). The measure of effectiveness is thus the result of the sum consisting of whether (i) there has been a successful achievement of (ii) the intended results (i.e. the goal or purpose of the relevant legislation).³x I. Maher, ‘Competition Law and Transnational Private Regulatory Regimes: Marking the Cartel Boundary’, 38(1) Journal of Law and Society 119-37, at 126 (2011). Consequently, the legislative objective has to be clear, and those parties who are required to comply (the addressees) need to be equally clear and, finally, the legislative approach needs to fit both the legislative objective and the addressees.
  In practice, it may be difficult to clearly determine the objective (parts of) financial legislation aims to achieve. This is partly because financial legislation often has complex structures, in the sense that it is layered, divided into many subsets of legislation, regulations and bylaws – all of them often having their own specifically defined individual objectives – and usually has interdependencies with other legislation. The fact that financial legislation has shown an almost exponential growth since the financial crisis of 2008 with a general view of ensuring that such a financial crisis would not occur in the future has proven to be an important contributing factor to this development.⁴x S.L. Schwarcz, ‘The ‘Principles’ Paradox’, 1 European Business Organization Law Review 175-84, at 178-80 (2009); C. Ford, ‘Principles-based Securities Regulation in the Wake of the Global Financial Crisis’, 55(2) McGill Law Journal 258-307, at 263-5 (2010); J. Black, ‘Forms and Paradoxes of Principles-based Regulation’, 3(4) Capital Markets Law Journal 425-57, at 426 (2008); J. Black, ‘The Rise, Fall and Fate of Principle-based Regulation’, LSE Law, Society and Economy Working Papers 2010:17, 1-25, at 3. The absence of a clear objective, or the absence of agreement thereupon, resonates throughout the legislative process and directly affects the possibility of performing adequate effectiveness reviews.⁵xSee Office of the Parliamentary Counsel, When Laws Become Too Complex. A Review into the Causes of Complex Legislation (2013). For example, if the intention is to establish a norm with which full and unabridged compliancy is required, e.g. deterring destructive or dangerous behaviour, such legislation would generally benefit from clearly formulated objectives aimed at either a general or specific group of addressees.⁶x D. Casey and C. Scott, ‘The Crystallization of Regulatory Norms’, 38(1) Journal of Law and Society 76-95, at 82 (2011); R.T. Morris, ‘A Typology of Norms’, 21(5) American Sociological Review 610-13, at 611 (1956). Generally, such legislation will usually engage in an instrumental approach and be rule-based in its application.⁷x K.W. Abbott and D. Snidal, ‘Hard and Soft Law in International Governance’, 54(3) International Organization 421-56, at 431 (2000). Alternatively, legislation may utilise a more holistic approach, intending to foster compliance through a principle-based approach through more generally formulated norms, focussing on the outcome rather than on strict norm compliancy.⁸x Schwarcz, above n. 4, at 178-80. Some laws aim to codify norms that are universal in nature and as such are mere representations of, to a certain extent, universal perceptions in society on what is acceptable behaviour, whilst others intend to create new norms (of expected behaviour),⁹x T.L. Meares, ‘Norms, Legitimacy and Law Enforcement’, 79 Oregon Law Review 391-415, at 398-9 (2000). the latter often being the case for financial legislation.¹⁰x J.H. Cochrane, ‘Challenges for Cost-Benefit Analysis of Financial Regulation’, 43(2) The Journal of Legal Studies 563-605, at 577 (2014). Consequently, if consensus on the legislative objective is absent or if the legislative objective proves unclear, the assessment as to whether such an objective has been achieved and, connected thereto, the proper addressees have been identified and if the appropriate legislative approach has been chosen will be difficult to assess.
  In this article, I argue that whether financial legislation is effective may theoretically be assessed through the questions: (i) what is the legislative objective? (Section 2), and (ii) who are the intended or expected addressees (Section 3) and, finally, (iii) what is the legislative approach and is this expected to be suitable to the outcome of (i) and (ii) (Section 4)? Following these Sections, I highlight the complexities connected to the actual application of these questions in practice, both through literature research (throughout Sections 2 to 4) and by applying these questions on a short case study of product approval regulation in, respectively, the Netherlands, the United Kingdom and as set out by the European Banking Authority (EBA; Section 5). I subsequently set out the considerations of why, despite the straightforward questions resulting from Sections 2 to 4, in practice the actual determination of effectiveness of financial legislation is so difficult (Section 6). I summarise my findings in Section 7.

• 2 What Is the Legislative Objective?

  Determine the legislative objective and what is the actual source of the legislative objective? Is this legislative objective also addressed in other legislation and how, if at all, is either affected by the other (e.g. which objective is the prevailing objective)?

  The first step in an effectiveness review is to determine the objective that the legislator aims to achieve through the specific legislation. The source of the legislative objective often is not (only) legislation itself but is also to be found in the legislative history and explanatory documents that are at the basis of the relevant legislation. In addition, the complexity of contemporary financial legislation, whereby general principles are set out in a main body of legislation or framework legislation, supplemented with subsets of legislation or regulation, results in different objectives depending on which layer of legislation is reviewed, all of those objectives arguably in support of the overarching objective of the main body of legislation. Simultaneously, the question has to be asked whether the objective may also be addressed directly or indirectly in existing legislation. In practice, this may prove to be a challenging question to answer, as with the still growing body of financial legislation, even the legislator finds it difficult to maintain full oversight.¹¹x Note that the OECD ‘Policy Framework for Effective and Efficient Financial Regulation’ of 2010 does not (explicitly) include a review of existing financial legislation to be a requirement for considering additional steps in financial regulation. See also R.A.J. van Gestel, ‘Wat is regeldruk? Een verkennende internationale literatuurstudie’, WODC 32 (2006) and Office of the Parliamentary Counsel, above n. 5. Such new or existing legislation may well be complementary or in fact serve as lex specialis in order to address specific matters that may not be adequately covered by existing general legislation and vice versa.¹²x L. Poloko, ‘Drafting of Legislation in Compliance with Model Laws’, 15(4) European Journal of Law Reform 415-50, at 421 (2013). Alternatively, new legislation may have a detrimental effect on the existing norms or vice versa. Such impact may extend to the undermining of existing or new legislation as well as the underlying motivation with the implementation of new legislation.¹³x J. Newig, ‘Symbolic Environmental Legislation and Societal Self-deception’, 16(2) Environmental Politics 276-96, at 278 (2007). For instance, if existing legislation applies a rule-based approach and the new legislation intends (or implies) a more principle-based approach, the latter could undermine the goal and purpose of the initial legislation. Such new or existing legislation does not need to have the exact same scope for either to exert some level of effect on any other legislation affected by it. The new or existing legislation may, for instance, have a (slightly) different scope or a more limited or alternatively broader scope of addressees, all of which are aspects that could both make a case for the parallel existence of both sets of legislation as well as support the argument that such additional legislation should in fact not be necessary but call for amendment of existing legislation.
  A complicating factor can be the role of supervisory authorities in the interpretation of the underlying legislative objectives, whereby specifically the principle-based approach may be interpreted in such a way that would effectively qualify as a rule-based approach. This concept is known as ‘regulatory creep’, whereby the regulator tends to fill in expected compliancy by means of, for instance, guidance in such a way that it is effectively an implicitly rule-based approach of an otherwise principle-based set of legislation.¹⁴x J. Black, M. Hopper and C. Band, ‘Making a Success of Principles-based Regulation’, 1(3) Law and Financial Markets Review 191-206, at 197-8 (2007); Black (2010), above n. 4, at 5; J.H. Walsh, ‘Institution-based Financial Regulation: A Third Paradigm’, 49(2) Harvard International Law Journal 381-412, at 390 (2008), who notes, quoting the SEC, “those addressed could choose to ignore its guidance but they do so at their own peril”. This raises the question whether in such cases the supervisory authority not only takes over interpretation from the legislator but additionally takes a role as a de facto legislator.¹⁵x D.C. Langevoort, ‘Cultures of Compliance’, 54 American Criminal Law Review 933-77, at 956-7 (2017). In financial legislation, the concept of regulatory creep is arguably quite prevalent, as evidenced by a multitude of guidance issued by national supervisory authorities, whereby supervisory authorities use guidance not only as mere clarification of legislation but arguably also to push a supervisory agenda (whether or not under influence of public opinion and political pressure),¹⁶xIbid., at 959. becoming what Black refers to as ‘substantive principle-based regulation’.¹⁷x Black (2008), above n. 4, at 435; Black (2010), above n. 4, at 14-18. A consequence can be that the original legislative intention, and along with it the wording and potentially the interpretation of the legislative objective, no longer fully apply as a matter of fact. This ties in with the principle that new legislation needs to not only anticipate future developments in order to avoid over- or under-inclusion but also require a reasonable degree of consensus among parties on interpretation for the intended legislative objective.¹⁸x Cochrane, above n. 10, at 577. Over time, this consensus may become increasingly challenged, since regardless of the legislative approach, legislation in its nature is generally static, meaning that legislation may not always be able to either immediately or even at all adequately compensate or anticipate future variables,¹⁹x Black et al. (2007), above n. 14, at 193-4. including those resulting from changing (social) norms and the corresponding expectations which, over time, it may prove unable to meet.²⁰x J.R. Sutton, Law/Origins, Origins, Interactions, and Change (2001), at 155.

• 3 Who Are the Intended Addressees?

  To whom does the legislative objective apply and what is the scope of addressees (i.e. is a specific group addressed or is the legislation addressed at a general group of addressees) and what is the motivation for this choice?

  The second step in an effectiveness review is to determine who should comply with the required norm.²¹x Coglianese, above n. 1, at 10. For this part, there are effectively two sources. First there is the simple establishment of who should comply according to the relevant legislation in effect. The scoping of the intended addressees should correspond with the legislative objective as this may be an important aspect in whether the legislative objective can actually be achieved. If the scope is too narrow, addressees that should be included may be omitted, which may cause legislation to be only partly effective, i.e. with regard to those addressees in scope. It would, however, also leave a number of addressees out of scope, which may cause the legislative objective to be only partly achieved. If the scope is too wide, it may affect the supervisory authority’s ability to efficiently supervise and/or enforce the required legislation, potentially creating room for addressees to be non-compliant without effective detection, consequently having a potential detrimental effect on the legislative effectiveness.²²x J. Steffek, ‘The Legitimation of International Governance: A Discourse Approach’, 9(2) European Journal of International Relations 249-75, at 254 (2003).
  In addition, there is the question of who should be the intended addressees required to comply with the legislation according to its legislative history. The legislative history will generally give (some) insight into the considerations that led to the eventual choices for the addressees to whom the legislation is addressed. Such legislative history should show the causality between the objective that is to be achieved and the addressees that should reasonably comply with the said norm. In this context, it can be argued that the selection of the addressees should be proportionate in relation to the objective. A too wide scope of addressees may be detrimental to the effective application of legislation, e.g. in the sense that the legislative objective may be addressed partly at parties to whom the objective is not intended, potentially causing ineffective application of the supervisory agenda. Effectiveness may be compromised if legislation affects not only addressees that should be affected, it also affects those that may not necessarily be obvious or logical addressees and could potentially not fully or even at all comply with such legislation. The question then becomes whether legislation may still be considered effective if it reaches its goal with respect to those that it should reasonably apply to but also applies to those it reasonably should not apply to (regardless of actual enforcement). Alternatively, a too narrow scope may cause addressees that should have been taken into account to remain unaffected, thus limiting the intended effect of legislation.²³x Inadequate scoping may have a greater effect than just affecting effectiveness; it may cause legitimacy issues as a result, see e.g. T.R. Tyler, ‘Psychological Perspectives on Legitimacy and Legitimation’, 57 Annual Review of Psychology 375-400, at 382 (2006) and www.kcwj.nl/sites/default/files/attachments/handvatten-voor-evaluaties-van-effectiviteit.pdf, at 9. Specifically, in those cases where effectiveness of legislation depends on enforcement (e.g. when an instrumentalist approach is leading), the lack of regulatory resources may detrimentally affect the effectiveness of the underlying legislation. By focussing on a too small group of addressees, the regulatory interpretation and enforcement might not be of particular concern; however, it could then be asserted that by not including and addressing all of the relevant addressees, the full goal of the underlying legislation would consequently not be met. In either case, it should be noted that in the long run, the damage may well exceed the question of legislative effectiveness, as such approaches can also (eventually) cast doubt whether the perceived legitimacy of the legislator could be challenged, as over-inclusion or, alternatively, exclusion of those addressees that should reasonably be excluded or included may cast doubt on the distributive fairness of such legislation.²⁴x J. Sunshine and T.R. Tyler, ‘The Role of Procedural Justice and Legitimacy in Shaping Public Support for Policing’, 37(3) Law & Society Review 513-47, at 514 (2003). This too may detrimentally affect legislative effectiveness, albeit not immediately, as a legislator who has its legitimacy called into question may potentially find it increasingly difficult to garner support and compliance with intended legislation.

• 4 What Is the Legislative Approach?

  Does the legislative approach match the combination of the legislative objective and its addressees? Is there an expectation of excessive or, alternatively, too little ambiguity in the wording and is it reasonably expected to affect the legislative objective and its addressees?

  The third step in an effectiveness review is to determine which legislative approach has been applied. In financial legislation, there are generally (a combination of) two approaches to be found, the rule-based approach and the principle-based approach. For the regulation of the conduct side of financial legislation, a principle-based approach is generally used, whereas prudential law usually leans more towards a rule-based approach.²⁵x Ford, above n. 4, at 265. The appropriate legislative approach provides a foundation for, and expectation of, the way in which the expected conduct of the addressees is to take place and thus which approach is most appropriate, which consequently plays an important part in the actual success – or effectiveness – of the underlying legislation.²⁶x Gestel, above n. 11, at 106-7; Walsh, above n. 14, at 381-4. On the one hand, the chosen legislative approach can be considered to be a reflection of the level of trust the legislator has in the level of compliancy and interpretation to be expected from the addressees in relation to the legislative objective, and thus logically the level of enforcement it should expect to be necessary. If, for example, the legislator considers that the addressees cannot be trusted to voluntarily submit to the relevant legislative requirements, it should generally not want to use a principle-based approach that would allow a large measure of freedom of interpretation and implementation. Rather it would employ a rule-based approach with a strong basis for the supervisory authority to enforce the rule that is to be followed.²⁷x Walsh, above n. 14, at 381-4. Consequently, the expectation should be that when the legislator chooses a rule-based approach, as a basic principle, both the supervisory authority and the addressees need to be able to rely on unambiguously worded rules as well as on clearly defined tasks, authorities and penalties.²⁸x Ford, above n. 4, at 263-4. The absence of such clarity in rule-based legislation may have a detrimental effect on the effectiveness of the relevant legislation.²⁹x S. Kirchner, ‘Effective Law-making in Times of Global Crisis – A Role for International Organizations’, 2(1) Goettingen Journal of International Law 267-92, at 275 (2010); Poloko, above n. 12, at 419-21; Sutton, above n. 20, at 155. Principle-based legislation, on the other hand, generally seeks a more holistic approach towards the intended objective and how it may be achieved. As Black notes, principle-based regulation focusses on the aim, or outcome, that is intended to be achieved, the ‘substantive objective’, rather than on mere rule compliancy.³⁰x Black et al. (2007), above n. 14, at 194; Walsh, above n. 14, at 381-4. Working with a principle-based approach implies that you trust the addressee to provide compliant output;³¹x Black et al. (2007), above n. 14, at 195; Black (2010), above n. 4, at 3. as such, it works towards the intention of the underlying legislation rather than punishing a rule-breaker.³²x K. Alexander, ‘Principles v. Rules in Financial Regulation: Re-assessing the Balance in the Credit Crisis’, 10 European Business Organization Law Review 169-73, at 196-8 (2009). Yet, whilst the intention of the legislator is to be considered the guiding principle, to ensure uniform compliancy the existence of consensus among the parties as to the interpretation of the relevant legislation is a conditio sine qua non; otherwise, a principle-based approach may result in addressees taking an overly cautious approach for fear of supervisory reprimand.³³xIbid., see also Schwarcz, above n. 4, at 178-80. The addressees should be able to trust the legislative and supervisory interpretation of the underlying principles to correspond with theirs and vice versa.³⁴x Black et al. (2007), above n. 14, at 196-8; D. Demortain, ‘Enabling Global Principle-based Regulation: The Case of Risk Analysis in the Codex Alimentarius’, 6 Regulation & Governance 207-24, at 209 (2012).
  Closely tied to the above is the influence that manner of drafting of legislation may have on interpretation of legislation,³⁵x Kirchner, above n. 29, at 275; Poloko, above n. 12, at 419-21; Sutton, above n. 20, at 155. whereby the way addressees may have implemented legislation based on their interpretation of the underlying legislation may prove contrary to interpretation of the same legislation by the relevant supervisory authority (which may potentially be a cause for regulatory creep), ultimately causing uncertainty amongst addressees on both compliance and interpretation as well as on potential penalties by the supervisory authorities for perceived non-compliance.³⁶x Such lack of clarity could also affect the level of legitimacy assigned to both the supervisory and legislative authority, creating a potential detrimental effect that could go further than the then current and underlying legislation, see e.g. Abbott and Snidal, above n. 7, at 433; Black et al. (2007), above n. 14, at 196-8; J. Jackson, B. Bradford, M. Hough, A. Myhill, P. Quinton and T.R. Tyler, ‘Why Do People Comply with the Law? Legitimacy and the Influence of Legal Institutions’, 52 British Journal of Criminology 1051-71, at 1063-4 (2012); Maher, above n. 3, at 132; Tyler (2006), above n. 23, at 381-3; T.R. Tyler, Psychology and the Law; The Oxford Handbook of Law and Politics (2008), at 718. An additional complication regarding drafting of legislation may be found in the transposition of supranational regulatory requirements into national legislation. It is argued that there is a risk that such requirements may get (partly) lost in translation or even become subject to alternative interpretation.³⁷x Poloko, above n. 12, at 420. In case of the latter, varying social norms and culture-specific regulatory approaches as well as socioculturally specific norms on the specific regulatory field may, to a more or lesser extent, influence the implementation of supranational legislation into a specific jurisdiction, leading to a potentially different approach than was originally intended.³⁸xIbid., see also F. Cafaggi, ‘New Foundations of Transnational Private Regulation’, 38(1) Journal of Law and Society 20-49, at 26 (2011).

• 5 Case Study

  The concept of product approval is not new to financial institutions. It is considered part of good governance to ensure adequate product approval procedures. Note in this context the recommendations of the Basel Committee on the subject of risk and product governance (BCBS 1998; BCBS 1999; BCBS 2006).³⁹x Respectively, BCBS (Basel Committee on Banking Supervision), Framework for Internal Control Systems in Banking Organisations (BCBS40) (1998); BCBS (Basel Committee on Banking Supervision), Enhancing Corporate Governance for Banking Organisations (BCBS56) (1999); BCBS (Basel Committee on Banking Supervision), Enhancing Corporate Governance for Banking Organisations (BCBS122) (2006); BCBS (Basel Committee on Banking Supervision), Principles for Enhancing Corporate Governance (BCBS176) (2010). The focus of these recommendations was mostly on the risk control and governance of the financial institution, so as to mitigate the financial risk associated with both new and existing financial products for the financial institutions. Over time, these recommendations became more detailed and included inter alia the recommendation to include scenario analysis prior to product launch; however, the focus of these recommendations remained largely aimed at the internal risk governance of the financial institution (BCBS 2010).⁴⁰xSee also EBA Guidelines on Internal Governance (EBA BS 2011/116, GL-44). The consumer-oriented ‘protective’ product approval and review policy (PARP) governance came later and mostly as a result of the financial crisis of 2008 as part of a regulatory and legislative push for containing the (systemic) risks associated with the financial services industry (BCBS 2015).⁴¹x BCBS (Basel Committee on Banking Supervision), Corporate Governance Principles for Banks (d328) (2015). And whilst many of the principles governance-wise are similar (such as the need for scenario analysis), the focus is different, aimed at protecting the consumer from mis-selling of financial products by financial institutions. PARP may be considered to be an instrumental part of governance for financial products and, as such, one might expect a robust legislative approach towards PARP, both in embedding legal requirements and how much financial institutions are allowed to use their own interpretation as to the implementation of PARP. In addition, PARP is a perfect example of the way in which the layered nature of financial legislation in practice is applied and the extent to which the objectives of the main financial legislation is reflected in secondary regulation. Thus, the choice for PARP to test the questions under Sections 2 to 4 due to its inherent importance seems a logical one.
  The choice for a review into UK and Netherlands PARP regulation – respectively, PROD and Bgfo Wft⁴²x Product Intervention and Product Governance Sourcebook (PROD), release 38 (April 2019), www.handbook.fca.org.uk/handbook/PROD.pdf (last visited 10 June 2019); Besluit Gedragstoezicht financiële ondernemingen Wft (Decree on Conduct of Business Supervision of Financial Undertakings under the Act on financial supervision) (Bgfo Wft), https://wetten.overheid.nl/BWBR0020421/2019-04-01 (last visited 10 June 2019). – is motivated primarily by the availability of English or Dutch documentation (and these languages being part of the skillset of the author), whereas the choice for EBA (EBA POG Guidelines)⁴³x EBA Guidelines on Product Oversight and Governance Arrangements for Retail Banking Products (2016), EBA/GL/2015/18. is motivated by their role in ensuring a somewhat level playing field in respect to, e.g. PARP as applicable to banks in the European Union, i.e. ensuring national supervisory authorities all work from more or less the same principles on PARP as well as their role of providing further clarification on certain (PARP) requirements that follow from EC Directives (as further clarified below).

  1. Determine the legislative objective and what is the actual source of the legislative objective? Is this legislative objective also addressed in other legislation and how – if at all – is either affected by the other (e.g. which objective is the prevailing objective)?

  The implementation of consumer-oriented PARP regulation in the Netherlands and the United Kingdom as well as the EBA Guidelines on this subject matter can be traced back to the assertion that the financial crisis was, at least in part, the result of mis-selling financial products to consumers.⁴⁴x This position was set out in the Netherlands in the Amendment Decree Financial Markets (Wijzigingsbesluit financiële markten) 2013, Stb 2012-695, pp. 70-1, implementing the PARP regulation in the Netherlands whereas in the United Kingdom this may be derived from the findings of the Parliamentary Commission on Banking Standards (2013), p. 14, www.bankingstandardsboard.org.uk/banking-standards-report.pdf. This position may also be found in the guiding explanation to the EBA POG-Guidelines, p. 7. The consideration at the basis of PARP was that the strengthening and assurance of sufficient governance on the development of new financial products should function as a safeguard against large-scale failure of financial products and the consequent mass damages to consumers.⁴⁵x Whilst this intention does follow directly from section 1.1.1 of PROD, for the Netherlands you need to review the legislative history of Bgfo Wft (Parliamentary Documents, House of Representatives (Kamerstukken, Tweede Kamer), 2010-2011, 31980-49) and the EBA POG Guidelines, p. 16. However, the consideration to avoid mass damages to consumers is not explicitly reflected in the actual legislation or regulation. The relevant articles in the Netherlands, the United Kingdom and set out in the EBA POG Guidelines, respectively, Articles 32 and 32a Bgfo Wft, sections 1.1 and 3 PROD and sections 4 and 5 of the EBA POG Guidelines, are focussed on the internal governance of the financial institutions. Consequently, depending on the source used, a different objective may emerge. Looking at the legislative history of PARP, it can be argued that the objective of PARP is either to avoid mass damages to consumers due to failure of financial products or, to this extent, to ensure adequate internal governance at the financial institution. Alternatively, when looking solely at the relevant legislation, the objective may also be interpreted as to ensure the implementation of the procedural requirements by the financial institutions that follow from legislation, i.e. to implement a PARP procedure as prescribed to the previously noted legislation, regulation or guideline.⁴⁶x Note in this context that the implementation of PARP is a legal requirement in the Netherlands, as follows from Arts. 32 and 32a Bgfo Wft, whereas in the United Kingdom, for banks at least, PARP is set out as a guidance (PROD1); yet for investment firms, it is largely set out as a rule (PROD3), whereby MiFID2 specific rules are supplemented with (explanatory) guidance by the FCA. This is problematic because, depending on which source is used as the basis on which effectiveness of PARP is to be assessed, both scope and outcome of the assessment will likely vary substantially.
  Adding further difficulty, in both the Netherlands and the United Kingdom PARP is considered to be a subset of rules or guidance intended to support the achievement of the larger purpose of (framework) financial legislation. Article 32 Bgfo Wft is a further interpretation of framework legislation, specifically, and respectively, clauses 4:14 sub 1 and 2 and 4:15 sub 2 of the Dutch Act on financial supervision (Wet op het financieel toezicht [Wft]),⁴⁷x Dutch Act on financial supervision (Wft), https://wetten.overheid.nl/BWBR0020368/2019-02-19. involving in part a duty of care for consumers. For PROD, however, it may be considered part of general governance provisions of the Financial Services Act; however, unlike the POG Guidelines and Bgfo Wft, there is not an immediate link to specific legislation, apart from the link to product intervention through clause 2.1.1. of PROD2 and clauses 137D and 138M of the Financial Services Act 2012 and the specific inclusion of MiFID2⁴⁸x EC Directive on markets in financial instruments and amending Directive 2002/92/EC and Directive 2011/61/EU (MiFID2), Dir 2014/65/EU product governance provisions applicable to those financial institutions offering financial instruments to consumers as set out in clause 3.2.3 of PROD3. Note also that the EBA POG Guidelines are considered supplemental to specific EC Directives as well as other EBA Guidelines and as such they explicitly refer to the EBA Guidelines on Internal Governance, which addresses not so much the conduct side of PARP but rather the prudential elements of PARP (e.g. the impact a new product may have on the balance sheet as well as on the internal cost and manpower needed to introduce the product as set out in principle 23 of the Guideline). Both of the aforementioned Guidelines are further guidance of the requirement for ensuring adequate internal governance as set out in Article 22 Capital Requirements Directive (CRD)⁴⁹xSee EBA BS 2011/116, GL-44, p. 9, point 29 jo. CRD Directive (2013/36/EU). and, respectively, Articles 74 sub 1 CRD IV, 10 sub 4 Payment Services Directive (PSD) and 7 sub 1 Mortgage Credit Directive (MCD),⁵⁰xSee EBA/GL/2015/18, p. 11, at 6. as a consequence of which it could be argued that both guidelines are further elaborations of the interpretation of aforementioned articles, but that the main principle (relating to PARP) is in fact considered to be covered elsewhere already. As such, PARP may not be considered to be more than an aid in design, despite some of the legislative history of PARP suggesting a more fundamental approach.

  2. To whom does the legislative objective apply and what is the scope of addressees (i.e. is a specific group addressed or is the legislation addressed at a general group of addressees) and what is the motivation for this choice?

  Article 32 Bgfo Wft applies to effectively all financial institutions that operate under a license in the Netherlands, initially excluding from investment firms, although these are now also subject to PARP through clause 32a Bgfo Wft per the implementation of MiFID2. Interestingly, the MiFID2 requirements relating to PARP are largely similar to the existing requirements, save for specific requirements such as the distinction between manufacturer and distributor, but a separate clause was included nonetheless. In this respect, it is worth noting that the Dutch supervisory authorities noted to EBA that they would comply with the EBA POG Guidelines as per the implementation of MiFID2,⁵¹xSee www.eba.europa.eu/documents/10180/1141044/EBA+GL+2015+18-Compliance+Table-GLs+on+POG.pdf/b675660b-cae9-4921-85a5-02937a3abd46. yet the aforementioned distinction for manufacturers and distributors now only applies to investment firms, implying continued non-compliance with the EBA POG Guidelines. Clause 1.1.1 of PROD1 refers to ‘firms’ as the intended addressees. Firms are defined in the Glossary of the Financial Conduct Authority (FCA) Handbook as, inter alia, authorised persons, which includes all those that undertake regulated activities (as defined in the Prudential Regulation Authority (PRA) Handbook and for the FCA Handbook those activities that are defined in Part II of the Regulated Activities Order, which largely coincide with those activities specified in the PRA Handbook).⁵²xSee the FCA Handbook Glossary on the respective definitions of firm, authorised person and regulated activitieswww.handbook.fca.org.uk/handbook/glossary/ (last visited 10 June 2019). Given the different approaches between PROD1 and PROD3, specifically the exclusive applicability of PROD3 on MiFID2 investment firms, it may well be argued that the FCA too does not fully comply with the POG Guidelines at this time. The EBA POG Guideline itself ultimately aims to provide guidance to specific financial institutions (i.e. banks holding a banking license); it cannot do so directly as the Guideline has no direct applicability towards the intended financial institutions. The Guideline’s primary addressees are the national supervisory authorities, of whom it is expected that they make every effort to comply with the guideline (but note this requirement is also directed at the financial institutions). This presents an interesting conundrum, because only if the national supervisory authority has the ability to draft and subsequently enforce binding rules can this approach imply that the supervisory authorities could be required to adopt an interpretation that complies with the POG Guidelines even though national legislation may (theoretically) not support such an interpretation.

  3. Does the legislative approach match the combination of the legislative objective and its addressees? Is there an expectation of excessive or alternatively too little ambiguity in the wording and is it reasonably expected to affect the legislative objective and its addressees?

  The legislative approach of Article 32 Bgfo Wft is principle-based, whereby the legislator provides a set of minimum requirements and general guidance for the PARP process without being all too prescriptive (regardless of whether it concerns a financial institution or an investment firm).⁵³x Parliamentary Documents, House of Representatives (Kamerstukken, Tweede Kamer), 2010-2011, 31980-47, p. 2. It should be noted that the supervisory authority provides further guidance as to the implementation of some of what it considers to be the most relevant parts of PARP, which, as has been noted before, may well be interpreted more as a rule-based guidance than the advice it may be purported to be (i.e. regulatory creep).⁵⁴xSee e.g. www.afm.nl/nl-nl/professionals/onderwerpen/productontwikkeling. PROD is a combination of principle-based (PROD1), rule-based (PROD2) and a combination of both (PROD3), whereby it is interesting to note that most rules set out in PROD (explicitly designated by the ‘R’) are accompanied by interpretative or explanatory guidance (explicitly designated by the ‘G’). Much like Bgfo Wft, PROD is part of layered legislation, meaning it should be read with other legislation and regulation in hand, both for interpretation and explanation as well as cross reference. Whilst technically not legislation as such, the EBA Guidelines do aim to create some sort of generally applicable interpretation on POG and PARP in order to ensure a level playing field amongst the member states so as to avoid all too great discrepancies in POG and PARP that would detrimentally affect the position of consumers of different member states by requiring national supervisory authorities (and indirectly financial institutions as well) to ensure compliancy with. As such, EBA is fairly prescriptive as to the procedural steps that should be taken, but not so much when it comes to the actual content of such procedural steps, as may be derived from pages 16 and further of the POG Guideline. The use of wording such as ‘should establish’, ‘should ensure’ and ‘should implement’ indicates that the addressed parties are not entirely free in the organisation of the POG procedures.⁵⁵x ‘Should’ is not the same as ‘must’; ‘should’ is a representation of what is considered to be the right thing to do but is not exactly compulsory. For non-native speakers, however, such a distinction may be difficult to interpret. See also https://keydifferences.com/difference-between-should-ought-to-and-must.html (last visited 10 June 2019); see also Xanthaki, above n. 2, at 433. Whilst the POG Guidelines do not outline detailed arrangements of the procedural steps required for PARP, certain minimum requirements are nevertheless stipulated. As such, it may be argued that the POG Guidelines consist of a combination of a more or less rule-based approach and a principle-based approach. The rule-based aspects relate to the required procedural aspects of the Guideline, whereas the actual content of each such procedural step is fairly open formulated and consequently would allow for a more principle-based interpretation. This approach is likely the consequence of the legislative status of the POG Guidelines, whereby it is firstly a guideline for the national supervisory authorities, of whom it may be required to ensure a level playing field by means of prescribing the procedural steps of POG that should apply throughout the European Union (EU), but leaving the actual content thereof up to the national supervisory authorities to align with the nationally chosen legislative approach.

  4. Assessment of the PARP case study

  The above review shows that the objective of PARP regulation may be different depending on the source used. PARP regulation can be seen to govern the implementation of a PARP process, ensure that mis-selling of financial products is to be avoided or to prevent (grave) consumer detriment due to mis-selling. Each of these separate objectives demands a different approach of assessment, and arguably as the objectives become less specific or more general in nature, may demand more and different resources than would be the case of a mere assessment whether, for instance, PARP procedures have been implemented by financial institutions. Adding further complexity, PARP can be seen as both a substantive legal obligation as well as an ancillary to existing obligations that arise from existing legal frameworks. In case of the latter, it can be argued that the assessed effectiveness of PARP holds but limited value due to its nature of being ancillary to a larger objective, or at least that in effectiveness review in such a case would need to be reviewed in conjunction or as part of an effectiveness review of the larger objective. The difficulties in establishing which is the objective to be assessed also impact the questions that follow, i.e. who are the intended addressees, and is the chosen legislative approach logical in the context of the previous two questions. If the objective of PARP is to be seen primarily as a singular substantive legal obligation, the way this is implemented into legislation as well as in practice simply needs to correspond and be conducive to the said obligation, whereas if the objective of PARP has to be regarded as ancillary to existing legal obligations, arguably the implementation of PARP will need to correspond with such existing legal obligations.
  In practice, both Article 32 Bgfo Wft and PROD1 have a fairly wide scope of intended addressees, applying to most financial institutions. However, Article 32a Bgfo Wft and PROD3 list specific PARP requirements for investment firms only, whilst, interestingly, Articles 32 and 32a Bgfo Wft are substantively similar. Both Articles 32 and 32a Bgfo Wft apply a principle-based approach which may be logical if considering PARP to be ancillary in nature, but, on the other hand, may be difficult to reconcile with an objective to avoid grave consumer detriment. PROD3 (applying to investment firms), on the other hand, is substantially more descriptive, leaning towards rule-based than towards PROD1 (having general application). And whilst this can be explained as being the result of the implementation of MiFID2, it is difficult to see how these two approaches to the same objective could reasonably lead to the same result. The POG Guidelines as a logical limitation only require PARP for banks, although within the European Supervisory Authorities’ framework this requirement is extended to both investment firms and insurance firms, although not explicitly to other financial institutions.⁵⁶x As set out, e.g. in the ‘Joint Position of the European Supervisory Authorities on Manufacturers’ Product Oversight & Governance Processes’ of the Joint Committee of the European Supervisory Authorities (ESA) (JC-2013-77). The POG Guidelines are fairly prescriptive in the procedural steps EBA considers are required for PARP. However, it has little way of enforcing such steps nor is it in a position to demand compliance. Essentially, it could be argued that the limited enforcement powers EBA has in respect of PARP enforcement are not reflected in the substantive content of the POG Guidelines.
  This simple review highlights some of the more fundamental issues with effectiveness review of financial legislation. There may be debate as to the intended objective, and those objectives are either so broad that they cannot reasonably be measured or even attributed solely to this specific legislation or regulation. The lack of clarity on the underlying objective resonates in the criteria on the addressees and the legislative approach. The overall conclusion would arguably be that PARP as is has the potential – and likely does – to have added value to the overarching principle of avoiding consumer detriment as a result of mis-selling, but that as it stands there are a number of substantive variables identified even in this brief review that would likely make the case that the main objective of PARP is unlikely to be achieved solely by PARP itself.

• 6 The Sense or Nonsense of Measuring Effectiveness

  The issues noted above are illustrative of the difficulties in assessing the effectiveness of (parts of) increasingly complex and elaborate financial legislation. The basic criteria for an effectiveness review as set out above could theoretically serve as a basis for an effectiveness assessment of legislation. However, as financial legislation becomes more complex, the fundamental question becomes whether it is actually practically possible for such an assessment to take place, and, if so, whether being able to answer these effectiveness criteria actually allows the researcher to determine whether the legislation under review is in fact effective? I argue below that a review of effectiveness of (parts of) financial legislation and regulation are not likely to provide any conclusive answers. This is because the level in which (subsets of) financial regulation is intertwined with not only other (subsets of) both financial legislation and other legislation (such as criminal law and private law); it is in fact subject to so many different variables that both individually as well as collectively influence the working and outcome of financial legislation to such an extent it is in practice incredibly difficult and requires tremendous resources for those to be isolated that an adequate effectiveness assessment would, however banal, become labour-intensive and costly to actually fully execute.⁵⁷x Coglianese, above n. 1, at 21-3; T.W. Merrill, ‘A Comment on Metzger and Zaring: The Quicksilver Problem’, 78 Law and Contemporary Problems 189-204 (2015).
  There are a number of arguments that support the above approach. To begin with, is the concept of identifying and isolating all relevant variables associated with the legislation to be reviewed? Such variables are not limited to whether the intended purpose of legislation is clear or also covered in legislation elsewhere. Such variables may concern socio-economical elements, political motivations and the position of the supervisory authority,⁵⁸x R. Pawson, Evidence-based Policy, A Realist Perspective (2006); Xanthaki, above n. 2, at 433. to name a few, and the fact that each of the aforementioned are prone to changing interpretation and perception over time⁵⁹x Note in this context the observance by E.A. Posner and E.G. Weyl, ‘Benefit-Cost Paradigms in Financial Regulation’, Coase-Sandor Institute for Law and Economics Working Paper 2014:660, http://ssrn.com/abstract=2346466 (last visited 10 June 2019), at 10-8, addressing the concept of cost-benefit analysis before legislative implementation, noting that the fact that there is a potential multitude of variables that could be considered in a cost-benefit analysis and as such on the whole relevant for the specific legislation, but that this might in fact not have actual added value for cost-benefit analysis itself, raising the question as to what would cause a relevant variable to be excluded nonetheless. See also Cochrane, above n. 10, at 568. or the fact that new or other related or unrelated legislation may have unexpected effects on the legislation under review.⁶⁰x J.N. Gordon, ‘The Empty Call for Benefit-Cost Analysis in Financial Regulation’, Colombia Law School and ECGI, Law Working Paper 2014:233, at 1-19. To use an example, the position and perception of banks in society have changed dramatically over the past decade since the 2008 financial crisis manifested itself; as has the correlating perception of legislative effectiveness. What was once considered effective has either proven to be regarded ineffective or at least failed to produce the required result. Often, this has resulted in a (political) push for yet more legislation,⁶¹x Cochrane, above n. 10, at 575. including a push to go from a principle-based approach to a rule-based approach,⁶²x Alexander, above n. 32, at 169-73. rather than investigating why legislation failed to provide the intended results (or maybe it did, but these results no longer were deemed to be the desired results).⁶³x Cochrane, above n. 10, at 567-9.
  There is also the question of how to perform an effectiveness review in practice. As a minimum, a baseline would need to be established prior to implementation of the relevant legislation as this would provide reference from which follow-up reviews or measurements can take place. This would be an interesting proposition as it would at least provide some point of reference. It can be argued that in order to establish the said baseline, the relevant variables that are expected to be of influence on the outcome, whether they be, e.g. legislative, sociological or economical in nature should be charted and would need to be incorporated in the effectiveness review. The reasoning is that this should show if any potential change can be attributed wholly or partly (or at all) to the new or changed legislation and what if any the influence of the identified variables is. In addition, upon an effectiveness review, it needs to be established if new variables that may also potentially affect the effectiveness of the legislation under review have been introduced.⁶⁴x Some of these variables could be qualified as ‘confounders’, which may be held constant in (certain) statistical models, thus reducing some of the resources that would otherwise be required for measuring. Note though that this obviously only works after these variables have in fact been identified (see Coglianese, above n. 1). But even if such a review would only be applied marginally (without charting the relevant variables), the mere fact that a baseline has been established would be very valuable if not imperative, as this would provide at least some basic reference on which a review can be conducted.⁶⁵x The Dutch Ministry of Finance, for instance, recently noted that the general absence of baseline measurements is one of the important causes why it is so very difficult, if not impossible, for effectiveness measurements of financial legislation to take place; see in this regard: Parliamentary Documents, House of Representatives, 2017-2018, 31935, nr. 45. Interestingly, the OECD even argues in the OECD “Policy Framework for Effective and Efficient Financial Regulation” of 2010 that a (full) cost-benefit analysis of financial regulation may not be necessary or appropriate depending on the impact of such legislation and should always be ‘proportionate’ to the regulation under review (see, e.g. p. 31). Alternatively, some argue that an effectiveness review should be performed in some sort of controlled environment,⁶⁶x Pawson, above n. 58, at 51. in which external factors could be controlled or even excluded. In practice, such a laboratory-like assessment of legislative effectiveness is unlikely to be executed or even possible,⁶⁷x Coglianese, above n. 1, at 38. since it would imply that subjects are placed in a contained environment that would allow for the exclusion of external influences and, arguably, also a set up in which one group of subjects would be allowed to not be subject to the specific legislation.⁶⁸x It is argued by Coglianese (above n. 1) that such a laboratory-like set-up may be mimicked by reviewing against a group of subjects placed in a jurisdiction similar to that of the target subjects but that do not have the (intended) legislation. Again, this would arguably only work if all other variables can be identified and contained. In practice, it seems very unlikely that such a situation would occur. Aside from the practical challenges, such as how to determine which external variables to exclude (e.g. other legislation?), there are ethical considerations which would likely prohibit any such experiment. In addition, considering that no external variables can in fact be controlled in a real-life situation, it would be virtually impossible to fully attribute measured outcomes solely to the legislation under review, and thus such variables should reasonably be identified as possible influencers beyond control.⁶⁹x Haarhuis and Niemeijer, above n. 2, at 13.
  In light of the above, let us review the first question of effectiveness review set out earlier in this article against the backdrop of the above, i.e.: what is the legislative objective? As noted in the example of PARP used, the mere question of what would be the legislative objective may in itself prove a difficult question to answer. For PARP, it may simply be the implementation of a PARP and its corresponding conditions. However, it may also be the more holistic objective of avoiding consumer detriment. Clearly, the objective of avoiding consumer detriment is not one that is limited to PARP; in fact, it may even be argued that most of the newly implemented financial legislation since the 2008 financial crisis has been, to a more or lesser extent, with a view of avoiding (large-scale) consumer detriment. So what if any influence may be attributed to PARP? And if, in certain cases, large-scale consumer detriment is not avoided, to what extent would that be attributable to PARP? How does other legislation contribute to the objective of avoiding consumer detriment and to what extent does this affect the purported effectiveness of PARP? The relative nature of these variables becomes even more apparent when looking at methods of measuring effectiveness of legislation. Take, for instance, the input-throughput-output model (typically used to assess the effectiveness of the supervisory authority).⁷⁰x Coglianese, above n. 1, at 8; B.P.A. van Mil, A.E. Dijkzeul and M. Noordink, ‘Beoordeling effectiviteit “Nieuwe Stijl”; Een praktijkperspectief op het beoordelen van de effectiviteit van een toezichthouder’, 3(2) Tijdschrift voor Toezicht 81-90, at 82 (2012). When applied to legislation, one of the fundamental problems with this model, as is noted by Haarhuis and Niemeijer, is that the outcome not only heavily depends on what input is but also tends to focus on but one aspect of all the factors that make up legislative effectiveness, i.e. the legislative objective and whether this has been achieved.⁷¹x Haarhuis and Niemeijer, above n. 2, at 10-11. This model may easily allow for manipulation of the (anticipated) output, specifically by allowing for limited input (such as merely asking the question: Did all relevant institutions implement PARP?), or may seek focus mostly on objectively verifiable indicators to the detriment of others indicators.⁷²x P.L.C. Hilbers, D.R. Rijsbergen, K. Raaijmakers and F. de Vries, ‘Effectmeting in het toezicht op de financiële sector’, 3(2) Tijdschrift voor Toezicht 38-49, at 42-7 (2012). The consequent output may become subject to both discussion and interpretation as a result, whereby it is noted that output that corresponds with existing practice is more likely to be accepted, further emphasising the risk of input manipulation.⁷³x C. Sorenson, M.K. Gusmano and A. Oliver, ‘The Politics of Comparative Effectiveness Research: Lessons from Recent History’, 39(1) Journal of Health Politics, Policy and Law 139-70, at 162 (2014). For a (strictly theoretical) example in this context, if effectiveness research were to show that PARP regulation has no actual influence over product governance or may actually even affect product governance detrimentally, such an outcome, according to Sorenson et al., would likely yield opposition to such outcome, as it is not the anticipated, desired or expected outcome and would undermine existing and accepted practice. It is also to be noted that those reviewing legislation are often performing such reviews for and on behalf of the legislator. As such, use of this model may not give a full account of the actual effectiveness of legislation due to lack of a holistic perspective.⁷⁴xIbid.; Mil et al., above n. 70, at 83. A further complication in the process of identifying all relevant variables is the information asymmetry between the legislator and regulator on the one hand and the addressees on the other hand. Such asymmetries comprise, for instance, information deficiencies (the legislator or regulator does not have all relevant information or does not understand it) or the legislative process is unevenly balanced by parties pushing for specific interests to be included in legislation. To this extent, Goldbach identifies several actors and motives that may be of influence on the global regulatory process, including but not limited to political and economic motives (both on a micro as well as macro level).⁷⁵x R. Goldbach, ‘Asymmetric Influence in Global Banking Regulation’, 22(6) Review of International Political Economy 1087-1127 (2015).
  When assessing the second question of effectiveness review set out earlier in this article against the backdrop of the above, i.e. the consequent selection of addressees, these may in fact not always be readily agreed upon including by the relevant addressees themselves. Arguably, this may not directly influence the effectiveness of legislation; however, as some argue, a lack of perceived legitimacy may be a cause of but partial or sometimes even full denial of regulatory obligations.⁷⁶x Jackson et al., above n. 36, at 1059; Casey and Scott, above n. 6, at 86-91. This may introduce a whole new variety of variables that may undermine any perceived effectiveness to a level that may not always be readily evident.⁷⁷x Pawson, above n. 58, at 26-34. Such effects may be further aggravated if the legislative approach allows for interpretative differences and regulatory creep (although the latter could be considered to be very particular problem in its own right). Let alone interferences that may occur from other legislation and regulation, changing societal and political norms and acceptance as well as (macro) economic interferences that are all, to some extent or the other, of influence over the performance of a specific piece of legislation. Addressees, on the other hand, are entities that are (usually) governed by a multitude of internal and external considerations, merely one of those being legislative and regulatory compliance.⁷⁸x Cochrane, above n. 10, at 567-8. Compliance with legislation may not be done because it is believed to be just or right, but sometimes merely because it is a must and consequently does not always lead to the expected results that are sometimes more holistic in nature (such as avoiding large-scale consumer detriment) but consist merely of the most literal interpretations of what are considered to be the regulatory requirements (the tick-the-box exercises), leading to maybe formal rule compliancy but not necessarily to the intended effects and thus lacking the required effectiveness.⁷⁹xIbid., see also Casey and Scott, above n. 6, at 79; Jackson et al., above n. 36, at 1063-4; Langevoort, above n. 15, at 950; Meares, above n. 9, at 398; Tyler (2006), above n. 23, at 381-3; Tyler (2008), above n. 36, at 718.
  As for the third question, on the intended addressees, the legislative approach should be the logical sum of the legislative objective and the indented addressees and, as such, legislative effectiveness (if any) will also rely on the matter of choosing the right approach. This implies that the need for legislation has already been objectively determined. In situations where such objective assessment may be less obvious, where legislation may be used, e.g. for political purposes more so than to achieve an objectively determined goal, it may well be argued that the choice of a legislative approach is more or less irrelevant.⁸⁰x Newig, above n. 13, at 277; Poloko, above n. 12, at 426-7. The multitude of variables surrounding the legislative approach may consequently cause the legislative approach to be of limited effect on the outcome. And as noted in Section 4 above, the legislative approach can in itself be nullified by those externalities that would cause a different approach than intended initially.
  I would argue that as a logical consequence of the above evaluation models that are limited in scope to assess effectiveness, i.e. those that (allow) focus on but limited aspects of the effectiveness question, yield results that are of limited value if any. In order to produce any type of reliable and valuable data on effectiveness, a holistic approach in measurement and assessment should be considered to be a requirement. One such model is the realist evaluation model, as proposed by Pawson, which serves as a holistic model intended to encompass the various variables that influence legislation and as such would seek to identify the mechanisms that are part of the core of legislation form the ‘policy’ theory that underpins legislation and could potentially assess the impact and influence of some of the identified variables noted above (the known unknowns).⁸¹x Pawson, above n. 58. This model seeks to incorporate as much research on the subject matter as has already been done in addition to identifying as much of the relevant variables as possible, all after having identified the appropriate research question.⁸²x Note that this is but a highly abstracted description of Pawson’s realist synthesis research model. Holistic models such as these, however, require that either extensive review of the relevant legislation and surrounding variables has been performed or will be performed as well as a basic assumption that at least a baseline has been established. Absence of this data obviously directly affects the essence and outcome of such a holistic review and will arguably detrimentally affect the reliability of the data produced. And even though holistic models such as the realist evaluation model would typically use much more and a wider variety of data to assess effectiveness, these too leave unanswered the extent to which the unidentified variables influence the actual effectiveness of legislation.⁸³x Cochrane, above n. 10; Gordon, above n. 60. Finally, a holistic review requires substantial resources in order to collect data and perform reviews, whereby arguably these resources only increase if the legislation under review shows more interdependencies with other legislation and variables. In practice, it will likely be the cost of lack of resources that will determine whether the holistic review is even an option to consider.

  5. Measuring effectiveness in practice: Evaluation of article 32 Bgfo Wft

  On 20 June 2020 an evaluation of Article 32 Bgfo Wft, ordered by the Dutch Ministry of Finance, was published.⁸⁴x Parliamentary Documents, House of Representatives (Kamerstukken, Tweede Kamer), 2019-2020, 32545-120, see also B.P.A. van Mil, P.C.H. van Dijk, B. Knoote, T.B.F. Beverdam and D. Busch, ‘Evaluatie wettelijke regeling productontwikkelingsproces; artikel 32 Bgfo Wft’, KWINK groep 2020, at 5. The evaluation, in short, consisted of an assessment of the workings in practice of Article 32 Bgfo Wft in the market – i.e. did the implementation of Article 32 Bgfo Wft result in changes in processes of institutions and did the offering of products change – and the extent to which it could be concluded whether Article 32 Bgfo Wft could in fact be classified as effective.⁸⁵x B.P.A. van Mil, P.C.H. van Dijk, B. Knoote, T.B.F. Beverdam and D. Busch, ‘Evaluatie wettelijke regeling productontwikkelingsproces; artikel 32 Bgfo Wft’, KWINK groep 2020, at 4. Much in line with what was noted in Section 5 above, the evaluation noted that to actually be able to assess effectiveness in fact proved difficult, as the level to which the legislative objective of Article 32 Bgfo Wft were also addressed by other (legislative) requirements or otherwise were not easily isolated and thus specifically attributable to specific legislation.⁸⁶xIbid., at 17, 20-21, 24-25 and 34. In addition, it was noted that it could not be established how many institutions, nor to what extent, complied with the regulatory requirements of Article 32 Bgfo Wft, or had in fact implemented it.⁸⁷xIbid., at 13, 19 and 27. In this context, it was noted that neither at the implementation of Article 32 Bgfo Wft nor any time thereafter, was a baseline established. Note that the baseline in this context was connected to the level of compliancy of financial institutions, rather than (holistically) assessing the situation vis-à-vis introducing (inherently) unsuitable products prior to implementation. The evaluation asserts that Article 32 Bgfo Wft may reduce the likelihood of introducing inherently unsuitable products, but statistical evidence cannot be produced and – in line with the above-mentioned caveats – the evaluation notes that the purported results may in fact not be (directly and certainly not solely) attributable to Article 32 Bgfo Wft.⁸⁸xIbid., at 25-27. In fact, it is noted that to date financial products are still being introduced, which may in fact be at odds with the guiding principles of Article 32 Bgfo Wft.⁸⁹xIbid., at 27-28. The evaluation finally classifies certain aspects of Article 32 Bgfo Wft as effective – specifically in the assumption that the chances of inherently unsuitable financial products being introduced is likely reduced – but on the whole the evaluation does not purport to assume that Article 32 Bgfo Wft can be qualified as (fully) effective in the sense that the legislative objective has been, or may be, achieved simply because the results could not be objectively linked (solely) to Article 32 Bgfo Wft.⁹⁰xIbid., at 34-35. The evaluation of Article 32 Bgfo Wft as such underscores the observations set out in sections 5.4 and 6 as set out above and serves as an illustration of the complexities and challenges of assessing effectiveness in practice.

• 7 Conclusion

  The theoretical principles that govern the effectiveness of financial legislation and regulation are in themselves fairly straightforward. The basic questions are (1) what is the legislative objective and is this by chance already addressed in other legislation; (2) to whom does the legislation apply and (3) what legislative approach has been chosen. The combination of answers to these three questions – stripped of any further variables – are, in principle, good indicators for the expected effectiveness of legislation. That is, provided that the answer to (1) results in an unambiguous answer. Given the many sources that may be available to determine such an objective, it may in practice even be difficult to find consensus on the subject of the relevant legislative objective. The PARP case study, for instance, shows that in order to determine the intended objective for PARP, the legislative history versus the formal texts may be cause for significant interpretative issues in determining the actual or intended objective, and consequently may be cause for debate as to what exactly should be the basis for effectiveness review. This uncertainty subsequently affects the consequent questions (2) and (3), i.e. who are the intended addressees and what legislative approach is appropriate in connection herewith, as both are closely intertwined with the first question and the results may vary depending on the actual objective.
  Moreover, for effectiveness reviews to be successful, a holistic approach, such as Pawson’s realist evaluation model, seems most appropriate in order for an accurate effectiveness review to take place. However, the amount of variables to be taken into account in such an approach is arguably so substantial that not only is it likely to take up significant resources for them to be determined and assessed, the extent to which such insight into all of the most relevant variables and their individual or collective influence and impact over the legislation under review is difficult and laborious in most cases and impossible in some. The explosive growth and increasing complexity of financial legislation over the past decade may have additionally attributed to the extent in which such insight has become increasingly difficult to determine, even if sufficient resources are in fact available. In addition, it is noted that the general lack of baseline measurements to compare potential results with present substantial issues, as absence of a baseline presents the problem, is there being no basis to compare to.
  Does the above then imply that endeavours to assess legislative effectiveness are pointless or impossible? No, but a word of caution is advised. Essentially, effectiveness reviews that are limited in nature are likely limited in objective value. And the more limited the review is, the more important it is to determine which objective was in fact reviewed and which objectives were left out of scope of the review, so as to avoid representing a review as a full effectiveness review when in fact it may only be a partial effectiveness review. The above should not be read as an argument that effectiveness reviews are useless or not to perform an effectiveness review. I would make the case, however, that the outcome of effectiveness reviews should be placed in the right context, and, as such, depending on the scope of the review and the sources used, the outcome would need to be viewed in the context of the generally limited review that is to date often the case for effectiveness reviews. And in exactly those cases one should be mindful of the potential manipulative aspects that may be introduced into an effectiveness query so as to steer the results a certain way. As soon as more variables are introduced, provided these are properly and adequately incorporated in the review model, the chances of manipulation diminish and more adequate and valuable data will follow. But the bottom line is, always, if a baseline is absent, there is little to compare the results to. And as such the consistent determination and application of a baseline in effectiveness reviews may very well be considered one of the most instrumental pieces of data, which is almost always missing. I would argue that this will leave most, if not all, effectiveness reviews that lack a baseline with a limited value, regardless of how many sources have been used in the review.

Noten

• * The views, thoughts and opinions expressed in the text belong solely to the author, and not necessarily to the author’s employer.

• 1 M. Weber (edited by G. Roth and C. Wittich), Economy and Society: An Outline of Interpretive Sociology (2013), at 34-5; see also C. Coglianese, ‘Measuring Regulatory Performance; Evaluating the Impact of Regulation and Regulatory Policy’, OECD Expert Paper August 2012:1, at 8.

• 2 C.K. Haarhuis and B. Niemeijer, ‘Wetten in werking; Over interventies, werking, effectiviteit en context’, 2 Recht der Werkelijkheid 9-34, at 10-11 (2008); M. Mousmouti, ‘Operationalising Quality of Legislation through the Effectiveness Test’, 6(2) Legisprudence 191-206, at 201 (2012); H. Xanthaki, ‘An Enlightened Approach to Legislative Scrutiny: Focusing on Effectiveness’, 9(3) European Journal of Risk Regulation 431-44, at 432 (2018).

• 3 I. Maher, ‘Competition Law and Transnational Private Regulatory Regimes: Marking the Cartel Boundary’, 38(1) Journal of Law and Society 119-37, at 126 (2011).

• 4 S.L. Schwarcz, ‘The ‘Principles’ Paradox’, 1 European Business Organization Law Review 175-84, at 178-80 (2009); C. Ford, ‘Principles-based Securities Regulation in the Wake of the Global Financial Crisis’, 55(2) McGill Law Journal 258-307, at 263-5 (2010); J. Black, ‘Forms and Paradoxes of Principles-based Regulation’, 3(4) Capital Markets Law Journal 425-57, at 426 (2008); J. Black, ‘The Rise, Fall and Fate of Principle-based Regulation’, LSE Law, Society and Economy Working Papers 2010:17, 1-25, at 3.

• 5 See Office of the Parliamentary Counsel, When Laws Become Too Complex. A Review into the Causes of Complex Legislation (2013).

• 6 D. Casey and C. Scott, ‘The Crystallization of Regulatory Norms’, 38(1) Journal of Law and Society 76-95, at 82 (2011); R.T. Morris, ‘A Typology of Norms’, 21(5) American Sociological Review 610-13, at 611 (1956).

• 7 K.W. Abbott and D. Snidal, ‘Hard and Soft Law in International Governance’, 54(3) International Organization 421-56, at 431 (2000).

• 8 Schwarcz, above n. 4, at 178-80.

• 9 T.L. Meares, ‘Norms, Legitimacy and Law Enforcement’, 79 Oregon Law Review 391-415, at 398-9 (2000).

• 10 J.H. Cochrane, ‘Challenges for Cost-Benefit Analysis of Financial Regulation’, 43(2) The Journal of Legal Studies 563-605, at 577 (2014).

• 11 Note that the OECD ‘Policy Framework for Effective and Efficient Financial Regulation’ of 2010 does not (explicitly) include a review of existing financial legislation to be a requirement for considering additional steps in financial regulation. See also R.A.J. van Gestel, ‘Wat is regeldruk? Een verkennende internationale literatuurstudie’, WODC 32 (2006) and Office of the Parliamentary Counsel, above n. 5.

• 12 L. Poloko, ‘Drafting of Legislation in Compliance with Model Laws’, 15(4) European Journal of Law Reform 415-50, at 421 (2013).

• 13 J. Newig, ‘Symbolic Environmental Legislation and Societal Self-deception’, 16(2) Environmental Politics 276-96, at 278 (2007).

• 14 J. Black, M. Hopper and C. Band, ‘Making a Success of Principles-based Regulation’, 1(3) Law and Financial Markets Review 191-206, at 197-8 (2007); Black (2010), above n. 4, at 5; J.H. Walsh, ‘Institution-based Financial Regulation: A Third Paradigm’, 49(2) Harvard International Law Journal 381-412, at 390 (2008), who notes, quoting the SEC, “those addressed could choose to ignore its guidance but they do so at their own peril”.

• 15 D.C. Langevoort, ‘Cultures of Compliance’, 54 American Criminal Law Review 933-77, at 956-7 (2017).

• 16 Ibid., at 959.

• 17 Black (2008), above n. 4, at 435; Black (2010), above n. 4, at 14-18.

• 18 Cochrane, above n. 10, at 577.

• 19 Black et al. (2007), above n. 14, at 193-4.

• 20 J.R. Sutton, Law/Origins, Origins, Interactions, and Change (2001), at 155.

• 21 Coglianese, above n. 1, at 10.

• 22 J. Steffek, ‘The Legitimation of International Governance: A Discourse Approach’, 9(2) European Journal of International Relations 249-75, at 254 (2003).

• 23 Inadequate scoping may have a greater effect than just affecting effectiveness; it may cause legitimacy issues as a result, see e.g. T.R. Tyler, ‘Psychological Perspectives on Legitimacy and Legitimation’, 57 Annual Review of Psychology 375-400, at 382 (2006) and www.kcwj.nl/sites/default/files/attachments/handvatten-voor-evaluaties-van-effectiviteit.pdf, at 9.

• 24 J. Sunshine and T.R. Tyler, ‘The Role of Procedural Justice and Legitimacy in Shaping Public Support for Policing’, 37(3) Law & Society Review 513-47, at 514 (2003).

• 25 Ford, above n. 4, at 265.

• 26 Gestel, above n. 11, at 106-7; Walsh, above n. 14, at 381-4.

• 27 Walsh, above n. 14, at 381-4.

• 28 Ford, above n. 4, at 263-4.

• 29 S. Kirchner, ‘Effective Law-making in Times of Global Crisis – A Role for International Organizations’, 2(1) Goettingen Journal of International Law 267-92, at 275 (2010); Poloko, above n. 12, at 419-21; Sutton, above n. 20, at 155.

• 30 Black et al. (2007), above n. 14, at 194; Walsh, above n. 14, at 381-4.

• 31 Black et al. (2007), above n. 14, at 195; Black (2010), above n. 4, at 3.

• 32 K. Alexander, ‘Principles v. Rules in Financial Regulation: Re-assessing the Balance in the Credit Crisis’, 10 European Business Organization Law Review 169-73, at 196-8 (2009).

• 33 Ibid., see also Schwarcz, above n. 4, at 178-80.

• 34 Black et al. (2007), above n. 14, at 196-8; D. Demortain, ‘Enabling Global Principle-based Regulation: The Case of Risk Analysis in the Codex Alimentarius’, 6 Regulation & Governance 207-24, at 209 (2012).

• 35 Kirchner, above n. 29, at 275; Poloko, above n. 12, at 419-21; Sutton, above n. 20, at 155.

• 36 Such lack of clarity could also affect the level of legitimacy assigned to both the supervisory and legislative authority, creating a potential detrimental effect that could go further than the then current and underlying legislation, see e.g. Abbott and Snidal, above n. 7, at 433; Black et al. (2007), above n. 14, at 196-8; J. Jackson, B. Bradford, M. Hough, A. Myhill, P. Quinton and T.R. Tyler, ‘Why Do People Comply with the Law? Legitimacy and the Influence of Legal Institutions’, 52 British Journal of Criminology 1051-71, at 1063-4 (2012); Maher, above n. 3, at 132; Tyler (2006), above n. 23, at 381-3; T.R. Tyler, Psychology and the Law; The Oxford Handbook of Law and Politics (2008), at 718.

• 37 Poloko, above n. 12, at 420.

• 38 Ibid., see also F. Cafaggi, ‘New Foundations of Transnational Private Regulation’, 38(1) Journal of Law and Society 20-49, at 26 (2011).

• 39 Respectively, BCBS (Basel Committee on Banking Supervision), Framework for Internal Control Systems in Banking Organisations (BCBS40) (1998); BCBS (Basel Committee on Banking Supervision), Enhancing Corporate Governance for Banking Organisations (BCBS56) (1999); BCBS (Basel Committee on Banking Supervision), Enhancing Corporate Governance for Banking Organisations (BCBS122) (2006); BCBS (Basel Committee on Banking Supervision), Principles for Enhancing Corporate Governance (BCBS176) (2010).

• 40 See also EBA Guidelines on Internal Governance (EBA BS 2011/116, GL-44).

• 41 BCBS (Basel Committee on Banking Supervision), Corporate Governance Principles for Banks (d328) (2015).

• 42 Product Intervention and Product Governance Sourcebook (PROD), release 38 (April 2019), www.handbook.fca.org.uk/handbook/PROD.pdf (last visited 10 June 2019); Besluit Gedragstoezicht financiële ondernemingen Wft (Decree on Conduct of Business Supervision of Financial Undertakings under the Act on financial supervision) (Bgfo Wft), https://wetten.overheid.nl/BWBR0020421/2019-04-01 (last visited 10 June 2019).

• 43 EBA Guidelines on Product Oversight and Governance Arrangements for Retail Banking Products (2016), EBA/GL/2015/18.

• 44 This position was set out in the Netherlands in the Amendment Decree Financial Markets (Wijzigingsbesluit financiële markten) 2013, Stb 2012-695, pp. 70-1, implementing the PARP regulation in the Netherlands whereas in the United Kingdom this may be derived from the findings of the Parliamentary Commission on Banking Standards (2013), p. 14, www.bankingstandardsboard.org.uk/banking-standards-report.pdf. This position may also be found in the guiding explanation to the EBA POG-Guidelines, p. 7.

• 45 Whilst this intention does follow directly from section 1.1.1 of PROD, for the Netherlands you need to review the legislative history of Bgfo Wft (Parliamentary Documents, House of Representatives (Kamerstukken, Tweede Kamer), 2010-2011, 31980-49) and the EBA POG Guidelines, p. 16.

• 46 Note in this context that the implementation of PARP is a legal requirement in the Netherlands, as follows from Arts. 32 and 32a Bgfo Wft, whereas in the United Kingdom, for banks at least, PARP is set out as a guidance (PROD1); yet for investment firms, it is largely set out as a rule (PROD3), whereby MiFID2 specific rules are supplemented with (explanatory) guidance by the FCA.

• 47 Dutch Act on financial supervision (Wft), https://wetten.overheid.nl/BWBR0020368/2019-02-19.

• 48 EC Directive on markets in financial instruments and amending Directive 2002/92/EC and Directive 2011/61/EU (MiFID2), Dir 2014/65/EU

• 49 See EBA BS 2011/116, GL-44, p. 9, point 29 jo. CRD Directive (2013/36/EU).

• 50 See EBA/GL/2015/18, p. 11, at 6.

• 51 See www.eba.europa.eu/documents/10180/1141044/EBA+GL+2015+18-Compliance+Table-GLs+on+POG.pdf/b675660b-cae9-4921-85a5-02937a3abd46.

• 52 See the FCA Handbook Glossary on the respective definitions of firm, authorised person and regulated activitieswww.handbook.fca.org.uk/handbook/glossary/ (last visited 10 June 2019).

• 53 Parliamentary Documents, House of Representatives (Kamerstukken, Tweede Kamer), 2010-2011, 31980-47, p. 2.

• 54 See e.g. www.afm.nl/nl-nl/professionals/onderwerpen/productontwikkeling.

• 55 ‘Should’ is not the same as ‘must’; ‘should’ is a representation of what is considered to be the right thing to do but is not exactly compulsory. For non-native speakers, however, such a distinction may be difficult to interpret. See also https://keydifferences.com/difference-between-should-ought-to-and-must.html (last visited 10 June 2019); see also Xanthaki, above n. 2, at 433.

• 56 As set out, e.g. in the ‘Joint Position of the European Supervisory Authorities on Manufacturers’ Product Oversight & Governance Processes’ of the Joint Committee of the European Supervisory Authorities (ESA) (JC-2013-77).

• 57 Coglianese, above n. 1, at 21-3; T.W. Merrill, ‘A Comment on Metzger and Zaring: The Quicksilver Problem’, 78 Law and Contemporary Problems 189-204 (2015).

• 58 R. Pawson, Evidence-based Policy, A Realist Perspective (2006); Xanthaki, above n. 2, at 433.

• 59 Note in this context the observance by E.A. Posner and E.G. Weyl, ‘Benefit-Cost Paradigms in Financial Regulation’, Coase-Sandor Institute for Law and Economics Working Paper 2014:660, http://ssrn.com/abstract=2346466 (last visited 10 June 2019), at 10-8, addressing the concept of cost-benefit analysis before legislative implementation, noting that the fact that there is a potential multitude of variables that could be considered in a cost-benefit analysis and as such on the whole relevant for the specific legislation, but that this might in fact not have actual added value for cost-benefit analysis itself, raising the question as to what would cause a relevant variable to be excluded nonetheless. See also Cochrane, above n. 10, at 568.

• 60 J.N. Gordon, ‘The Empty Call for Benefit-Cost Analysis in Financial Regulation’, Colombia Law School and ECGI, Law Working Paper 2014:233, at 1-19.

• 61 Cochrane, above n. 10, at 575.

• 62 Alexander, above n. 32, at 169-73.

• 63 Cochrane, above n. 10, at 567-9.

• 64 Some of these variables could be qualified as ‘confounders’, which may be held constant in (certain) statistical models, thus reducing some of the resources that would otherwise be required for measuring. Note though that this obviously only works after these variables have in fact been identified (see Coglianese, above n. 1).

• 65 The Dutch Ministry of Finance, for instance, recently noted that the general absence of baseline measurements is one of the important causes why it is so very difficult, if not impossible, for effectiveness measurements of financial legislation to take place; see in this regard: Parliamentary Documents, House of Representatives, 2017-2018, 31935, nr. 45. Interestingly, the OECD even argues in the OECD “Policy Framework for Effective and Efficient Financial Regulation” of 2010 that a (full) cost-benefit analysis of financial regulation may not be necessary or appropriate depending on the impact of such legislation and should always be ‘proportionate’ to the regulation under review (see, e.g. p. 31).

• 66 Pawson, above n. 58, at 51.

• 67 Coglianese, above n. 1, at 38.

• 68 It is argued by Coglianese (above n. 1) that such a laboratory-like set-up may be mimicked by reviewing against a group of subjects placed in a jurisdiction similar to that of the target subjects but that do not have the (intended) legislation. Again, this would arguably only work if all other variables can be identified and contained. In practice, it seems very unlikely that such a situation would occur.

• 69 Haarhuis and Niemeijer, above n. 2, at 13.

• 70 Coglianese, above n. 1, at 8; B.P.A. van Mil, A.E. Dijkzeul and M. Noordink, ‘Beoordeling effectiviteit “Nieuwe Stijl”; Een praktijkperspectief op het beoordelen van de effectiviteit van een toezichthouder’, 3(2) Tijdschrift voor Toezicht 81-90, at 82 (2012).

• 71 Haarhuis and Niemeijer, above n. 2, at 10-11.

• 72 P.L.C. Hilbers, D.R. Rijsbergen, K. Raaijmakers and F. de Vries, ‘Effectmeting in het toezicht op de financiële sector’, 3(2) Tijdschrift voor Toezicht 38-49, at 42-7 (2012).

• 73 C. Sorenson, M.K. Gusmano and A. Oliver, ‘The Politics of Comparative Effectiveness Research: Lessons from Recent History’, 39(1) Journal of Health Politics, Policy and Law 139-70, at 162 (2014).

• 74 Ibid.; Mil et al., above n. 70, at 83.

• 75 R. Goldbach, ‘Asymmetric Influence in Global Banking Regulation’, 22(6) Review of International Political Economy 1087-1127 (2015).

• 76 Jackson et al., above n. 36, at 1059; Casey and Scott, above n. 6, at 86-91.

• 77 Pawson, above n. 58, at 26-34.

• 78 Cochrane, above n. 10, at 567-8.

• 79 Ibid., see also Casey and Scott, above n. 6, at 79; Jackson et al., above n. 36, at 1063-4; Langevoort, above n. 15, at 950; Meares, above n. 9, at 398; Tyler (2006), above n. 23, at 381-3; Tyler (2008), above n. 36, at 718.

• 80 Newig, above n. 13, at 277; Poloko, above n. 12, at 426-7.

• 81 Pawson, above n. 58.

• 82 Note that this is but a highly abstracted description of Pawson’s realist synthesis research model.

• 83 Cochrane, above n. 10; Gordon, above n. 60.

• 84 Parliamentary Documents, House of Representatives (Kamerstukken, Tweede Kamer), 2019-2020, 32545-120, see also B.P.A. van Mil, P.C.H. van Dijk, B. Knoote, T.B.F. Beverdam and D. Busch, ‘Evaluatie wettelijke regeling productontwikkelingsproces; artikel 32 Bgfo Wft’, KWINK groep 2020, at 5.

• 85 B.P.A. van Mil, P.C.H. van Dijk, B. Knoote, T.B.F. Beverdam and D. Busch, ‘Evaluatie wettelijke regeling productontwikkelingsproces; artikel 32 Bgfo Wft’, KWINK groep 2020, at 4.

• 86 Ibid., at 17, 20-21, 24-25 and 34.

• 87 Ibid., at 13, 19 and 27.

• 88 Ibid., at 25-27.

• 89 Ibid., at 27-28.

• 90 Ibid., at 34-35.