Published in the Journal for the advancement of performance information and value (2016, 8(2)).http://cibw117.org/journal/archive/
Introducing the Best Value Quality Checklist in Procurement.
For PDF-version, or to share this column via mail, Facebook, Twitter or LinkedIn, click on the icons on the left-hand side.
Introduction: The Best Value Approach (BVA) is a change in paradigm, replacing the non-experts decision making, management, direction and control (MDC) and thinking with the utilization of expertise. The best known BVA model is the Best Value Procurement model (BVP). Problem: With the further proliferation of BVP, a growing number of tenders are labelling themselves as “Best Value” without adhering to proper BVA practices. These ‘hybrid’ tenders producing discouraging results, and have a detrimental effect on the wider proliferation of BVA and BVP. Solution: An easy-to-use tool is proposed that requires no BVA-expertise to assess whether a tender is in fact run as a BV-tender, and which may be used to avoid or minimize decision making. Methodology: A ‘BV Quality Checklist in Procurement’ is established by applying first principles to the Pre-Qualification, Selection and Clarification phase. Result: The three phases have been analyzed with the focus on avoiding all types of decision making resulting in 21 checks. Conclusion: Using the checklist it can be assessed whether a tender procedure is a BV-tender. The checklist can also assist in avoiding decision making in the tender process. The approach used here may furthermore be applied in other fields than procurement.
The Best Value Approach (BVA) replaces the owner/buyer’s decision making, management, direction and control (MDC) and thinking with the utilization of expertise. BVA includes a procurement model, a risk management model, and a project management model. The procurement model is both referred to as Best Value Procurement (BVP) or Best Value Performance Information Procurement System (BV PIPS). In the past these acronyms have been used interchangeably. In recognition of the fact that BVA is not a procurement process, but the application of a set of principles stemming from Information Measurement Theory (IMT), only the acronyms BVA and BV (Best Value) will be used here. ‘BVA’ refers to the higher level principles (as in ‘BVA-experts’), and BV is used as an adjective indicating the Best Value Approach has been applied (as in ‘BV-tender’).
Given the potential for confusion we here explicitly define ‘procurement process’ as the process culminating in the ‘award’ of a contract to a vendor. Hence, the procurement process here covers the phases (until completion) of Pre-qualification, Selection and Clarification as described in (Kashiwagi, 2016). The Execution phase, where BVA’s project management model may be applied (see (Rivera, 2016)), has not been included in working towards the Best Value Checklist in Procurement.
In the procurement process of acquiring goods and services the methodology of BVP creates an environment that optimizes the performance of vendors. It can be used by the buyer or by the expert contractor, and it minimizes decision making, the use of MDC, and thinking by the buyer (the ‘non-expert’ with regards to the expertise he is in need of). Historically BVP has evolved from a past performance information based system and a trust based system to a utilization of expertise approach. This continuing evolution results in different versions of the step-by-step approaches as documented in various textbooks, and may be ground for confusion of and in itself.
However, while BVP methods have evolved, the foundational philosophy has remained constant (see Information Measurement Theory (IMT) and Kashiwagi Solution Model (KSM) (Kashiwagi, 2016a)). Over the years BVP has been simplified and has thus become a very transparent system. Nonetheless, as BVA represents a shift in paradigm, it is not easy to implement BVA for organizations mired in the traditional model of management, direct and control. The BVP process itself, however, is clear and its steps are well documented, and with the guidance of a BVA-expert, each buyer can successfully run a BV-tender (even when BVA is not adopted within the organization).
With the increased public awareness of the results that have been achieved, a larger group of buyers who, in absence of proper BVA training and guidance, have started to use ‘elements’ of BVP in their own specific procurement process, and often present these processes under the flag of running a BV-tender. Consequently a growing number of tenders labeled as ‘Best Value’ are in fact hybrids of BVP and traditional procurement methods. These tenders do not strictly adhere to the underlying IMT/KSM concepts, resulting both in suboptimal results as well as growing confusion and frustration with (expert) vendors who have identified the occurrence of hybrid tenders as one of the most confounding aspects in determining their own approach to BV-tendering (Van Abeelen, 2014).
This development will ultimately result in the slowing down of the broader proliferation of BVP. Furthermore, the resulting predictably poor outcomes and confusion caused by this development will dilute BVP’s performance information, and thus the strength of the BVP performance indicators. As BVP and BVA are generally not distinguished between, it is also likely to distort the perception of what BVA stands for or tries to achieve, and may thus also hinder the application of BVA (IMT/KSM principles) in fields other than procurement.
The development of hybrid procurement forms has been reported early on by Van de Rijt and Santema (Van de Rijt, 2012). They state that, based on IMT, this development is unavoidable and that the initial conditions of the environment in which BVP becomes more popular ‘lead to the predication that here will be many different “ways of using BV” (final conditions)’ which lead them to claim that ‘the heavy demand of the BV/PIPS methodology has the risk of non-experts posing as experts of the BV approach.’
Given that a multitude of hybrid BV-labelled tender forms will appear and develop, several approaches have been suggested to help identify, to maximally prevent and or to ‘dampen’ the negative impact of hybrid tenders.
- Whether BVA’s underlying IMT/KSM principles are being understood, can be assessed simply by observing the process and actions of the contracting organization (see the section on Level II Foundational KSM Characteristics (‘Observable Actions’) in (Kashiwagi, 2016a)).
- A certification process for proven experts in the successful delivery of services using BVA has been developed (Van de Rijt, 2012), providing a transparent tool to identify BVA-experts.
- Witteveen & Van de Rijt (Witteveen, 2013) proposed education as a key concept to successfully implement BVA.
In (Witteveen, 2013) it is stated that ‘education will lead to a correct understanding of the philosophy’ and that ‘this will lead to a better application of BVA, also by the early majority and other “followers”.’ Observations on ten common misunderstandings on the concept and theory which may hinder a successful further proliferation of BVA are laid out. With respect to misunderstanding number 9 (‘There is Only One Way to Apply Best Value’) they state that it is ‘natural that there are various ways of applying the Best Value approach’ and that ‘it is important that the main considerations of the Best Value philosophy remain intact.’
- Vendors without BVA-expertise may not be able to rightly interpret the actions of the contracting organization.
- Certified BVA-experts may not always be sufficiently proficient in the understanding of IMT/KSM principles.
- Certified BVA-experts may not be sufficiently involved in the process.
- Continuing education may not reach everybody who comes across BVA for the first time.
In line with the statement that the main consideration for a BV-tender is that ‘the Best Value philosophy remain(s) intact’, the authors suggest an additional approach to contribute to the successful further proliferation of BVA: a simple checklist based on first principles following from IMT/KSM to verify whether or not ‘the main considerations of the Best Value philosophy’ do in fact remain intact in BV-labelled tenders. The checklist will be derived at by applying the five principles known as TONNNO and part of the approach called ‘Decision Free Solutions’ to the field of procurement.
Decision Free Solutions (DFS) is a generic, systemic approach to minimize risk by avoiding decision making and based on IMT/KSM (Verweij, 2016). DFS is congruous with BVA, with a larger emphasis on the definition of the aim. DFS identifies four generic steps (DICE) and five principles (TONNNO) which can be applied in each and every field. The four steps are Definition (of the aim), Identification (of the expert), Clarification (by the expert), and Execution (by the expert). The five principles to be observed to avoid all types of decision making by the non-expert which manifest themselves as ‘decision making’, ‘MDC’ and ‘thinking’ are: Transparency, Objectivity, No details, No requirements and No relationship.
The link between BVP, IMT/KSM and the five principles known as TONNNO (transparency, objectivity, no-details, no-requirements, and no-relationship) will be described. By systematically applying the principles of TONNNO to the phases as they are known within BVP (Pre-Qualification, Selection, and Clarification) the authors will device a BVP checklist.
With the further proliferation of BVA as applied in the procurement process (BVP), a growing number of tenders labelled as ‘Best Value’ are run in absence of the required BVA-expertise and without adhering to the IMT/KSM concepts which underlie the BVP process. These ‘hybrid’ tenders (mix of BV and traditional procurement processes) have a detrimental effect on the perception of the value of BVP in particular and BVA in general, and result in confusion and frustration with expert vendors. In order to be able to distinguish between a hybrid and a BV-tender process an easy-to-use tool (that does not require BVA-expertise) is required that will allow both buyers and vendors to assess the tender and to make the necessary adjustments (e.g. in expectations, in the tender structure). Such a tool does not yet exist.
By introducing a ‘Best Value Quality Checklist in Procurement’ both buyers and vendors will be given an easy-to-use tool to assess whether or not a tender process is structured as a BV process.
By systematically applying the principles of TONNNO (following from IMT/KSM and resulting in the avoidance of all types of decision making) to the phases as they are known within BVP (Pre-Qualification, Selection, and Clarification) a checklist will be compiled.
IMT, KSM, and no decision making
The BVP process, as described step-by-step in ‘2016 Best Value Approach’ (Kashiwagi, 2016), has at its foundation the concepts following from IMT/KSM. These concepts are ‘no decision making’, ‘no management, direction and control (MDC)’ and ‘no thinking’. These are manifestations of three different types of decision making by non-experts: ‘decision making’ (i.e. choices not substantiated to contribute to achieving an aim), ‘decision making of the past’ (e.g. protocols), and ‘precursors to decision making’ (i.e. thinking), see (Verweij, 2016).
When strictly following the BVP step-by-step the IMT/KSM concepts will automatically be adhered to. However, when project conditions (e.g. legal requirements, company policies) result in deviations from the step-by-step description it is pivotal that these deviations stick to IMT/KSM concepts. The authors argue that in order to label a tender a `Best Value’ or BVP tender, it is not necessary to strictly follow the step-by-step description of the BVP process as described in (Kashiwagi, 2016), but it is necessary to abide by the IMT/KSM concepts as described in (Kashiwagi, 2016a). Consequently, a good understanding of IMT/KSM is required when designing the structure of a BV-tender which deviates from the step-by-step approach.
IMT was first published by Dr. Dean Kashiwagi in 1991 at Arizona State University. It has been continuously developed, tested and refined by the author ever since. IMT was published as a structure for optimizing the effectiveness of information by creating “easy to understand” information environments. The purpose of IMT is:
- Minimize subjective decision-making through the use of dominant information.
- Minimize the amount of data required to accurately transfer information.
- Identify the relationship between information usage, processing speed, and performance.
- Identify a structure (KSM) that minimizes the requirements for decision making, direction, and control of another entity.
- Optimize processes by identifying and removing entities which increase risk and add no value.
IMT can be defined as: ‘A deductive, logical and dominant observation/explanation of an event. It includes the use of relative and related data to predict the future outcome of an event.’
Three related concepts of IMT are ‘natural laws’, ‘conditions’ and ‘events’. Every event (‘anything that happens which takes time’) has a unique set of initial conditions and a unique set of final conditions, and is constrained by unchanging natural laws. Knowing all the initial conditions, and all the natural laws, thus equals knowing the final conditions, or, worded differently, ‘knowing the future’. IMT thus defines ‘the expert’ as the person/organization who can predict the future because the expert perceives all initial conditions and all natural laws.
In reality no person or organization can accurately perceive all the natural laws and all the initial conditions of their environment. In IMT the ‘cycle of learning’ was created to explain how a person’s (or organization’s) perception changes, and how the rate of change can be used to predict their future behavior. In short, the ‘Cycle of Learning’ proposes that the application of newly perceived information causes change, and, by observation, change leads to the perception of more information.
Combining the definition of an expert with the ‘Cycle of Learning’, ‘expertise’ in the worldly sense is now explained: the expert vendor is the vendor that perceives most of the initial conditions (more than other vendors), has a higher rate of change, and thus will perceive still more information (than other vendors).
It is pivotal to understand that expertise is linked to the perception of information, and that the more information is perceived the higher the rate of change, resulting in a still higher perception of information. In other words, the expert’s amount of expertise increases quicker than the amount of expertise the non-expert possess. From this follows that the expert is better able to ‘look into the future’. Consequently, by definition, the greater the expertise, the greater his possession of information to identify or predict the future, the fewer decisions need to be made.
The ‘amount of decision making’ is thus a predictive factor for expertise. IMT states that not just ‘decision making’ but that all characteristics (of a person or organization) are relative and somehow related to the capacity (of a person or organization) to perceive, process, and apply information. In KSM the concepts of IMT are used to show the relationship between different characteristics.
The tenets of KSM are:
- Characteristics are related to the ability to perceive information
- All characteristics are related in relation to the amount of information perceived and used by an individual
- A person or organization’s behaviors can be predicted by only knowing a few characteristics.
From KSM follows that persons or organizations that perceive more information will make fewer decisions and will be more efficient (they expend the minimum amount of resources to meet the accurate expectations). They will have more experience (or information), fewer expectations and they will align people based on their capability (as opposed to trying to manage, direct and control people). For a longer and more comprehensive list of related characteristics see chapter 4 of (Kashi-wagi, 2016a).
IMT/KSM gives the foundational understanding to explain the ‘why’ behind the BVP structure. A structure that does not depend on decision making or experience of individuals, and that avoids the need for details, frequent communication, thinking, and the all too common tendency for MDC. The common denominator that is to be avoided is all types of decision making.
The first principles to avoid decision making (TONNNO)
The aim of the ‘BV Quality Checklist in Procurement’ is to verify that all the conditions are in place to select/award the vendor who achieves the aims of the buyer with minimal risk. This aim translates into ensuring that nothing in the process obstructs identifying the expert, and that the conditions are in place to fully utilize the expert’s expertise. BVP sets out to achieve this through ‘no decision making’, ‘no MDC’, and ‘no thinking’ by the non-expert.
In essence the Best Value Approach, and the approach of Decision Free Solutions, is about identifying anything that is either a decision, a result of a decision, or a precursor to a decision – where a ‘decision’ is defined as ‘a choice not substantiated to contribute to achieving an unambiguous aim’ (Verweij, 2016) – and then consistently applying a set of principles to avoid (the result of, or the lead up to) any kind of decision making itself.
Decision making occurs when information is insufficient and, thus, the outcome of an event cannot be accurately predicted. When the buyer makes a decision he incurs risk. In the BVP process this risk may be defined as ‘not awarding the contract to the expert vendor’. The buyer should avoid all types of decision making in order to select the expert vendor.
In order to avoid unsubstantiated choices (i.e. decisions) transparency is pivotal. Transparency allows all to perceive the initial conditions. As the buyer is to identify the expert-vendor based on the definition of the aim, an unambiguous aim is pivotal in avoiding decision making. The aim shall be measurable (objectifiable) and not constrict the use of an expert’s available expertise by containing details or enforcing unnecessary requirements. When a client enforces too many requirements, vendors tend to only focus on meeting these requirements instead of providing the best value. When all vendors try to reach the same requirements, they become similar, making expertise more difficult to recognize. The buyer should avoid making decisions by relying on existing relationships, thereby discriminating against vendors. Relationships have a high risk of directing the process away from expertise. In short, transparency, objectivity, no details, no requirements and no relationship are all principles to be adhered to in avoiding decision making.
These same principles will thus also decrease the non-expert’s MDC and thinking. MDC constraints expertise. When the non-expert tells the expert what to do or how to do it (e.g. making the expert follow established protocols (decisions made in the past)), the expert cannot fully employ his expertise. Transparency here is ‘the risk management mechanism for an expert, replacing MDC’ (chapter 3 of (Kashiwagi, 2016)). The non-expert is not to place requirements on how the expert is to use his expertise, and not force the expert to communicate details (e.g. on how the expert does his job), as these will result in forcing the non-expert to think and make decisions. Details (lack of simplicity) make matters more complex and pull one away from a high-level overview. Details are indicative of lack of understanding, make matters less transparent, and place restrictions on solutions. Thinking by the non-expert means the expert has failed in making the non-expert understand (not transparent enough) that what the expert does contributes to achieving the aim (relationship to objective not clear).
In summary, the first principles collectively referred to as ‘TONNNO’ which need to be applied to avoid all manifestations of decision making (‘no decision making’, ‘no MDC’ and ‘no thinking’) are the following:
- No details
- No requirements
- No relationship
These principles are applied in the phases of ‘Pre-Qualification’, ‘Selection’ and ‘Clarification’ to compile the Best Value Quality Checklist in Procurement.
European procurement law, TONNNO, and BVP
The European public procurement directives, as interpreted in the case-law, expresses three guiding principles: transparency, equal treatment, and proportionality (Weller, 2011). Of these ‘transparency’ is already identified as a first principle to avoid decision making, and ‘equal treatment’ follows from i) not excluding vendors from the tender process based on the buyer’s decision making in defining the aim (e.g. by including ‘details’ or ‘requirements’ which steer the solutions that may be proposed away from certain vendors), and ii) from avoiding decision making during the Selection phase. For example, if the aim is not measurable (objectifiable), then the buyer tends to make a decisions to determine when the aim will actually be achieved (and which vendor comes ‘closest’). If some vendors have greater access to the buyer’s time or information than others, than this defines a ‘relationship’, etc.
Whether the principle of proportionality has been complied with in a given instance requires a two-step enquiry: first, determining “whether the measure at issue is appropriate for attaining the objective pursued” and second, determining “whether the measure at issue goes beyond what is necessary to achieve the objective” (Weller, 2011). For example, if the buyer introduces the need for ‘sustainability’ within its aim to achieve the broader objective (e.g.objectives of environmental protection and improvement of social cohesion) then the proportionality principle requires the determination of whether the criterion of ‘sustainability’ is appropriate for achieving the broader objective, and whether the criterion goes beyond what is necessary to achieve these objectives. By ensuring that all elements of the buyer’s aim can be made objective during assessment, then the proportionality principle will always be adhered to. In conclusion, by adhering to the TONNNO-principles one also adheres to the guiding principles of the European procurement law.
BVP replaces the traditional evaluation methodology which entails a long list of requirements which are to be scored. BVP creates an effective system which allows an assessment team to select an expert vendor using only a small number of documents and brief interviews. This system replaces traditional “objective” methodologies (scoring of individual requirements) with an inherently more “subjective” one (assessing quality documents). Running BVP care must be taken to avoid the situation where scoring quality documents is perceived to be at odds with the guiding principle of “equal treatment”.
A particularly useful ruling in this respect is the one provided by the “preliminary relief judge” G.P. van Ham (in Dutch: voorzieningenrechter) on September 16, 2015 in the case between a vendor and the buyer using BVP for the procurement of medical equipment (Ham, 2015; Verweij, 2015). In this case the vendor claimed that the scoring methodology of BVP was subjective and therefore not allowed. In his ruling, the judge recognizes the inherent tension between the objective scoring methodology required by European procurement law and the subjectivity inherent to assessing qualitative documents. He goes on to state that this does not mean, however, that this subjectivity is, by definition, “contrary to the principle” of European procurement law. The ruling then mentions the following requirements when running a BV tender:
- It must be absolutely clear to the tenderer what is expected of him
- The scoring of the BVA-documents are to be assessed using a scoring system that is ‘as objective as possible’
- The contracting authority is to motivate its assessment in a manner which allows the rejected tenderer:
- to assess the way the assessment took place
- to verify the assessment validates the (pre-)award decision
These requirements, as well as several other meritorious issues raised in the buyer’s plea in this legal case (see (Verweij, 2016a)), are all addressed in the BV Quality Checklist in Procurement.
Best Value Checks in the Pre-Qualification phase
When the ‘Pre-Qualification’ phase is understood in its narrowest sense (as it may be interpreted to be in (Kashiwagi, 2016)) to reduce the number of vendors to take part in the selection phase, this phase does indeed not have to be run. The associated risk of running a pre-qualification phase is that expert-vendors may be excluded.
Here, however, the Pre-Qualification phase will be interpreted in line with DFS’s ‘Definition’ step, in which the aim is defined and verified to be unambiguous. The Pre-Qualification phase now becomes the most important phase, as in this phase the non-expert’s aim gets defined which:
- is used to identify the expert,
- will be achieved by this expert.
The Pre-Qualification phase creates the optimal conditions for the buyer to identify the expert in the Selection phase. The following activities take place in the Pre-Qualification phase:
A – Definition of the aim and verification that the aim is unambiguous and understood by both the buyer and prospective tenderers
B – Education of both the assessment team and prospective tenderers in BVA and the procedure of BVP
C – Definition of ‘functional requirements’ and ‘minimum requirements’ (e.g. legal, financial) for the actual pre-qualification of prospective tenderers.
A – Definition and verification of the aim
It is pivotal to clearly define what is meant with ‘aims’ and how these are different from ‘deliverable’ and ‘scope’. Following the definition as found in (van de Rijt, 2016), the ‘deliverable’ is a higher level description of the ‘what’. The ‘scope’ is defined by the awarded expert vendor and comprises the ‘how’. The ‘aims’ go beyond the ‘deliverable’ and typically address critical success factors for the project. Aims may carry in them an ambition, or may describe ‘the vision, goal or intent of the client’ (Kashiwagi, 2016). It is with respect to the aims the tenderers can distinguish themselves, not with respect to the the deliverable. As noted by (van de Rijt, 2016), the deliverable is to be described separately, and is not to be part of the (description of the) aims. See figure 1 for a schematic representation.
As stated, the aims determine what is ultimately achieved by the expert vendor, the aims shall allow tenderers to distinguish themselves, and the aims must allow for an unambiguous assessment of the tender. (Verweij, 2016) provides the following remarks and suggestions with respect to defining the aim:
- The aim must be defined in relation to the ‘system’ in which it is to be realized
- When performing a ‘market consultation’ among prospective-experts (to aid in defining the aim), the observations made of the characteristics of the prospective-experts will aid in positively identifying experts (whether or not they are also experts-in-achieving-the-aim is then to be determined in the Selection phase)
- The buyer may provide information on ‘sought performances’ or ‘what-we-think-we-want’ to further increase the understanding of the aim by the vendor
- The buyer is to prioritize the various elements of the aim
- The buyer is accountable for defining the aim
On the one hand aims are the ‘vision, goal or intent of the client’, while on the other hand the aims must be sufficiently transparent and objective to allow for an as-objective-as-possible assessment of the tender documents. An aim is always defined within a certain context or ‘system’, and this context provides the prospective-experts with valuable information. Examples are the relation between the aim and an organization’s mission or strategy, recent or expected internal or market developments, financial and political situation, dependence on personnel or priorities which may change, etc.
The selection of the expert-vendor for the aim hinges on the transparent and objective definition of this aim. For this reason the non-expert is advised to provide opportunity for vendors to ensure the aim is unambiguous and understood the same way. This can be done by way of a market-consultation. A market consultation may then have three purposes: i) aid the buyer in defining the aim, ii) ensuring the aim is understood the same by buyer and vendors, iii) to observe the characteristics of vendors to be able to assess whether they are (likely to be) experts. When doing a market consultation the principle of ‘no relationship’ is to be adhered to: avoid discrimination and either allow all vendors to participate in the market consultation or have objective guidelines which can be linked to the aim.
BVP sets out to align expertise. The buyer tends to be the expert with respect to ‘how’ and/or ‘why’ to use the product or service being tendered for. This generally makes it of critical importance to provide vendors with a description of the the buyer’s own expertise. A tenderer may then incorporate this expertise in how best to achieve the aim (and when selected include it in how the deliverable will be achieved, see also figure 1).
To provide further context with respect to the aim the buyer may define ‘sought performances’ or ‘what-we-think-we-want’. ‘Sought performances’ provide a map to the buyer’s intended use of the solution, and ‘what-we-think-we-want’ sets an expectation of certain key performances relevant to how the buyer intends to use the solution. It should be noted, however, that even though these ‘sought performances’ and ‘what-we-think-we-want’ are not ‘minimal requirements’, they may still incite some vendors to perceive them as such. When providing this context the buyer must carefully consider whether or not this may ‘level the playing field’ and thus make it harder to identify experts able to achieve the aim.
The buyer is advised to explicitly prioritize the various elements of an aim. This may be a reflection of the buyer’s expertise, and it may also be used in the assessment of the BVP documents and interviews in relation to the ‘most relevant’ aims. Even though the buyer may not always be in the position to assess the priority of the elements of the aim, as it does not preclude the expert from substantiating a re-prioritizing of the elements, there is no perceived downside to prioritization.
Finally, care must be taken that the aim is understood in an unambiguous way by the buyer’s organization itself. When changes have been incorporated, they should be shared within the buyer’s organization prior to the start of the tender. The aim does not define the scope (as it is defined by the expert-vendor), but it is pivotal in selecting the expert-vendor. The aim is always owned by the buyer, and it is what the expert-vendor sets out to achieve.
The following checks are defined:
- The aim is linked to the context or ‘system’ of the buyer
- The aim is transparent, objective, and does not contain details or requirements
- The aim is understood the same by vendors, buyer and the buyer’s organization
- During the pre-qualification phase all the prospective vendors are provided with the same opportunities
- The buyer’s own expertise in relation to the aim is clearly defined
- Care is taken to ensure that provided context (e.g. ‘sought performances’, ‘what-we-think-we-want’) is not interpreted as minimal requirements compounding the identification of the expert
- The various elements of the aim are clearly prioritized
B – Training of both the assessment team and prospective tenderers in BVA
The BVP method uses several filters to avoid situations in which a non-expert is awarded the contract over an expert based on the non-expert having hired the better BVA-expertise ( see also (Verweij, 2016b)). However, by providing BVA training, including in the use of dominant information/performance metrics, the selection of the expert will be facilitated.
The same goes for BVA-training of the assessment team in how to score BVP documents in relation to the (prioritized elements of the) aim. An assessment team trained in BVA will be able to substantiate its scores, achieve consensus more rapidly, and minimize the risk of any decision making taking place.
The training is also to include how the buyer will run the BVP process from beginning to end.
Note that training and explanation of how BVP will be run can also take place after the Pre-Qualification phase.
- The (prospective) tenderers and members of the assessment team are trained in BVA
C – Definition of pre-qualification requirements
While ‘legal’ or ‘financial’ requirements may be defined to pre-qualify vendors, these requirements carry the risk of excluding experts (when these requirements are merely decisions). These requirements shall be relevant in relation to achieving the aim (they may e.g. avoid a perceived risk).
However, using ‘functional requirements’ that reflect the expertise of the buyer, and thus can be substantiated by the buyer to be relevant with respect to the aim, do not carry the risk of excluding experts able to achieve the aim.
An example of a functional requirement used in the procurement of innovative medical equipment (Verweij, 2015a) was the need for the solution to be able to rotate a full 360 degrees around the patient. This reflected the buyer’s expertise of the need to ‘access’ the patients from all angles while having to avoid moving the patient as this was scientifically shown to have a negative impact on the quality of the treatment. This requirement excluded several vendors from participating in the tender.
- All requirements used to pre-qualify vendors are substantiated to contribute to achieving the buyer’s aim
Best Value Checks in the Selection phase
Almost every BVP procedure is unique. Various formal procurement procedures (i.e. in case of public procurement in Europe) can be used, and the number and type of documents to be provided may vary depending on local conditions. The essence is always that the buyer’s organization avoids all types of decisions by applying the principles of TONNNO.
The Selection phase is to identify the expert-tenderer in relation to the aim. The buyer provides tender regulations, detailing the BVP process and how the pre-award and the award takes place, and provides a price-ceiling.
The tenderer is to demonstrate its level of expertise through claims and substantiations in a series of BVP documents (usually ‘Level of Expertise’, ‘Risk Assessment’, and ‘Value Added’), and up to three interviews with personnel the tenderer determines to be key to achieving the aim (see (Kashiwagi, 2016)).
The following activities take place in the Selection phase:
D – Buyer’s organization publishes tender regulations
E – Publication of price ceiling by buyer
F – Agreement (legal contract) is shared
G – BVP documents and interviews are scored by individual team members
H – Achieving consensus scores and writing up motivations for rejected tenderers
D – Publishing of tender regulations
From the legal ruling of (Ham, 2015) the tender regulations:
- Must make it absolutely clear to the tenderer what is expected of him
- Must describe the method of scoring of the BVA-documents which is ‘as objective as possible’
- Explain how the assessment will be motivated by the buyer (whereby the motivation must allow the rejected tenderer to assess the way the assessment has taken place, and to verify the pre-award decision has been validated by the assessment)
Transparency and objectivity are key. The tender regulations are to describe the BVP procedure and what is expected of the tenderer, how the BVP documents and interviews will be assessed, and how the assessment of the individual tenderer will be motivated, also against the tenderer who will be selected.
In order to avoid discrimination during the interview stage, interview questions are standardized and enforced for all potential vendors. These should sufficiently assess the relevance of the interviewees expertise in relation to the aim. To the standard set of questions the tenderer may be asked to clarify statements made in the BVP documents, but care shall be taken to not trigger exchanges filled with detailed information: it is not the tenderer’s ‘how’ or ‘scope’ which is to be assessed.
Special attention is warranted to describe the awarding criteria. The BVP documents, the interviews and the price will be scored. It is to be stressed that any criteria used to determine the score for each awarding criterium is itself also an awarding criteria. All awarding criteria, and how they are assessed, must be transparent, objectifiable and non-discriminatory. If ‘ambition’ or ‘commitment’ is deemed relevant in scoring, it is advised to include this as an awarding criterion and have a clear idea (and description) of how it will be assessed.
A transparent and objective description of the awarding criteria is to be accompanied by transparent and objective description of the scoring terms. What are the distinguishing features between scoring e.g. ’good’ or ‘excellent’? Both descriptions are furthermore essential in providing the motivation of the pre-award to the rejected tenderers. It is important to pay close attention to how the rejection will be motivated in a way that is sufficiently transparent to the rejected tenderers. An important element which may be used in the motivation is the prioritization of the various elements of the aim (which is then to be included in the tender regulations).
It comes recommended to mention in the tender regulations what the rejected tenderers may expect in the motivation they will receive. IMT/KSM suggests that rejected tenderers who are not experts are more likely to litigate than rejected tenderers who are experts, but comparatively, not the best experts in relation to the aim. A rejected tenderer who is not an expert will look for excuses, is more likely not to have understood BVP, thinks in win-lose, and relies more on contracts. From there litigation will only be a small step. If the motivation only refers to ‘not dominant enough’, ‘too few metrics’ etc., and then hides behind ‘confidentiality’ with respect to the selection claims, this may not prevent the buyer from receiving a court date (Verweij, 2016a).
- The BVP process, the awarding criteria, how the assessment will take place (including scoring criteria), and how the assessment will be motivated for rejected-tenderers are all transparently described and as objective and complete as possible.
- To avoid discrimination interviewees receive the same set of questions and interviews all have the same duration. The number of clarification questions shall be limited and detailed discussions shall be avoided.
E – Publication of price ceiling
The price ceiling shall not be the result of decision making. This means that it is, at the very least within the buyer’s organization, to be substantiated, for example by making a business case. When it is not possible to determine a reliable price ceiling, than it shall be left to the tenderers. The price ceiling shall not be used as a MDC-instrument by the buyer’s organization. This will increase the likelihood that a non-expert will be identified through strategic tendering below cost price. Even when the non-expert will be discarded during the Clarification phase the tender procedure will be considerably prolonged and at risk of being invalid.
Note that BVP is often used in the Value Based quadrant II of the Industry Structure (see chapter 8 in (Kashiwagi, 2016)) where the need for high performance occurs in a market where the perceived competition is above average. In these cases, price is often of much lesser importance than quality, and often the quality of a product or service (e.g. ‘productivity’) may have a very large impact on the overall business case that goes over and beyond mere the price difference of the product or service.
- The price ceiling is substantiated and realistic
F – Agreement (legal contract) is shared
The agreement is (generally) to be shared with the tenderers at the same time as the tender regulations: at the beginning of the tender procedure. Traditionally the agreement is a MDC-tool containing many ‘decisions made in the past’ like guidelines and company policies and instructions. The agreement tends to be long, hard to read, detailed, and include remunerations when certain requirements and or timelines are not met (chapter 5 of (Kashiwagi, 2016)).
In the BVP-contracting model the expert vendor is the offerer and the buyer is the acceptor of the offer. The expert vendor thus writes what and how they are delivering, how they will minimize the buyer’s risk and concerns, and what is required from the buyer. In BVP the legal contract no longer is about Quality Control but about Quality Assurance, whereby the expert vendor becomes the main author of the legal contract.
In practice, the structure and the MDC-nature of the legal contract is often unchanged from the traditional procurement. Frequently the expert vendor may indeed provide the ‘how’ and ‘what’ of what will be delivered (the deliverables of the clarification phase tend to become part of the legal contract), but the legal contract generally still contains standards, guidelines, requirements and remunerations. The buyer cannot always avoid this, as there may be national or company standards and or guidelines that must be abided by, and/or standard legal contracts that must be used.
When running a BVP tender, the practical importance of the agreement is starkly reduced as the expert vendor has no internal risk and risks not under his control are identified and mitigated. This however, does not make the agreement a document of lesser importance.
Care should be taken that the agreement is primarily a tool for Quality Assurance, describing not so much how things need to be done, but how matters are to be resolved. Elements of control (e.g. remunerations) that cannot be removed from the agreement shall be, for as far as possible, substantiated against achieving the aim. Ultimately, it is in the buyer’s interest that the agreement does not restrict the awarded tenderer’s use of expertise in achieving the buyer’s aim.
- The agreement is used for Quality Assurance, not Quality Control
- Any control element the buyer still leaves in (e.g. ‘minimal up time’) is to be substantiated against the aim
- Any remuneration included shall pertain to the vendor’s internal risk only
G – BVP documents and interviews are scored by individual team members
Training of the members of the assessment team in scoring the documents and interviews is of great importance. The substantiations provided by the team members are valuable both for the discussion to come to consensus and can be used in the motivation letters to the rejected tenderers.
To avoid decision making in the individual scoring, it shall be properly described what score to give, and when. When scoring using terms such as ‘poor’, ‘insufficient’, ‘neutral’, ‘good’ and ‘excellent’, the distinguishing characteristics between e.g. ‘good’ and ‘excellent’ should be unambiguous. Also here the prioritization between the various elements of the aim can be used to provide guidelines. The definitions of the various scoring-terms are to be provided in the tender regulations.
The advantage of using the mentioned scoring-terms over numbers from 2 to 10 is the absence of the connotation that a ‘10’ is supposed to mean ‘perfect’. It will never be perfect, but it may be excellent.
- The assessment team members are trained in how to score using the scoring criteria, and how to substantiate their scores
H – Achieving consensus scores and writing up motivations for rejected tenderers
With the assessment team trained in how to score using transparent and objective scoring criteria and in how to substantiate their individual scores, coming to consensus and writing up motivations for rejected tenderers becomes straight forward. The key remains to avoid decision making. As a general rule protracted discussions on how to score a document means transparency is lacking somewhere. Namely: the text in the BVP document is ambiguous, the scoring criteria are not sufficiently clear, or a team member’s substantiation is not understood.
It is important that it is clear both to the assessment team and to the (rejected) tenderers what the procedure is when the assessment team does not achieve consensus on an individual score (especially when this may impact the overall ranking of the tenderers). This situation is always the result of a lack of transparency or decision making earlier in the process.
Of special interest is the selection of the members of the assessment team. IMT/KSM recognizes that not everybody will follow the paradigm shift of the Best Value Approach. Traditionally, ‘relationships’ tend to play an important role in selecting the assessment team. Managers or procurement officers would often attempt to “control” or sway the selection process by appointing biased evaluators. In such a situation, it is more preferable to appoint an outside party with an understanding of IMT/KSM principles (someone who has the characteristics of an expert) to serve as an arbiter who can substantiate accurate scores without bias.
- In assembling the assessment team ‘relationships’ are avoided – or else risk mitigation measures (determining how consensus will ultimately be achieved) put in place
- The procedure to come to a consensus is mentioned in the tender regulations
Best Value Checks in the Clarification phase
If the selected vendor passes the verification ‘phase’ following selection, and if decision making has been avoided in the Pre-Qualification and Selection phase, and all Best Value checks have been ticked off, then everything is lined up for success in the Clarification phase. The selected vendor has been positively selected as the expert to achieve the buyer’s aims (and is therefore likely to understand BVA), and the assessment team members are both trained and now also fully familiar with BVA and the importance of avoiding decision making.
In practice, this remains a crucial phase where signs of ‘remission’ are all too likely. Both buyer and selected vendor may experience ‘relief’ and are excited to move forward. The building of a ‘relationship’ is both logical (both are working to achieve a win-win situation) as it is a risk (giving a pass on ‘substantiations’ and relying on ‘trust’ instead). In this phase, decision making still must be avoided, as it is in this phase that the pre-awarded vendor is to clarify how the aim will be achieved.
The only type of ‘trust’ that is permissible in this phase is the trust that is an extrapolation of demonstrated performances. The observed characteristics and performances until now may have demonstrated to the assessment team that the selected vendor is not only an expert (in IMT/KSM’s sense of the word), but also the expert to achieve the buyer’s aim. This only means that the BVP procedure has been successful so far and that the selected vendor is very likely to be awarded too. This does not mean that the BVP procedure has been successfully completed.
The following activities take place in the Clarification phase:
I – Determination of scope and plans by selected vendor
J – Periodic reporting and determination of performance measurements (KPI’s)
I – Determination of scope and plans by selected vendor
In the BVP procedure it is the vendor who is in the lead in the Clarification phase and who is to determine the schedule for providing and substantiating all the needed documents. These documents typically become part of the agreement.
The buyer’s organization is to accommodate the vendor wherever possible by having the (required) assessment team members available for meetings and to assess the provided substantiations. The assessment team is to make it explicit when provided substantiations with regards to the vendor’s deliverables in this phase have been sufficient. The assessment team is advised to define the procedure by which it ‘ticks off’ deliverables as being accepted from the buyer’s perspective.
To avoid the risk of entering into detailed discussions, the vendor is to ensure the various elements of the aims will be achieved, and to provide language that links its scope and performances to these aims. The situation shall be avoided whereby the assessment team is put in a position of having to trust the vendor in the wrong sense of the word. For example, if the vendor lists the performances with which he will achieve the aim, and he provides a list of (technical) tests by which the performances have been determined to have been achieved, then he forces the assessment team to ‘trust’ the vendor that passing the tests does indeed guarantee that the aim will be achieved. The vendor shall instead guarantee that the aims will be achieved, and that the tests are merely (internal) checks to demonstrate meeting the required performance-criteria to achieve the aim. A few tests demonstrating a working system, as it will be used by the buyer, are much more useful than a long list of technical tests. Still, if all defined tests are passed, but the aim is not achieved, the vendor has still failed.
- The assessment team has a procedure in place by which the deliverables provided by the vendor are accepted as being sufficiently substantiated
- The vendor consistently substantiates against, and guarantees the achieving of, the aims (and avoids the assessment team starting to think)
J – Periodic reporting and determination of performance measurements (KPI’s)
The vendor is to ensure that it will be able to fully use its expertise in achieving the aims in the Execution phase. The vendor thus has to ensure that the buyer’s organization will not employ MDC-measures. The BVP procedure employs the Weekly Risk Report (WRR) as a means to keep the buyer’s organization informed on the status of the plan and the occurrence and impact of eventual deviations to the plan to the achieving of the aims (in terms of time, cost, quality).
This periodic reporting can already commence during the Clarification phase itself. The combination of a detailed plan with milestones and frequent reports on (any) deviations to the plan (including the mitigation measures taken if required) will take away the underlying reason for the buyer’s organization to MDC the vendor.
By defining Key Performance Indicators (KPI’s), the vendor will be able to both communicate progress to the plan during the Execution phase using metrics, the ‘language of transparency’, as well as performance of the provided solution once operational. Care is to be taken that the KPI’s are related, in one way or another, to the achieving of the aim. They shall not be indicators for the vendor’s internal achievement. KPI’s shall also not become a control instrument. Examples of KPI’s that may be of relevance to the buyer’s organization may include the number (or ratio) of open and resolved issues, the average time it takes to resolve issues, number of days ahead or behind schedule, percentage of performance tests successfully concluded, percentage of system availability, etc. etc. (see also (Hutten, 2016) for the use of KPI’s in Best Value).
- The vendor sets up periodic reporting on (any) deviations to the plan during the Execution phase using relevant KPI’s relating to the achievement of the aim and not to the vendor’s internal performance
Overview of Best Value Checklist in Procurement
A List of 21 checks has been defined which are designed to avoid all types of decision making. Given the complexity of the process, there are many more instances of decision making to be found which may not all be covered by these checks. In this sense, the number of checks provided is arbitrary. Nonetheless, the authors are of the opinion that the list of checks provided is a useful addition to existing methods put in place to assure a BVP process is set up in line with IMT/KSM principles.
Since BVA presents a paradigm shift, both vendors and buyers are likely to have strong tendencies to use practices which are not in line with IMT/KSM principles. The checklist will aid in identifying instances of decision making, and thus will allow them to be avoided. The availability of (access to) BVA-expertise (even more so than merely experience with the BVP process) remains pivotal, as is the timely training and eduction of both (prospective) tenderers, the buyer’s organization and the assessment team members.
As a mere suggestion, running a ‘BVP-test’ by requesting (prospective) tenderers to provide i.e. a single page BVP document in relation to a particular aim, having the assessment team members score and substantiate the pages, and have a BVA-expert provide feedback to all, will go a long way in making the BVP procedure more transparent to all.
The time in which what check-related activity should take place may vary between procurement procedures and also depend on the possibilities the procedure allows (e.g. in case of public procurement in Europe). Pre-qualification of vendors may be required simply to keep the work load to a level that can still be handled by the buyer’s organization. In other instances, the phase may be only be used to come to an unambiguous definition of the aims. Time and financial constraints usually apply, and there is no one way to run a BVP procedure that will meet all needs. Pivotal is only that IMT/KSM principles are adhered to.
As the title of the article implies, Best Value checklists can be devised for fields other than procurement. Another field under consideration to apply the principles of TONNNO to is the field of project management.
With the further proliferation of BVA as applied in the procurement process (BVP), a growing number of tenders labelled as ‘Best Value’ will be run in absence of the required BVA-expertise and without adhering to the IMT/KSM concepts which underly the BV process. In order to be able to distinguish between a hybrid and a BV-tender process an easy-to-use tool (that does not require BVA-expertise) will allow both buyers and vendors to assess the tender and to make the necessary adjustments (e.g. in expectations, in the tender structure).
By introducing a ‘Best Value Quality Checklist in Procurement,’ both buyers and vendors are provided an easy-to-use tool to assess whether or not a tender process is correctly labelled as a BV-tender.
By systematically applying the principles of TONNNO (following from IMT/KSM and resulting in the avoidance of all types of decision making) to the phases as they are known within BVP (Pre-Qualification, Selection, and Clarification) a Best Value Checklist in Procurement has been arrived at which is believed to be of benefit to both buyers and vendors, and in the hope it will contribute to the further proliferation of the Best Value Approach, both within and outside of the field of procurement.
Ham, G. P. (2015), Ruling by court of The Hague (the Netherlands), case number: C/09/490634 /KG ZA 15-877, September 16, 2015.
Hutten, J. (2016), Demming en Seddon in Best Value, chapter 3 in (van de Rijt, 2016).
Kashiwagi, D. (2016). 2016 Best Value Approach. Kashiwagi Solution Model (KSM), Mesa, Arizona
Kashiwagi, D. (2016a). 2016 Information Measurement Theory with the “Kashiwagi Story”. Kashiwagi Solution Model (KSM), Mesa, Arizona
Rivera, A., Kashiwagi, J., Kashiwagi D. (2016), Improving the Management of Environmental Engineering Projects through the Best Value Project Management Model for a State Agency, Journal for the Advancement of Performance Information & Value, 8(1), 6-1 to 6-34.
Van Abeelen, A., Koreman G., Verheul, D., De Best Value-aanpak: dilemma’s voor de aanbieders, chapter 1 in (Van de Rijt, 2014)
Van de Rijt, J., Santema, S. C. (2013). Prestatieinkoop. Met Best Value Naar Succesvolle Projecten. Graphicom International.
Van de Rijt, J., Santema, S.C. (2012). The Best Value Approach in the Netherlands: A Reflection on Past, Present and Future. Journal for the Advancement of Performance Information & Value, 4(2), 147-160.
Van de Rijt, J., Witteveen, W. (2014), Best Value Werkt – Ervaringen uit de praktijk, Gaphicom International
Van de Rijt, J., Witteveen, W., Santema, S.C. (2016), Best Value Stroomt – Inzichten uit de praktijk, de jurisprudentie en de wetenschap, Graphicom International
Verweij, J., de Vries, W., Kashiwagi, I., Kashiwagi D. (2015). A Procurement Method that Considers Innovation. Journal for the Advancement of Performance Information and Value, Vol 7, No 1.
Verweij, J., de Vries, W. (2015a). Use of Best Value for high-value highly-complex medical equipment (presentation). Retrieved http://ksmuniversity.ksm-inc.com/
Verweij, J. (2016), Introducing Decision Free Solutions – a generic systemic approach to minimize risk by avoiding decision making. Journal for the Advancement of Performance Information and Value, Vol 8. No 2.
Verweij, J. (2016a), A legal challenge to BVA! The case, the ruling, the lessons (presentation). Retrieved http://ksmuniversity.ksm-inc.om
Verweij, J. (2016b), BVA in a new market – Is there a risk of selecting the BVA-expert instead of the expert-vendor? Drawing a general lesson from a unique case (presentation). Retrieved http://ksmuniversity.ksm-inc.om
Weller, C., Pritchard, J.M., Maitre, E. (2011). The guiding principles of public procurement transparency, equal treatment and proportionality; Identifying Opportunities for Sustainable Public Procurement Briefing Series; Briefing No. 3; http://www.clientearth.org/
Witteveen, W., Van de Rijt, J. (2013), Possible Barriers to a Successful Further Diffusion of the Best Value Approach in the Netherlands: Observations of Major Misunderstandings on the Concept and Theory. Journal for the Advancement of Performance Information and Value, Vol 5, No 2.