Public Procurement of AI: Current and Future Challenges
Artificial intelligence (AI) is the hot topic of the moment. Interest in the use of AI systems is growing in both the private and public sectors. The motives are obvious: the hope is to increase the efficiency and speed of decision-making processes, save costs and achieve better results overall.
However, the regulation of AI is still in its early stages. At the end of 2023, the EU Parliament and Council agreed to adopt a regulation to regulate AI systems based on the proposal submitted by the Commission in 2021 (AI Act). It was formally adopted by the EU Parliament on 13 March 2024 and has yet to be adopted by the Council. The AI Regulation enters into force 20 days after its publication in the Official Journal of the EU and will be fully applicable - with some exceptions - 24 months after its entry into force. The AI Act is intended to ensure a reasonable balance between risk and innovation. The needs of citizens, SMEs and start-ups will be particularly taken into account and protected.
In light of the expected AI Act, potential challenges in relation to the procurement of AI systems shall be identified. Questions in the context of direct use of AI in a procurement procedure, e.g. in tenders or tender preparation with the help of AI, are explicitly not addressed.
The following considerations are a continuation in a series of BLOMSTEIN briefings addressing AI-related aspects of public procurement law, competition law, trade/direct investment (FDI) and ESG.
AI Act with a risk-based approach
The AI Act is intended to pursue a risk-based approach, i.e. AI systems are assigned to different risk levels. The group deemed to bear minimal risk is likely to include the majority of current AI systems. The AI Act should impose hardly any regulatory obligations on them. The subsequent high-risk group should, for example, introduce systems for risk minimisation and human control and guarantee minimum requirements regarding the robustness, accuracy and security of their systems. Such high-risk AI systems include, in particular, critical infrastructure - e.g. water, gas and electricity, access to education and financial resources or areas of public administration such as border control management and justice. If AI systems are associated with unacceptable risks, particularly for fundamental rights, they should be banned (e.g. systems for recognising emotions at the workplace). Specific transparency risks are also addressed, in particular by requiring the use of AI to be disclosed (e.g. use of chatbots, emotion recognition applications, labelling of AI-generated content).
Adjustment of the standard contractual clauses
Parallel to implementing the AI Act, the Commission has published an updated draft of standard contractual clauses. These are intended to simplify the procurement of AI systems for public contracting authorities and, in particular, ease compliance/comply with the regulatory requirements under the AI Act. Unsurprisingly, the risk-based approach to AI regulation is reflected here. There is a standard set of clauses for high-risk AI systems, which map the requirements for high-risk AI systems in accordance with the AI Act. The standard clauses for non-risk AI systems provide a "light version" for the procurement of AI systems with minimal risk.
The use of AI in the public sector
The use of AI by government entities may increase their efficiency. In the long term, leaner and more efficient government structures can be achieved this way. The associated potential can hardly be overestimated, especially in times of tight national budgets and impending waves of retirements.
Although government cooperation with AI developers and the use of AI in the public sector are still "uncharted territory", there are already some initial successful examples in Germany. For instance, the AI start-up Aleph Alpha has developed the AI-based text assistant "F13" in collaboration with the regional government of Baden-Württemberg via a platform known as "InnoLab_bw". The aim is to relieve employees of the regional administration in their daily text work. Aleph Alpha also recently concluded a framework agreement with the Bavarian state government for the joint development of administrative AI systems.
General requirements for the public procurement of AI systems
The procurement of goods or services in the public sector must always comply with public procurement law. The procurement of AI in particular poses a number of challenges:
Transferring the complex legal framework for AI into the tender specifications
When procuring AI, public contracting authorities should pass on all current and future legal obligations to potential tenderers. Otherwise, there is a risk of purchasing a service that cannot be used in a legally compliant manner in the future. With regard to the requirements of the AI Act, the Commission's standard contractual clauses on the procurement of AI systems should provide relief. These can be used by public contracting authorities to fulfil the requirements of the AI Act. However, these standard contractual clauses do not cover the requirements of other areas of law such as the GDPR and do not comprehensively regulate the service relationship between public contracting authorities and contractors. In Germany, the standard contractual clauses of the various EVB-IT contracts will continue to play an important role. These are mandatory for many public clients and must be harmonised with the Commission's standard contractual clauses and the special requirements of AI in individual cases.
Risk of cementing dominant market positions
A challenge that has not been solved yet is posed by the way AI works: the pioneering positions of individual companies can be permanently cemented by awarding public contracts. After all, AI systems learn independently from the data made available to them and potentially become better, more accurate and more efficient through such training - this is precisely where the enormous innovation potential of AI systems lies. The problem is that the successful tenderer’s AI is trained and potentially improves during the implementation and provision of the service. Competing companies without access to the data stemming from the performance and execution of the contract cannot train their AI systems accordingly. In a new procurement procedure, it seems very likely that the AI trained in this way has an advantage over its competitors. Thus, there is a risk that the state will "breed" a company with a market-leading position by awarding the contract. As a result, a situation comparable to the so-called "Vendor Lock-In" would be created, which otherwise describes situations in the IT sector in which competition is narrowed down to just one provider due to closed APIs or other barriers for compatibility with other systems. In such situations, it is still possible to award a contract in accordance with procurement law - often even in a negotiated procedure without prior publication with the respective provider alone. However, this contradicts the competition for the most economically advantageous tender intended by public procurement law, which ultimately leads to more expensive and lower quality services.
Choosing the right type of procurement procedure
When procuring AI systems, public contracting authorities must also first think about a suitable type of procedure (see also Sanchez-Graell's How to Crack a Nut, 24 February 2024). As with other procurement procedures for technically or legally complex IT services, a negotiated procedure with a call for competition is generally a good option for the acquisition of AI. Unlike in a so-called open procedure, negotiations with tenderers on the content of the tender are not only not prohibited but are an explicitly desired part of the procedure. In this way, tenderers and public contracting authorities alike are given the opportunity to harmonise their mutual technical and legal requirements. Due to the different functionalities of individual AI systems and the special features of the data sets used in each case, it will also be possible to find unique technical features in the AI sector which, in individual cases, may also enable a negotiated procedure without prior publication with one provider alone.
The innovation partnership procurement procedure, which aims to develop an innovative supply or service and then acquire it, also appears to be worth considering for the procurement of AI in a competitive process. The prerequisite for an innovation partnership is that the underlying procurement requirement cannot (yet) be covered by tenderers available on the market. This is often the case with AI systems for use in public administration, which need to be trained with the relevant administrative data. Joint development of the service to be procured can therefore bring significant added value for both sides, especially because the use of AI in the public sector is still largely unexplored territory. In the future, the procurement of AI systems may therefore lead to the innovation partnership procedure, which has rarely been used in practice to date and thus becoming more relevant. The innovation partnership, which is awarded in a competitive process, consists of a two-stage system with an initial research and development phase, followed by the service phase in a second stage.
Challenges in the preparation of the tender specifications
In particular, defining the subject matter of the service is likely to pose considerable challenges. Firstly, the question arises as to whether the desired AI system can be described clearly and comprehensively. The use of "functional tender specifications" provides additional flexibility here. Instead of a catalogue of specific features or services, tenderers are given the task to be performed. In this way, tenderers are encouraged to participate in the search for the best solutions in technical, economic and creative terms. In order to determine the most economical tender among the solution approaches, tests could be used to compare the different AI systems with each other.
In addition the data generated during the use of AI systems and their future usability, usually play a decisive role in determining the object of the service. Any IP rights to data sets used or generated as part of the service provision can result in possible restrictions for subsequent competitive award procedures. Any future dependencies can be anticipated and ideally avoided by means of an intelligently designed specification of services.
Limiting tenderers/applicants to skilled and efficient companies for which there are no grounds for exclusion
When using AI systems that come into contact with government data, public contracting authorities also have a considerable interest in solely working with skilled and efficient companies. Ensuring this through suitability requirements may also involve certification by recognised bodies such as the German Federal Office for Information Security, in addition to any reference requirements. However, currently there are rather few established standards for AI, so it remains to be seen which generally recognised certifications and standards will emerge, especially in the area of public procurement.
Finally, intellectual property rights could also become more relevant, as current discussions surrounding copyright infringements in the AI sector might become the subject of exclusion grounds under public procurement law via the gateway of "serious professional misconduct".
BLOMSTEIN will continue to monitor and assess the developments in public procurement law in context of AI and keep you informed. If you have any questions on the topic, Pascal Friton, Jasmin Mayerl, Ines Horn and Moritz Schuchert will be happy to assist you.