We need to measure progress in good AI, says Partnership on AI CEO Rebecca Finlay

Governments and other stakeholders need to agree on “common rules of the road” for artificial intelligence, but that doesn’t have to come at the cost of innovation, says the boss of the non-profit advocating for responsible AI, at last week’s UN gathering in Geneva.

The United Nations first Global Dialogue on AI Governance is done and dusted. The 4,000-plus participants – government, big tech, research and civil society representatives – who attended the conference in Geneva last week have moved on.

But the real work begins now, and starts with being able to define and measure what “good artificial intelligence” practices looks like, according to Rebecca Finlay, the chief executive of Partnership on AI (PAI). The Canadian non-profit coalition brings together over 150 organisations in 17 countries, including academic institutions, media, and technology companies such as Meta and Salesforce, to ensure developments in AI advance positive outcomes for people and society. Last week, it launched two initiatives to benchmark progress on responsible AI.

Finlay, who co-chaired one of the Dialogue’s four key thematic panels on building safe, secure and trustworthy AI, and has advised on UN initiatives such as its Global Digital Compact, debriefed Geneva Solutions on last week’s outcomes.

Rebecca Finlay: There were three or four that I would highlight. Firstly, we need an open, inclusive and independent evidence base to inform both policymakers and the public, with close coordination between the UN’s International Scientific Panel on AI with other international assessments, for example, the upcoming Global South AI Safety report. Secondly, interoperability – the railroad between different AI models and systems – requires shared frameworks and common standards, grounded in universal principles such as human rights. And the third takeaway I heard loud and clear was the need to understand and measure what progress is actually happening and ensure it serves the public interest.

One of the things I hear a lot in my leadership role is that despite more than a decade of discussion about AI frameworks and principles, there has not yet been a shared public-interest resource showing what responsible AI actually looks like in practice. Many leaders ask: “What does ‘good’ look like, and how can we learn from others?” The progress hub is intended to begin creating that shared resource by outlining what is currently known about good AI practices and by highlighting the insights, processes, and techniques of organisations that are already acting responsibly. Alongside it, the new Measures of Responsible AI Progress report will assess how effectively AI systems are being delivered in ways that work for people, citizens, and communities.

PAI brings together industry, civil society, academia, and policy makers from all regions of the world to focus on exactly the question you identified. That can be challenging to do, because the necessary guardrails and infrastructure are only just being built. In 2023, we issued a set of guidelines for frontier AI developers, covering the entire lifecycle from research and development to deployment, on what it looks like to be responsible. Over the past two years, we’ve been evaluating progress against those goals. One example is transparency around energy use, carbon footprint, and environmental impact reporting. Though reporting practices are now widespread, they are very broad-ranging, highlighting the need for shared metrics that make it possible to understand the actual impacts and compare performance.

In some ways, we’re building the field as we go and trying to draw on what werve learnt from established sectors such as civil aviation, healthcare, and pharmaceuticals and adapting their approaches to AI governance.

Road safety provides a particularly useful analogy. We have common rules of the road, technical standards, and international agreements on safety measures such as seat belts and brakes. It took many years for that level of technical specificity to happen. But these rules, on seat belts for example, have also allowed us to get to where we’re going more safely and faster. The same principle applies to AI. Often, people will talk about safety and innovation as being opposite to each other, but in fact, the two must go hand in hand, especially in those sectors where there can be significant risks.

These are all things we need to attend to. We have to better understand what we’ve seen in the past when sectors have put in place different levels of shared common standards and whether or not they've been sustainable. At PAI, we believe it’s important to have the technologists at the table who are on the front lines developing the technology, so that when we do our reporting, we’re attempting both to measure what the progress has been in an honest, independent way. But alongside that, we speak to some of the challenges of meeting some of those standards as well. For example, transparency is really important in AI development. It started with early work around model cards [a standardised document explaining to the end user how a given AI model works -ed.] and has now moved into all sorts of different levels of disclosure. But you have to be extremely cautious about privacy protections as well. So, we need to try to understand how we move the field forward while also meeting all the intersectionality of the goals and values.

At the end of the day, I just truly believe we have got to do this openly, inclusively, and in the public interest. It's why the UN Global Dialogue is so important, because it can convene a very diverse group of perspectives around a table to navigate, negotiate, and define what good looks like for the world.

I was so intrigued when Guterres, in his opening remarks at the Dialogue, said that in his very first opening address as secretary general, in 2017, he talked about AI. This is evidence of how central this issue was – from his perspective – to his leadership. Over the years, we’ve seen a number of UN initiatives – for example, the high-level advisory panel on AI, created in 2023, and the Global Digital Compact, which I was a part of.

Going forward, we need to continue to invest in these endeavours, as well as other initiatives that Guterres has called for, such as on child safety, better understanding the environmental impacts of AI, and capacity building. At the same time, if we can keep support going for the UN Global Dialogue and the Scientific Panel, those two pieces will create some of the public policy infrastructure we need at the global level to move things forward.

Absolutely. We’ve seen many corporate partners around the table at the sessions that we held last week and engaged in the work that we’re doing. There’s been a rich, multistakeholder conversation.

Geneva Solutions content is licensed under Creative Commons BY 4.0.