AI Governance, How We Work it Out When AI Fails?

If you work with AI and especially if you are a decision maker, you must read this article.

#AIGovernance #HumanInTheLoop #AIAccountability #ResponsibleAI #AITransformation #ProjectManagement #PMO #ProjectControls #ArtificialIntelligence #AIAssurance #HumanExpertise #BusinessContinuity #DigitalTransformation #XPMOAI

I have to admit that what I wrote about AI governance in my earlier article, “AI-Automated Zalbasir Project Portfolio Management,” was premature.

Over the past two years, I have been involved in designing, planning, developing, implementing and testing our AI-powered Project Management Office platform, XPMOAI.

During the same period, I have also been invited to test and evaluate other AI systems from the perspective of a PMO and project controls expert.

These experiences have changed the way I think about AI.

There is a big difference between watching an AI demonstration and trying to implement AI in a real project management environment.

In a demonstration, AI can produce impressive schedules, reports, dashboards, forecasts and recommendations within minutes.

But when you start implementing these capabilities, you face much more difficult questions.

Where did the data come from? Are the calculations correct? Can the results be reproduced? Does AI understand the contractual requirements? Can we trust its recommendations? And who is responsible when something goes wrong?

Most importantly, how do we verify work that AI completes in minutes but would traditionally take experienced professionals several days?

These are not questions I developed only by reading research papers or government policies. They have emerged from my practical experience of working with AI systems, examining their capabilities, identifying limitations and considering how they can operate within a real PMO.

And the more I work with AI, the more I realise that the biggest governance challenge may not be AI itself.

It may be the humans we expect to control it.

My Original Thinking About AI Governance

When I first wrote about AI governance for XPMOAI, I identified five important governance and ethical safeguards:

1. Immutable audit trails with cryptographic signatures.
2. Bias detection and fairness constraints.
3. Human override rights with documented justification.
4. Third-party model audits and certification.
5. Fallback to rules-based logic if AI fails.

I still believe these safeguards are important.

However, after two years of practical work, I have started questioning whether some of them are sufficient in real operating conditions.

Human override sounds straightforward. But can a human actually understand and challenge a complex AI decision quickly enough?

A rules-based fallback sounds reassuring. But can it handle situations requiring professional judgment, particularly when the people who originally understood the process are no longer available?

Third-party audits are valuable. But an audit at one point in time cannot guarantee that a continuously evolving AI system will always produce reliable results.

We may need additional safeguards, including independent AI verification, continuous monitoring, automatic circuit breakers and emergency shutdown mechanisms.

But even these measures raise further questions.

What Are Governments Thinking About AI?

My practical experience encouraged me to study how governments are approaching the same challenges.

I have been reading AI policies, laws and governance frameworks from Australia, Europe, the United States, Japan and South Korea.

Although these countries have different approaches, I was particularly interested in one subject:

What will happen to human involvement and accountability when AI takes over much of the work?

On 21 October 2026, Australia’s Digital Transformation Agency published a minor update to its Policy for the Responsible Use of AI in Government, following the more substantial update that took effect on 15 December 2025.

The continued updates reflect the challenge governments face in keeping their policies aligned with rapidly developing technology.

The policy is available here:

https://www.digital.gov.au/sites/default/files/documents/2026-10/Policy%20for%20the%20responsible%20use%20of%20AI%20in%20government%202_1_1.pdf

Then, on 30 October 2026, the Australian Government announced its new AI Reference Architecture (AIRA).

This provides government agencies with a common foundation for designing, implementing, procuring and governing AI systems.

It covers data, AI models, infrastructure, security, trustworthiness, assurance and human oversight.

The architecture is available here:

https://www.digital.gov.au/sites/default/files/documents/2026-10/AI-Architecture-ACCESSIBLE.pdf

What interested me most was how the Australian Government is trying to balance two objectives.

It wants to accelerate AI adoption, improve efficiency and deliver better services.

At the same time, it recognises the importance of maintaining human control, accountability and public trust.

This is exactly the balance we have been considering while developing XPMOAI.

We want AI to perform complex project controls activities quickly and accurately.

But we also need clients to trust the results, understand the limitations and remain in control of consequential decisions.

The Good News: Humans Are Still Needed

The Australian AI Reference Architecture places considerable emphasis on Human Agency, Oversight and Control (HAOC).

Humans remain important in defining operating boundaries, supervising AI behaviour, reviewing significant decisions and intervening when necessary.

The architecture also recognises that humans do not need to approve every AI action. Automated verification, anomaly detection, circuit breakers and other safeguards can operate within authorised limits.

This is a sensible approach.

But it immediately reminded me of a problem I have encountered when evaluating AI capabilities.

AI can perform the technical work much faster than a human can verify it.

And this creates a difficult question for every organization considering AI adoption.

If we still need the same people to check everything AI produces, where are the savings?

The Problem I Keep Encountering in AI-Powered Project Controls

Consider a practical example.

An AI system generates a complete Primavera P6 baseline schedule containing 5,000 activities, including logic relationships, calendars, resources, productivity assumptions and contractual milestones.

The AI completes the work in one hour.

That is impressive.

But now the Planning Manager must verify and approve it.

Can an experienced Planning Manager genuinely check 5,000 activities, their relationships, calendars, assumptions and contractual compliance in another hour?

Of course not.

Proper verification may require several days.

And if the manager simply accepts the output because the AI appears confident, what has human oversight actually achieved?

This is not limited to scheduling.

The same problem appears when AI generates cost forecasts, risk assessments, delay analyses, portfolio recommendations or tender evaluations.

AI may produce a sophisticated answer in minutes.

But the real challenge is establishing whether the answer is correct, complete and based on reliable information.

In my work on XPMOAI, one of the important lessons has been the need to distinguish between producing an answer and proving that answer.

A professional dashboard may look impressive. An AI-generated recommendation may sound convincing.

But neither appearance nor confidence is evidence of correctness.

We need to know where the information originated, which assumptions were used, how calculations were performed and whether the result can be independently reproduced.

This is why traceability, deterministic calculations, validation and independent assurance are so important.

And it is also why simply adding a human approval button does not solve the problem.

The Economic Contradiction

Let us imagine a company employing ten planning engineers.

After implementing AI, the system can perform much of their routine scheduling, forecasting, reporting and analysis.

The company expects significant productivity improvements and lower operating costs.

But if the same ten engineers must remain to verify everything AI produces, the company is now paying for both the AI system and the original workforce.

It also needs infrastructure, cybersecurity, maintenance, governance and assurance.

Where is the business benefit?

AI may still create value through faster decisions, better quality and improved project outcomes. But those benefits must be measured against the full cost of operating and controlling the system.

We cannot simply assume that faster AI processing automatically means lower total costs.

We want AI to reduce human effort, but we also want humans to remain responsible for controlling AI.

Both objectives are reasonable.

Achieving them together is much harder than it sounds.

What Does the Australian Architecture Propose?

The Australian AI Reference Architecture provides a detailed structure for designing AI systems.

It includes capabilities covering data, intelligence and analytics, application orchestration, infrastructure, communication, trustworthiness and assurance.

It also identifies a separate domain for Human Agency, Oversight and Control.

Figure 2— Detailed view of the Australian AI Reference Architecture

Within Human Agency, Oversight and Control, the architecture identifies two important capability groups.

Decision Support includes alerting, anomaly highlighting and escalation routing.

Figure 2 — (Extract of ) Human Agency, Oversight and Control components

These are important architectural capabilities.

I believe Australia has provided a useful foundation for organizations planning to implement AI.

The guidance also recognises risks such as automation bias, reduced human attention and skill erosion.

However, my practical experience has taught me to distinguish between defining an architecture and proving that it works.

A control shown in an architecture diagram is not necessarily an effective control in real operations.

The actual implementation must demonstrate that the control can detect problems, respond appropriately and protect the organization under realistic operating conditions.

And this leads to what I now consider an even more important question.

What Happens Five Years After AI Implementation?

Imagine a PMO in 2031.

For five years, AI has been preparing schedules, calculating forecasts, analysing delays, evaluating risks and generating reports.

The organization has become more efficient.

It has reduced its workforce because AI is doing much of the work.

Some senior engineers have retired. Others have moved into different roles.

Junior engineers have not gained the same practical experience because AI has been performing much of the technical work.

Then one day, a serious AI failure occurs.

The system begins producing unreliable forecasts or incorrect recommendations.

The monitoring system detects the problem.

The circuit breaker activates.

The kill switch works perfectly.

Excellent. We have successfully stopped AI.

But what happens the following morning?

Who will prepare the schedules?

Who will calculate the forecasts?

Who will understand the original assumptions?

Who will know why certain logic relationships were established?

Who will identify which historical AI decisions were correct and which were not?

And who will have enough practical experience to continue operating the PMO without AI?

This is not an argument that AI will inevitably destroy professional expertise.

It is a warning about what may happen if organizations reduce human involvement without deliberately preserving the capabilities they still depend on.

Stopping AI is not the same as being able to continue without AI.

A Kill Switch Cannot Restore Lost Experience

My original governance proposal included human override rights and fallback to rules-based logic.

I still support both.

But I now see that they address only part of the problem.

A kill switch may stop an unsafe AI process.

A rules-based fallback may continue certain calculations or predefined operations.

But neither can automatically restore years of lost engineering experience.

What happens when a situation requires interpreting contractual requirements, evaluating unusual construction risks or understanding the assumptions behind a project schedule?

What happens when the people who developed those assumptions are no longer available?

A fallback system cannot replace professional judgment in every situation.

And a human override button is of limited value if the person responsible for taking control no longer understands the operation.

We need to distinguish between three capabilities.

Shutdown: The ability to stop AI safely.

Fallback: The ability to transfer operations to another validated system or process.

Recovery: The ability to continue essential operations using qualified people, reliable information and alternative procedures.

The first two can be supported by technical architecture.

The third also requires maintaining organizational knowledge and professional competence.

That is where long-term AI governance becomes much more complicated.

How Much Human Capability Must We Preserve?

After two years of working with AI-powered PMO systems, this is the question I believe every organization should be asking:

What is the minimum sustainable human capability required to govern, challenge and recover an AI-dependent operation, and what does maintaining that capability cost?

This does not necessarily mean retaining the entire original workforce.

It means identifying the knowledge, experience and practical skills that the organization cannot afford to lose.

For a PMO, these may include independently developing and validating project schedules, analysing delays, assessing cost forecasts, interpreting contractual obligations and managing critical project decisions.

Organizations may need to preserve these capabilities through practical training, retained expertise, periodic exercises and tested recovery procedures.

For example, an AI-dependent PMO could periodically simulate the unavailability of its AI systems and test whether its team can continue essential project controls activities.

Such a test would reveal whether human oversight is still operationally meaningful.

But all these measures cost money.

And those costs must be included in the AI business case.

There is another issue that concerns me.

If junior engineers stop performing the routine technical work through which they traditionally develop their expertise, where will the next generation of senior engineers come from?

We may solve today’s productivity problem while creating tomorrow’s shortage of professional expertise.

Can AI Independently Verify AI?

My experience testing AI systems has also encouraged me to consider whether AI itself can become part of the assurance solution.

One possible approach is to use several independent AI systems to analyse the same information.

For example, three AI systems could independently assess a project schedule, cost forecast or tender submission.

Three additional AI reviewers could then challenge the recommendations, examine supporting evidence and identify inconsistencies.

The human decision-maker would receive a consolidated assessment highlighting disagreements, unresolved issues, assumptions and significant risks.

Instead of repeating all the analysis, the human could focus on exceptions and decisions requiring professional judgment.

This could potentially reduce the verification workload.

But there is a serious limitation.

If three AI systems agree, does that mean they are correct?

Not necessarily.

They may all rely on the same incorrect data, misunderstand the same contractual requirement or share similar weaknesses.

Six AI systems can agree on the same mistake.

Therefore, independent AI assurance must include reliable source data, objective calculations, different verification methods and rigorous testing.

This is an approach worth investigating, not a proven replacement for human oversight.

The objective should be to develop an assurance architecture that reduces unnecessary human workload without compromising decision quality or introducing excessive operating costs.

Three Research Papers That Changed My Thinking

While working through these practical challenges, I also started examining research into automation, human judgment and accountability.

What surprised me was discovering that researchers had already identified many of these problems decades ago.

These three research papers were published in 1983, 1999 and 2019—long before ChatGPT brought generative AI into widespread public use in November 2022. Although they examined earlier forms of automation and human–machine interaction, their findings are remarkably relevant to today’s AI governance challenges.

1. Ironies of Automation — Lisanne Bainbridge (1983)

Bainbridge identified a fundamental contradiction in automation.

As machines take over routine work, humans become less involved in everyday operations.

But when automation fails, those same humans are expected to intervene and solve the most difficult problems.

Over time, reduced practice can weaken the skills and awareness needed to respond effectively.

This research closely reflects my concern about the future of AI-powered PMOs.

If AI performs most of the planning, forecasting and analysis for five years, how will our engineers maintain the expertise required to challenge it?

The more work we automate, the more important it becomes to preserve the human capabilities needed when automation fails.

2. Accounting for the Effects of Accountability — Jennifer Lerner and Philip Tetlock (1999)

This research challenges the assumption that making someone accountable automatically improves their decisions.

Accountability can encourage careful analysis.

But depending on how it is structured, it can also encourage conformity, defensive reasoning and pressure to justify decisions.

Imagine a PMO Director receiving an AI-generated recommendation that senior management expects to be approved immediately.

The director may have formal approval authority.

But does that mean the director has enough time, information and independence to challenge the recommendation?

A signature does not necessarily prove that meaningful verification has taken place.

Accountability without the practical ability to exercise independent judgment can become a weakness rather than a safeguard.

3. Moral Crumple Zones — Madeleine Clare Elish (2019)

This research raises perhaps the most uncomfortable question.

Elish explains how human operators can absorb disproportionate blame when complex automated systems fail, even when their ability to control those systems is limited.

Imagine an AI-generated forecast for a multibillion-dollar infrastructure programme.

The PMO Director approves it.

Six months later, the forecast proves seriously wrong.

The investigation starts with one question:

Who approved it?

But what about the people who developed the AI, supplied the data, tested the model, defined the operating limits and decided how much time the director had for verification?

If we investigate only the final human approval, we may miss the real causes of failure.

Human accountability must be connected to actual authority, information, competence and control.

What Two Years of Working With AI Have Taught Me

Two years ago, I was mainly thinking about what AI could do for project management.

How could we automate reporting?

How could we improve forecasting?

How could we connect project data, identify risks earlier and provide better recommendations to management?

I am still working toward those objectives through XPMOAI.

But designing, implementing and testing AI systems has changed the questions I ask.

Today, I am equally concerned about how we prove the results, how we maintain accountability, how we preserve essential human expertise and how we recover when AI fails.

Reading the Australian AI Reference Architecture and international governance guidance has helped me put these practical concerns into a wider context.

I do not believe the Australian framework is wrong. On the contrary, it provides a valuable foundation.

But an architecture diagram, a governance policy or a human approval requirement cannot by itself guarantee meaningful control.

The real test is what happens during implementation, after years of operation and when something goes seriously wrong.

We need AI systems that can be independently verified, safely interrupted and effectively governed.

We need organizations that preserve the professional expertise required to challenge AI and recover essential operations.

And we need business cases that include the real cost of maintaining those capabilities.

The question is no longer simply whether we should keep a human in the loop.

The question is how we can keep humans meaningfully in control without losing the economic benefits that made us adopt AI in the first place.

Because in the end, a working kill switch may stop the AI.

But it will not bring back five years of lost human experience.

Good Luck for you all.