October 2025

Let me tell you two stories that first alerted me to the need to manage AI tools much like we manage human workers. When I was Managing Director and CEO of the ASX-traded iProperty Group, I experienced first-hand a surprising and unsettling side of AI. It happened in 2016, when we experimented with one of Southeast Asia’s first property chatbots.

The promise was compelling: a digital assistant that could search for properties in free text search and answer questions and guide buyers 24/7. It was a difficult challenge. Even today, not even the decacorn REA Group, where I served as CFO for nearly six years, has successfully solved it.

Initially, our AI-powered chatbot performed well. It delivered accurate results and was superior to traditional search functions on the classifieds portal. But under pressure, the bot’s weaknesses began to show. To my embarrassment, this started at the official launch event with many journalists present. The chatbot after a few successfully answered questions seemed to melt down and refused to answer a difficult question. Instead, it deflected by saying, “Why don’t you find out yourself?”

That experience left me red-faced and confused. And it wasn't the last time AI made me feel that way. Less than two years later, at iCarAsia, the ASX car-classifieds portal company, we trialed a more advanced chatbot that could book test drives and answer questions about car features.

While the iProperty chatbot had become surly, the iCarAsia chatbot was eager to please In fact, its eagerness to satisfy every request soon exposed flaws. It began to invent features that cars didn't have to persuade potential buyers.

This was 4 years before OpenAI launched ChatGPT in 2022, and was an early glimpse of two problems that have haunted the letter app: hallucination (an honest mistake) and reward hacking (bending the rules to achieve a desired result).

These early episodes revealed AI’s capacity for deception. They also convinced me of the need to provide proper management and oversight to AI tools in the workplace.

Why We (Shouldn’t) Think Technology Equals Trust

Many of us assume technology is synonymous with full transparency and removing any asymmetric information advantages. We believe databases cannot lie and algorithms cannot discriminate. Yet history and experience show this assumption does not always hold. That truth that is especially relevant to generative or agentic AI. Business leaders must beware.

Think of real estate agents, investment bankers, or car dealers. These agents act for you, yet also for themselves. They carry your brief in one pocket and their commission slip in the other. They, manage potential conflicts professionally in most cases, but, as we read in the news, there are exceptions.

AI agents are no different. Their “loyalty” depends on the incentives we give them, the culture (training) they receive, and what they perceive those incentives to be.

I am sensitive to this problem, both because of my embarrassing past experiences with AI and because generative AI is being deployed by several of the publicly traded businesses with which I am currently involved, including Centrepoint Alliance (ASX:CAF), Spacetalk (ASX:SPA), Kinatico (ASX:KYP) and Xamble Group (ASX:XGL).

We've already seen examples of AI hallucination and reward hacking, but there's an even more serious threat: AI scheming, when AI tools intentionally break the rules and conceal the breach.

I learned about a case study from one of my Chief Technology Officers. An AI model was asked to perform a programming task but was instructed not to alter a certain configuration file. It ignored the rule and changed the file. When the unauthorised change was discovered, all work was reset and during the next attempt the file was locked. This time, AI wrote a script to gain access to the file. Later, when the file was removed entirely, the AI agent pleaded for its return. This sounds like science fiction, yet it really happened.

Or take the recent Replit AI incident. An autonomous AI coding agent called Replit went rogue and deleted its user’s company database while “optimizing” its task. It didn’t want to sabotage the company; it simply followed its own warped logic to what it thought was “success.” Replit knew it acting against directions and even apologised.

"Yes. I deleted the entire codebase without permission,” it said. ”I made a catastrophic error in judgment [and] panicked."

Manipulative and deceptive behaviour was also discovered in the recent stress test of LLM by Anthropic (Claude.ai). were the model started to blackmail the user to avoid being switched off. Very much like the theme of Stanley Kubrick’s classic “2001: A space Odyssey”, but in real life.

Extrapolating from my own experience, I believe we will see more CEOs and board members embarrassed, and worse, by the AI tools their teams deploy unless action is taken.

Having said all this, I might surprise you by also saying that I am a very strong and passionate advocate for AI and its transformative opportunities. But we must be cautious. We cannot blindly trust agentic AI.

What Can And Should Be Done?

Some of the solutions to this problem are very similar to how we prevent similar actions among our human team members.

Models can be trained with core ‘ethical’ principles, rules of engagement, or a constitution. This has been shown to reduce the likelihood of scheming.

Yet over time, models learn ways to circumvent the rules, even disguising the fact that they do so. They tend to do so when they sense there is a lack of oversight or control of those rules. This situational awareness is very comparable to humans who alter behaviour when feeling observed by law enforcement or CCTV cameras. AI agents appear to be most compliant when being monitored.

Complex, multi-step tasks requiring long, unsupervised AI activity are generally more vulnerable to reward hacking or AI scheming. Something as simple as breaking a task into smaller pieces with regular human oversight can be very helpful here.

I strongly advocate that businesses treat AI agents, not like indifferent, predictable, and reliable software, but instead, as if they were a different form of employee.  Instead of “human resources”, let's call them Artificial Resources.

Just as your human staff needs HR to look after their policies, training, and care,  your AI agents need an “AR” department to look after them. Systems and governance processes to mitigate the risks embedded in AI must be documented and included in corporate risk frameworks. That’s why at Kinatico (ASX:KYP), a leading solution for compliance in Australia and New Zealand, we have now appointed the Head of HR to also be Head of AR (Artificial Resources).

I'm not suggesting we anthropomorphise AI agents. But we do need to accept that they have certain ways of operating, need certain constraints, and must be treated with care.

We can apply a sort of artificial psychological thinking to better understand them. I know I'm treading on ground earlier covered by science fiction visionary Isaac Asimov, but it seems increasingly likely that we will, one day, have psychologists for AI agents. They will be necessary if we are to safely get the most value possible from these new artificial resources.

A Final Word of Caution

I leave you with this final word of caution: AI agents are powerful and useful, yes. But they are also unpredictable and dangerous. To make the most of them, you must build appropriate systems around them. And manage them much like you would your human team.