Artificial intelligence is already being used in South Australia’s mental health sector, with a counsellor warning that chatbots are being used to help practitioners during sessions and as an initial contact point on helplines.
The concerns have emerged as South Australia prepares for a royal commission examining the risks and opportunities presented by the rapidly advancing technology.
The counsellor, who has worked for more than one organisation over the past year, said AI was initially being used to identify missed appointments in patient databases and help ensure screenings were completed.
She said the systems could not recognise when a patient had particular needs that should influence how they were approached or contacted.
But she said the growing use of chatbots was of even greater concern, claiming some practitioners had begun relying on them for prompts during counselling sessions.
“Anecdotally, I have spoken to people who say that when they get stuck they have started using chatbots to provide prompts for what they should say during sessions,” she said.
“That is really poor practice.”
The counsellor said AI was also being used to take notes during appointments. While the technology could save time, she argued it risked removing an important part of the human interaction between counsellors and the people they support.
“It is being regularly used too for note-taking during sessions, which while timesaving, takes away benefits which come from human note-taking to develop stronger understanding of the people you are working with,” she said.
“Counselling and mental health support is such a human-orientated process, when you take the human away from it, what’s left?”
She said she had also heard of AI-assisted chatbots being used as the first point of contact for people calling helplines.
“Until there is AI that is capable of picking up on human telltale signs- things like tone of voice, breathing speed – this just isn’t going to work,” she said.
“What concerns me just as much is when AI is being given access to the kind of sensitive data we deal with, what then happens to it?”
Workers raise concerns about AI in the workplace
The counsellor is a member of the Australian Services Union, which represents workers in industries including airlines, local government and community services.
The union has surveyed its members ahead of the royal commission, finding that many welcomed AI’s potential to deal with repetitive work but feared it could ultimately replace staff rather than support them.
Almost two-thirds of respondents, or 64 per cent, said they wanted the technology used for repetitive tasks. Six in 10 said their workplace was already using AI in some form, while 46 per cent identified ethical issues as their biggest concern.
Ella Waters, the union’s South Australian branch secretary, said staff were being asked to work with the technology without always receiving suitable training.
“What we are hearing from our members is that they are positive about it being used to taker away some of the more menial, repetitive tasks, but they are extremely concerned about whether if will lead to workers being replaced,” Ms Waters said.
“Lots of our members also have concerns about data security.”
She said some workers did not fully understand how AI was being used in their workplaces and had not been trained in its operation, raising ethical concerns.
Ms Waters rejected the idea that AI was affecting only white-collar jobs, pointing to recent announcements by Qantas and Bunnings about job cuts linked to partnerships with offshore technology companies.
“There are absolutely opportunities for both workers and businesses to benefit from new technologies like AI, but it needs to be done in a way that benefits both,” she said.
Royal commission to examine risks and opportunities
The royal commission, announced in August, is expected to cost about 3 million Australian dollars and will examine how AI is being introduced into everyday working life.
Former judge Iain Ross was announced as its lead commissioner last month. The inquiry is due to report its findings no later than July 1 next year.
South Australian Premier Peter Malinauskas said the commission would build on work already carried out in the state, which led to the introduction of the country’s social media ban.
“I want South Australia to be a leader in the policy to respond to this technology in a way that puts people’s interests first,” he said.
“The ongoing expansion of AI presents the world’s most significant technological shift since the industrial revolution.”
Mr Malinauskas said South Australia could not prevent AI from being deployed but could help shape how it was used, describing the disruption as presenting significant risks and opportunities for the state.
Ms Waters said the Australian Services Union would prepare submissions to the inquiry. It wants employers to receive clear guidance on training workers to use AI and retraining staff where roles are replaced, while ensuring the technology protects workers’ interests.
