Reflections on AI Coaching #5 – Does AI Coaching really work?

After understanding what will drive people to use Coach Vici (performance!) we set out to ask the most important question of them all: does Coach Vici work?  And more generally, how well does an AI Chatbot Coach fare compared to a human coach?

This was a very ambitious undertaking with no certainty guaranteed (as should be the case for any research). I was very fortunate to meet Dr Joanna Molyn at a conference in Switzerland in 2018 were we begun a very productive collaboration that ultimately included Prof Erik de Haan and Viktor Nilssen. We did an experiment on Coach Vici’s efficacy using the gold standard in research designs: longitudinal randomised controlled trail (RCT). We asked one group of users to use Coach Vici for 10 months while a second group did not use the bot. We measured both groups and found that Coach Vici was able to help its users at double the rate of the non-users with their goal attainment. The findings were published here: https://radar.brookes.ac.uk/radar/items/05cbe328-2b58-4a51-8f64-1e70401e28fd/1/

This was a stunning result! Coach Vici worked! And the more often people used the bot, the better their goal attainment was. We were in a very fortunate situation: the year before this same study was done using human coaches where it was shown that people receiving human coaching outperformed those not receiving human coaching. Because of the similar research designs we could now compare Coach Vici with the human coaches. And what did we find? Coach Vici was comparable to a human coach performance in terms of helping people with goal attainment. The paper is available here: https://journals.plos.org/plosone/article/authors?id=10.1371/journal.pone.0270255

I was astonished to put it mildly! My humble attempt at creating a AI chatbot coach was holding its own against human coaches! And keep in mind this is before the availability of generative AI such as ChatGPT which means at the time Coach Vici was a rules-based, scripted chatbot.

This finding set many amazing things into motion, most importantly the creation of a spin-out start-up company called “AI Coaching Pty” and the commercialisation of Coach Vici (coachvici.com). I was invited to do a keynote speech and Columbia University by Prof Terry Maltbia (thanks Terry!) where I met Jonathan Kirchner, CEO of Aiir Consulting and our very first client. We created a custom coaching chatbot called Aiiron for Aiir based on Coach Vici that was launched in Feb 2023.

All of this is a testimony to curiosity and the power for research!

And what about ChatGPT? We are putting the finishing touches on the new Coach Vici that incorporates ChatGPT’s generative AI capabilities. The initial results are truly remarkable! The natural, context aware conversations that Coach Vici are now able to have has the potential to dramatically improve Vici’s efficacy. And I suspect in certain use cases it could even outperform human coaches.  But of course we will have to conduct a properly designed research study to prove it. Watch this space…

In the next post I will reflect on all the user feedback we received over the years on AI Chatbot Coaches and how ChatGPT could address some of the historic challenges we faced.