Reflections on AI Coaching #6 – In which context does AI Coaching work?

We did a number of studies to get a better understanding of how and where Coach Vici (and Vidi) could be used. Coach Vidi was an early attempt at a chatbot coach that could assist coaches by helping clients reflect on their human-coaching sessions. We are incorporating Vidi into the new version of Coach Vici. There are three studies of note.

The first one tried to repeat the longitudinal randomised control trail of before with a group of unemployed youths in South Africa. Did you know that South Africa has an unemployment rate of 18-35 year olds of over 50%! Imagine if Coach Vici could help these people with their goal attainment. We did the study over a period of 2.5 months and unfortunately the time was too short. We found no significant difference between the group that used Vici and the control group, however there was a small tendency for higher goal attainment in the Vici group amount those that used Vici more often. We learnt from this that goal attainment takes time! Two, even three months is too short to measure change. We also learnt that using Telegram as a front-end is not as good as using WhatsApp which is more popular.  I would love to repeat this experiment over a longer timeframe, especially once we have ChatGPT enabled Coach Vici.

The second study looked at the use of Vici to support final year university students as part of a series of human-led coaching workshops they attended. The paper is available here: https://www.tandfonline.com/doi/abs/10.1080/17521882.2022.2094278. We found that students perceived the AI coach as accessible, easy to use, intelligent and quick to respond and that they would use it more if it’s performance improved.

The third study that is currently undergoing peer review looked at the impact an AI coach (Vidi) could have on the human coach-client working alliance. We found that in general coaches were a more sceptical of using an AI coach assistant than clients. Coaches were concerned the chatbot coach would interfere with their relationship with client. Both parties could however see the advantages such as improved goal attainment, accountability and always available accessibility. Interestingly clients found the bot to be psychologically safe and they felt unjudged since it was not a human coach.

As we incorporate ChatGPT into Coach Vici we are being very careful to maintain this need for psychological safety. Generative language models are far from perfect and an inappropriate remark or response from a ChatGPT-based Coach Vici could ruin the relationship. A consistent theme from the qualitative studies we did so far with various groups such as students, coaches, clients and graduate employees all support the idea that AI chatbot coaching can be very useful. However the robotic and repetitive nature of the chatbot conversation is also consistently criticised. The early results from the ChatGPT-based Coach Vici suggests that the natural tone and context awareness of the conversation has improved dramatically! This addresses a major limitation of previous generation AI coaching chatbots. And as new version of ChatGPT are released we expect this to improve even more. Exciting times!