How do you feel? Well, if you’re not sure, the work that’s being done at nuom can tell you. To help you to become more conscious of your emotional state through the daily challenges you face.
This is the current development work of Gishan, our ML expert, who has a PhD from the University of Cambridge in Machine Learning, and who has been a part of the nuom journey for the last seven years.
The Research & Development (R&D) being carried out is for Talk It Out, as part of an Innovate UK Grant for their SaaS platform. Talk It Out is an app to boost wellbeing, positivity, and productivity, and to unlock creativity too. We’ve worked with this client right from the start, helping them to digitise their offline business by developing and launching their app, which allows you to record your thoughts, in order to analyse and process ideas and emotions objectively.
Detecting a shift in emotion
The next step for the app will be to add a feature to suggest the emotions the user is feeling when they record their voice note, and in particular for this current work, any shifts in emotion exhibited by the user as they speak.
The feature will detect the mood(s) of the user with the aim to then make practical suggestions on what they can then do (from that emotional standpoint) to feel better and/or boost their productivity.
This can be for both negative and positive emotions, as a tool for users to monitor their emotional state, to be mindful of it, and to receive proactive suggestions on how they can be their best selves at any time.
Everyone speaks differently
However, analysing and identifying emotions precisely through a new and novel voice recording isn’t easy. People speak differently, depending on age, gender, character, ethnicity, and they speak with different intonations, pitch, and variations. And beyond this, the aim is to identify shifts in emotion, which can also be extremely complex.
The app currently uses a rule-based engine (if-then-else statements), which is useful, but this type of engine is unable to deal with the complexity of this particular requirement. It could be used for one user, or a set of users that talk in the same way, but cannot be used for a wide and varied audience of users who all have different speech characteristics.
This issue is known mathematically as the combinatorial explosion problem - in this case the rapid growth of complexity of different users' voice-types, that cannot be covered by a rule base for every possible situation.
As such, machine learning is a powerful tool to navigate this complexity, by learning and responding to the speech patterns of each individual new user. Thereby offering accurate sentiment analysis relatively quickly, and thus enabling functionality that otherwise wouldn’t be possible.
Improving accuracy
Machine learning can quickly learn from each user, however the key challenge is the need for the feature to be sufficiently accurate the first time a new user uses it.
An initial fast-paced hackathon model claimed 65% accuracy, but this was based on using the same sample of users. Gishan and the team then got to work to carry out a hackathon model assessment to show that using the same people for training the model and testing the model meant it overperformed.
When the model was tested on a sample of 10 new users, as a real-world scenario, the accuracy dropped to 45%, which wasn’t enough for an initial start point for a user.
Models were then developed to improve the accuracy: kNN, Linear SVM, RBF SVM, Gaussian Process, Decision Tree, Random Forest, Neural Net, and Naive Bayes.
These different algorithms were tested, and the mode that produced the best baseline performance was the Neural Network model, producing 60% accuracy, which created a good baseline for accuracy without any performance enhancing techniques.
Hybrid modelling solved the problem
The aim was then to improve on this level of accuracy. Using performance enhancing techniques such as data augmentation and boosting didn’t make any significant improvements to the baseline results. However, using a hybrid modelling approach (using multiple algorithms together to complement and augment each other) raised the accuracy to a very promising 68%.
The benefits of the hybrid model was to offer enhancements to the entire modelling pipeline from data preprocessing to model training and predication.
The team also made sure to use a wide variety of users, including different ages (including children and elderly), genders, and ethnicities, in order to stress-test accuracy as much as possible.
The feature is ready to test on real users
The model performed well within the sample, and was able to quickly learn to detect shifts in emotion from each individual user accurately. This model will now be tested on real users to see if there is equivalent accuracy in real-world usage.
Caveats will be added to note “this is a new model, as you use it it will learn and improve”, in order to mitigate any potential initial errors, and facilitate the initial experience. The app will also ask the user if it detected the shift in emotion correctly, which will create a further feedback loop to help the model to learn, and improve accuracy quickly.
We look forward to sharing real-world testing results and progress soon, which will show both initial accuracy and the model’s ability to learn and improve during every use.
We also look forward to user feedback on this feature as it helps them to be more mindful of their day-to-day emotional changes, in order to manage their mood, and be their most productive selves.
Are you looking for machine learning support?
If are interested in using machine learning or AI within your products or services, or want to know more about how we can help across discovery, design, and delivery, then please get in touch at hello@nuom.co.uk
If you would like to know more about how we created the current app, you can find out here: Talk It Out: Untangling thinking with a mental wellbeing startup.
