Evaluating Hotel Guest Comments: A Comparison of AI Assessments for Generating Testimonial Pages
Introduction
One of the features of our guest feedback platform is an AI curated Guest Testimonial Feed that clients can use on their websites. It is a real-time, automatically generated feed from guests who opt-in from their guest experience survey.
The basics of the AI curation process is this: once a guest has completed their survey, our algorithm evaluates their answers to determine whether they have had a positive experience. If so, they're asked whether they'd like to share their comments publicly, and the algorithm analyzes those comments for sentiment analysis, as well.
We've been using this technology for about a decade and we thought it would be useful and interesting to see how our algorithm stacks up against the latest and greatest AI and NLP.
The Exercise
We selected 3 guest comments that passed our AI curation process, but only marginally, and therefore would just pass the threshold required to make it to a hotel's testimonial feed1. We then asked ChatGPT to evaluate the comments and gave it an additional piece of information2 that our algorithm uses; how the guest felt their experience met with their expectations3. We then asked ChatGPT to write up, in blog post format, its evaluation of how it assessed the guest comments compared to our algorithm and the idea of using this curation process to generate testimonial pages for hotel websites.
Here's ChatGPT's blog post (scroll below it to see our thoughts):
Introduction
In this blog post, we'll compare how two AI language models assessed the sentiment of three guest comments provided by a hotel, with the goal of evaluating the effectiveness of using AI-based sentiment analysis to generate testimonial pages.
Body
Comment 1: "General Comments: The room and facility was generally clean and safe. Friendly and helpful staff. Guestroom Comments: The TV did not have options and the heater / AC was too loud Staff Comments: The lady who cleaned the room before our stay did a great job. The room was very clean with extra towels as requested. And the whole front desk staff was very friendly and helpful."
GuestInsight's assessment: Marginally positive
My AI assessment: Marginally positive
Guest Expectation: Met
Effectiveness for generating testimonial pages: Effective
Comment 2: "General Comments: Well the hotel did meet my expectations it was clean, we got to move into a two bed room which was much more spacious. I didn't like the fact that there were two shuttle vans in the parking lot but we could not get a ride to the hospital. What is that about? If you can't provide a shuttle service why do you have the vans parked right in front. Very odd! Guestroom Comments: The one bedroom is very small for 2 people (husband and wife), you can barely walk around. It really is for 1 person. It's just not worth the money. We did upgrade to a 2 bed room and that was more spacious for us. I wish you had that size room with a king bed. That would be wonderful and I would not mind paying the price. Staff Comments: I don't remember the gentleman's name who checked us in. He was really really nice, personable and knowledgeable regarding the surrounding area, where we could go have dinner etc."
GuestInsight's assessment: Marginally positive
My AI assessment: Neutral to slightly negative
Guest Expectation: Met
Effectiveness for generating testimonial pages: Not effective
Comment 3: "General Comments: none
Guestroom Comments: I notified the front desk that the lightbulbs in the night stand lamp were white versus warm. They immediately addressed my input. Lighting makes all the difference, and I was told it should be warm light.
Staff Comments: Both on check in and check out the reception staff were very warm and friendly!!! I don't remember all their names, but I checked in on March xx around 4:30pm and checked out on March xx at 10:30am. I want to say thank you to all 4 of them. The attitude and welcoming of the staff make all the difference."
GuestInsight's assessment: Marginally positive
My AI assessment: Positive
Guest Expectation: Met
Effectiveness for generating testimonial pages: Effective
Conclusion:
Based on the comparison of AI assessments for the three guest comments, it's clear that there can be variations in sentiment analysis depending on the model used and the specific comments being evaluated. While both GuestInsight's algorithm and my AI model assessed the first and third comments as marginally positive, my AI model was more critical of the second comment, which had mixed feedback.
One important factor that can affect the accuracy of AI sentiment analysis is the context in which the comments are made. In this case, the fact that all three guests reported their expectations as being met likely had a positive impact on the overall sentiment of their comments. However, even with this information, there was still variability in the AI assessments.
Overall, AI-based sentiment analysis can be a useful tool for hotels to automate and scale their guest feedback analysis and generate AI-curated testimonial pages for their websites. However, it's important to evaluate the accuracy and effectiveness of the model used in the specific context of your property and guest feedback.
--
Our Thoughts:
For us, the main goal of this exercise was to see how our model stacks up against ChatGPT when there are mixed signals in guest comments; we are happy with how it played out. Two things to note:
- Separately, we did pass some guest comments more clearly ranked positively by our algorithm to ChatGPT. It agreed with our algorithm's assessment in every case.
- As mentioned in the notes below, the exercise wasn't on completely level ground because of other survey answer data that factors into our algorithm. This other survey data was what nudged the one comment where we disagreed with ChatGPT more positive.
Over the years, our AI curated testimonial feed has proven to be a useful marketing tool for our clients. We will continue to adapt it to incorporate the newest AI technologies. That said, we'll keep our human curation tools in place for now- we feel that sometimes a human needs to sort out the mixed signals that another human sends through written comments.
We'll dive into this in much more detail in an upcoming podcast episode. If you're interested in seeing some of the other things we've done with the incredible new AI tools, check out some of our other recent blog posts.