…or (un)happy enough to take time out of their day to write an online review?

knitr::opts_chunk$set(echo = FALSE, collapse=TRUE)

# load libraries
library(tidyverse)
library(hrbrthemes)
library(kableExtra)
library(here)
library(markdown)
library(glmnet)


# load raw data
reviews <- read_csv(here("data", "reviews_european_airlines_clean.csv"))

# set theme for plotting
theme_set(theme_ipsum_gs())

The short story

The (perceived) quality of human interaction matters more than perks and infrastructure.

Key insights:

  • Humans over perks: Good experience with cabin crew and ground services are the main drivers of satisfaction.
  • Cultural proximity: Airlines get more favourable reviews from domestic than international customers — with some very notable exceptions.
  • Value for money: Customers are happier if they get their money’s worth.

The fascination

With a long-standing fascination for aviation, I was interested in the airline review data sets in the community. However, many analyses seemed to miss the essence of Data Analytics: can we gain deeper insights than descriptive distribution of numbers? Can we ask better questions?

Some analyses felt off: they suggested that a substantial amount of passengers recommended an airline despite giving them the worst rating — “huh, wait, what?”

I’ll therefore focus on these two areas below:

  • Insight quality: Can we do more than descriptive plots and gauge into more subtle patterns?
  • Data quality: The role of a critical eye in the process long before doing the plots and writing up the analysis.

As with any dataset of this type and quality, the insights are limited to and by the nature of the data and results must be treated with a big grain of salt.

Yet, collectively speaking, there is an interesting overall pattern emerging.

The data

Airlinequality.com collects reviews of airline experiences. Passengers rate individual flights on aspects such as cabin service or food, and give an overall recommendation, and can leave a text review.

Khushi Pitroda’s dataset over on Kaggle is a popular set with ~23,000+ reviews. I tweaked their code, which I describe in this Python Notebook.

Note: both airlinequality.com and the original code sadly went offline recently.

My changes & improvements to the existing approach:

  • Scrape all reviews for 40 European airlines (compared to max. 100 reviews for airlines world-wide)
  • Fix a bug that extracted an erroneous overall rating — see Data quality in the Appendix.
  • Extract reviewer location to determine if the customer is from the airline's country (‘domestic') or not (‘international’).

The raw data contains the results from a sentiment analysis using Twitter-roBERTa-base. It contains further extracted features that were too instable or quirky for this first analysis.

All data and cleaning scripts are in my Github repository.

Insights

General

The overall timeline shows expected and more subtle patterns:

  • Seasonal effects: more reviews in summer than winter
  • COVID effects for 2020-2022
  • Overall ratings go down over time

Whether the last observation is due to a higher proportion of critical reviewers (2016-2019), or whether the industry in general has been suffering from an increasingly negative perception from 2018 onwards is impossible to tell from this data.

It is noteworthy though that the lowest overall ratings occur during the COVID period and it’s recovery phase when disruptions to the industry were highest and customer satisfaction at its lowest.

Passenger satisfaction

If we treat the responses to the five main travel aspects as your typical survey response data, a Likert-type presentation will look like this.

Basically, variables that “pull” in both directions are stronger influencing factors.

Three main observations:

  1. The human factors Cabin staff service and Ground service pull strongly in both directions — if airline employees are perceived as good, passengers will recommend the airline (and vice versa).
  2. Infrastructure like Inflight entertainment or Wifi, if perceived as poor, favour ‘not recommended’. But the reverse does not hold: passengers who recommend an airline may not necessarily rate these perks as ‘good’, which suggests a weaker importance of those factors for an overall recommendation.
  3. The aspect Value for money behaves similar to the human aspects: a passenger who does thinks they got their money’s worth will recommend an airline (and vice versa).

For what it’s worth: wifi is generally viewed disfavourably, even in otherwise positive reviews, so either this variable is a candidate for a time-series analysis or the investment in this tech was historically not meeting its prospected target.

Cultural proximity

Do domestic customers respond differently than international ones? They do!

Three main observations:

  1. Smaller airlines fare better than larger ones.
  2. Budget airlines aren’t necessarily rated poorly (Volotea, Jet2, easyJet).
  3. All Swiss airlines deviate from this pattern and get better reviews from international customers.

The results are not statistically significant for half the airlines, which is likely a data amount issue.

However, the pattern that emerges is clear: It’s probably no coincidence that three of the four airlines which are seen more favourably by international customers are Swiss, and that all Swiss airlines fall into this category.

Drivers of recommendations

Of course, the dimensions on which passengers give responses for are individually associated with passenger (dis)satisfaction. The question is whether some dimensions of airline travel are more impactful for positive recommendation than others.

This is where a variable importance analysis comes in handy. It tries to identify the strongest predictors of a passenger recommendation, when controlled for all other factors.

We’ll keep it simple and use only the primary factors from the reviews plus a couple straightforward meta variables. That said, we exclude wifi and travel class, as it’s missing in a lot of the reviews and is also systematically absent from many airlines (e.g. Ryanair, easyJet).

Basically: the longer the bar, the higher the relevance of that factor for a recommendation. Left and right bars distinguish the direction of the effect: to the left favours negative recommendation (and vice versa).

The patterns:

  1. Again, human factors come out on top.
  2. In-cabin travel experience leads to greater satisfaction.
  3. Low-costers are not associated with ‘no recommendation’.
  4. International customers are less likely to recommend the airline.
  5. Leisure travel has happier customers (compared to ‘Business’).
  6. Smaller travel groups are happier — the fewer, the merrier!
  7. Unhappy customers write more in their text review.

In combination, the patterns for ground service, multileg, and international customer suggest that especially passengers who have stuff going wrong during for example a layover have to engage more with ground and customer service, often in another country.

Conclusions & interpretation

While the data is really messy for many methods, there is are clear overall pattern emerging that I found quite insightful. Granted, they may not sound all that surprising, but it’s still interesting to see this clarity in the noise.

Here are my main takeaways:

Humans over perks

  • Cabin & ground (customer) service: A good wifi connection doesn’t fix stuff gone wrong or miscommunication.
  • Cabin experience: How comfortable the inflight experience feels is important, but less impactful.
  • Value for money: Low-costers are not necessarily perceived a bad experience – customers acknowledge that low fares won’t buy comfort. Perhaps efficient, no-frills operations with lowcosters also contribute, as does their point-to-point network without multilegs where stuff can go wrong.

Cultural proximity

  • Shared cultural values: Unwritten rules of how things work – the unspoken stuff.
  • Shared language: Not simply and literally linguistically, but also how people interact with each other. Language is more than exchanging words – there is cultural pragmatics involved too. What one community perceives as normal might be perceived as rude in others.

The exceptions Swiss, Edelweiss & Helvetic have two possible location-specific explanations:

  • Swiss scepticism: Swiss customers expect more from their ‘home’ airlines; conversely, their international customers are happier about a smooth experience with Swiss hospitality. Or maybe it’s the Schoggi.
  • Zurich hub: The airlines are based in Zurich (ZRH), which is frequently ranked as one of the best airports in Europe for passenger satisfaction. ZRH is known for efficient connections and passenger operations, thus international passengers might be more likely to have fewer hickups on multilegs and perceive ground services as smooth.

Travel type

  • Purpose of travel: Many more highly recommended airlines are either small or associated with holiday travels, including low-costers. Relaxed travellers with time are happier – and perceive stuff going wrong as less impactful (to write a review, at least).
  • Travel group: Single and couple leisure travellers are happier than larger groups – for the latter, when stuff goes wrong, it spoils the fun for an entire family.

If that was my contribution to a decision-making process: don’t cut the funding for interpersonal and emphatic communications in the air and on the ground.

Appendix

Data quality

As with any data set, there are general issues with data quality. I’ll go into a bit more depth on (2):

  1. Review data is inherently skewed — happy customers are less likely to spend time to do it than unhappy ones. Distributions of values are hence unsuitable for most statistical methods. That was the motivation to choose to predict the binary outcome ‘recommended’ over a numerical, or ordered nominal ‘overall outcome’ that are typically U-shaped in survey response data.
  2. The original webscrape clipped the trailing “0” from all ratings of 10 — leading to the absurd effect that glowingly positive reviews with ‘yes’ recommendations ended up with the worst overall rating.
  3. Many free-text fields in the data require(d) extensive cleaning. For example, reviewers can provide the aircraft they flew with. This leads to immense variability ranging from ‘flight number’ or ‘A320 / A350’ to ‘Boeing 737’ vs. ‘B737’. It also depends on a customer’s knowledge of aircraft types, and responses range from ‘Boeing’ to detailed model and/or ICAO type codes (e.g., B737-800 or B738).

The first issue is systematic due to the nature of the data; this needs to be kept in mind the choice of method and the interpretation of results. The third issue is the most quirky — but the value of that information for the questions I was interested in was low. Those variables might be useful in future analyses asking different questions.

The second issue is the interesting one: in rarely any of the publicly available portfolio was it identified correctly as an issue with the scraping method, instead suggesting that reviewers hadn’t understood the rating scale. If the issue had been detected at all before throwing fancy Machine Learning at the data.

The plot below shows dots from a LLM sentiment analysis of the review text: Positive-sounding reviews should be associated with high overall recommendations (in general).

This highlights the importance of critical review and extensive data exploration:

  1. Top-left square: No 10-ratings in the old data — despite reviews on the website with a rating of 10.
  2. Bottom-left square: So many reviews that were very clearly positive in tone had a rating of 1.
  3. Bottom-right circle: Expected absence of positive-tone reviews with lowest rating, leaving the remainder to suggest true misunderstanding of the rating scale, or a misclassification of the sentiment algorithm.

Stats selection

Binary ridge regression was used for the variable importance analysis predicting the outcome of ‘recommended’ (yes/no). This method isn’t for feature selection, hence the restriction to the actual review response variables, but used if many of fewer features are highly auto-correlated (when one goes up, so does another).

Use of AI statement

I use AI — in pretty much the same way I have (always) been using strackoverflow: for inspiration on how to get the code to do what I want it to do.

I am proficiently functional, bordering fluent in R, but I don’t eat and sleep its thousands of cool packages for the varied tasks I use it for alongside all my other tasks, so there’s that.

As for writing, I write all my own stuff. There was a period where I used AI for proof-reading, as to not make me look like a fool when claiming I am a thorough person with attention to detail. But I stopped doing that considering how much time I wasted undoing AI’s patronising suggestions of “style” and “conciseness” — and accepted that I’ll happily live with my own typos in the same way I happily live with everybody else’s typos. They usually result from honest attempts at making a piece of text more legible — it’s not a reflection on anyone’s carelessness.

I have been using the em-dash for decates—nothing to it.