The analysis of a live chat strategy

If you offer live chat in your app, you will surely have asked yourself one of the following questions at least once so far:

Let me share with you the real story of one of our customers who wanted to find out how the live chat influences clients to finish the onboarding.  

This is what we did to find out what could be improved

The first question we asked was: How many people finished the onboarding after using the live chat support, and how many finished it without the live chat?

After doing the maths, we came to the conclusion that, overall, the ones who interacted with the live chat had a conversion rate twice as big as the ones who didn’t.

Brilliant. That’s just what we were hoping for. However, we later found out that the interpretation of these results was misleading.

But at this point we were concerned with finding out what we could improve. To figure that out, we went on to step 2.

The next question came naturally: where exactly during the onboarding process do people need the help of the live chat?

Basically, we analysed things systematically to see what happened at each step.

We were interested in:

We were taken aback by the results.

The overall analysis showed that the people who were using the live chat converted twice as much as the ones who didn’t. But the step-by-step analysis revealed the fact that there was no difference in conversion between those who used and those who didn’t use the live chat (see the last column in the image below).

How could this be? It seemed illogical.

We checked for any errors that might have sneaked in the data, but there were none.

So we took a closer look at how we analysed them and noticed that people usually interacted with the live chat towards the end of the onboarding process, when the onboarding rate was bigger anyways. It was over 50% as opposed to the first steps when the onboarding rate was around 12%.

This is where the error occurred: when calculating the overall conversion rate, it was split evenly between all the onboarding steps. In reality, the vast majority of people interacted with the onboarding at the end of the process, when the conversion rate was naturally bigger.

In the end we came to the following conclusion: the fact that people interacted with the live chat did not change their onboarding behaviour. The conversion rate was just the same, with or without the live chat.

Not the kind of news you wanted to hear if you were the CEO of that company… There were plenty of resources invested in the support team, tens of employees only for the live chat.

Did they have to lay off people? No, of course not.

Luckily, the CEO’s intuition came into play. He refused to believe that the live chat was completely useless. He made two very good points:

In the end, the CEO’s intuition was not to be ignored. And he was right. The whole point was how to improve the efficiency of the team.

So, back to the drawing board.

Money is usually the key, isn’t it? It was instrumental in this case as well. After all, what mattered most was how many people became paying customers, and not just how many finished the onboarding process.

So, the one question that would determine the efficiency of the live chat was:

how many people who became paying customers interacted with the live chat during the onboarding vs. how many didn’t?

These were the overall results.

This time we were in for a good surprize: overall, approx. 24 times more people finished the onboarding and became paying customers after interacting with the live chat than those who didn’t use it. That’s huge!

Naturally, we didn’t want to fall in the same trap as the first time, so we also took a look at the step-by-step results. This is what we found out:

As you can see from the table above, if you look at each step individually, up to 9 times more people converted and became paying customers after using the live chat than those who didn’t.

This time we had clear confirmation that, no matter how you look at it, there was a significant impact that the live chat had on the sales. Decreasing the resources invested in the live chat would have been a huge mistake.

So, we now had a clear vision of the process. The next step was optimizing it.

There were two major things that could be improved:

In the end I would like to offer just a quick tip: however many steps you need to take to reach the right insights, remember that data never lies. But you can easily lie to yourselves with data. 

To reach the correct conclusion you have to look at the data from all angles. And in order to prove that your conclusion is right, you can always try to prove it’s wrong. If you can’t, then it must be correct.