How's Twitter Doing Post-Elon Musk?

Hello! Today’s post isn’t F2P game dev related, but it is related to app performance and the importance of understanding data for product managers.

Some of the lessons here are the same lessons we should learn when conducting market research on games. Too often, I’ve seen data used in support of corporate political narratives, thus arriving at bad conclusions.

Finally, the conclusions of this post were a subject of debate within a private game development slack group I’m part of, so I decided to dig into the data here.

To all,

A recent Buzzfeed article titled “Twitter Is Shedding Users, Most Of Them Democrats, A New Survey Shows” highlighted the results from a survey conducted by the COVID States Project based on public opinion polling of ~25,000 Americans across all 50 US States. The report was a joint study involving researchers from Northeastern University, Harvard, Rutgers, and Northwestern University.

Why the National Science Foundation would fund such a study is beyond me.

However, as someone who hates both Democrats and Republicans equally, I am uniquely qualified to avoid all of the partisan bias that affects many Americans today. Further, I have access to data.ai. This allows me to give you better data than those fancy nerds who spent tons of time and money on their surveys and help you understand what’s really happening.

While the study itself was pretty boring AF to someone who’s not very political like me, I was more interested in understanding the data itself. Further, from a product manager’s perspective, an app that’s “shedding users” by losing lots of Democrats is an interesting business implication.

Here are some of the claims from Buzzfeed:

• “The number of people using Twitter in the US has decreased almost 9% since Elon Musk took over.”

• “In October 2022, just before Musk took ownership, the study found, 32.4% of Americans were using Twitter. In December and January, that figure had dropped to 29.5%.”

The first quote above is a poor conclusion and false claim. Note that the Buzzfeed author took the polling survey data, as noted in the second quote, as the basis for the ~9% drop. To be clear, the study was a poll asking people to self-report whether they are using Twitter or not.

Didn’t we learn this lesson during the last presidential election polling, when people self-reported they would not vote for Trump, but then they did?

I live in the SF Bay Area, which is mainly left-leaning; I know many people here who said they would never use Twitter again, yet there they are, still posting away.

In F2P game development, PMs need to know how to separate various stakeholder narratives from reality. And in this case, as well, we can rely on data.ai to take a look at the numbers.

Did this Buzzfeed author go too far?

While data.ai estimates may also be based on sampled data, I would submit that they would likely be far more accurate than a survey of only 25K data points. Note for PMs: this critical assumption underpins most of the conclusions from here on out.

Data.ai data estimates seem to suggest the author went too far:

Looking at monthly active users from October 2022 through January 2023 just in the US, as per the research study, data.ai shows a drop from 69.4M to 67.3M monthly active users. An ~3% drop in monthly active users. 

If this were a game analysis, the next thing I would do is baseline these numbers.

For example, something really fast and easy we can do is to compare the US drop to what happened worldwide. Worldwide monthly active users data is shown below:

Worldwide, we see a drop from 543M to 533M users. This is an ~1.8% drop in monthly active users. 

From this data, I suspect that the research study was likely impacted by significant polling biases that did not reflect actual user behavior.

A better conclusion (but still potentially false - read to the end) would be that the ~3% drop in monthly active users in the US compared to a ~1.8% drop worldwide indicates some impact from Democrats leaving Twitter. However, the scale of the impact is likely highly overstated.

More disturbing, however, was the language used to push a narrative based on questionable methodology but stated as fact: “The number of people using Twitter in the US has decreased almost 9% since Elon Musk took over.”

Now, as a PM, many might stop here. However, better PMs would ask more questions. For example, “Could other effects be at play as well?” Generally, the specific practice a good PM here would employ would be some form of root cause analysis: such as 5 Whys or decision tree analysis.

What other effects could be impacting monthly active users?

Let’s now take a look at data.ai estimates of paid downloads by Twitter over the past two years. We notice that Twitter has been actively buying users for its service but abruptly stopped in July of 2022.

The above data shows that before Elon’s acquisition, Twitter typically acquired between 300K - 800K users per month. This likely does not explain the entire monthly active user gap, and trying to back out this impact fully involves complexity outside this post's scope. However, not buying, let’s say, 1.3M in aggregate users from October through January would have some impact. And you have user stacking from previous months of buying as well.

Other complicating factors are seasonality, post-Covid normalization, and probably the biggest factor: the elimination of bots.

It’s not clear how effective Twitter has been at reducing bots. Still, to the extent that Twitter has eliminated bots, the service may be adding real users and usage rather than losing users.

If Twitter eliminated more than 3% of monthly active users from bots, then in complete contradiction to the Buzzfeed author’s claim, Twitter may be adding real users.

The real conclusion seems to be that it’s difficult to reach any conclusion based on the data we’re seeing.

Further, a PM follow-up should look at the historical trends from the October through January period. Over the past four years, we have not seen a similar decrease as post-Elon Musk acquisition, although there are many complicating factors, as noted above.

A couple of final issues to consider: 1. user sentiment and 2. average user time.

First, the overall user sentiment pre and post-Elon Musk acquisition. Are users actually rating the Twitter app experience more critically or favorably?

Critical reviews based on data.ai estimates are shown below:

While there does seem to be a spike in critical reviews from last November, one month after Elon’s Twitter acquisition, critical reviews seem to be dropping. Further, critical reviews from a historical perspective are not nearly as high as whatever happened in May of 2021.

Similarly, favorable reviews are relatively stable as well:

Favorable reviews remain fairly steady at around 90%, with a recent, slight dip in November of 2022.

Closing out our analysis, we now want to look at Twitter engagement per-user. Let’s look at the average time spent per user pre and post-Elon Musk acquisition:

From a historical perspective, there hasn’t been much change to average user time. However, if whatever happened in December can be made more sustainable, this can indicate increasing user engagement for Twitter. Note, however, I’m not sure how bot elimination impacts this metric as well.

In conclusion, please don’t believe everything you read, and always think critically about fast and hard conclusions from surveys and user polling.