From raw usage logs to a product decision. About 60 minutes. ⏱ ~4 min to read this page

Objective

You are given a dataset describing how users interact with Vibe Work (formerly known as Le Chat), an AI assistant. The data is synthetic but reflects patterns inspired by real-world usage. Your objective is to understand user engagement and retention, and to help the team decide where to focus.

Key areas of focus

• Data quality: can we trust these logs, and what do the fields really measure?
• User engagement: how often and how intensively do users interact with Vibe Work, and how is that evolving?
• User retention: how many users stay, what is associated with users leaving, and where would you focus the team's effort?
• Experimentation: how would you test whether a fix actually works, and how would you read the results?

Dataset

One row per request.

Data quality

You've just received this file: 258,786 rows, one per request.

Before any analysis, what data quality checks would you run? What could make you distrust this data? ⏱ ~2 min

Data quality: a first look

You load the file in a notebook and run the usual first commands.

Here is what df.info() and df.describe() return. What do you conclude? ⏱ ~2 min

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 258786 entries, 0 to 258785
Data columns (total 10 columns):
 #   Column                           Non-Null Count   Dtype 
---  ------                           --------------   ----- 
 0   request_uuid                     258786 non-null  object
 1   user_uuid                        258786 non-null  object
 2   request_at                       258786 non-null  object
 3   request_date_at                  258786 non-null  object
 4   user_country                     258786 non-null  object
 5   intent_category                  258786 non-null  object
 6   has_error_during_generation      258786 non-null  int64 
 7   is_dislike                       258786 non-null  int64 
 8   number_of_characters_in_answers  258786 non-null  int64 
 9   time_per_characters_ms           258786 non-null  int64 
dtypes: int64(4), object(6)
memory usage: 19.7+ MB

       has_error_during_generation  is_dislike  number_of_characters_in_answers  time_per_characters_ms
count                    258786.00   258786.00                        258786.00               258786.00
mean                          0.02        0.02                          1261.98                    3.14
std                           0.15        0.14                           837.32                    1.63
min                           0.00        0.00                            50.00                    1.00
25%                           0.00        0.00                           795.00                    2.00
50%                           0.00        0.00                           975.00                    3.00
75%                           0.00        0.00                          1347.00                    4.00
max                           1.00        1.00                          8419.00                   47.00

                                request_uuid                             user_uuid           request_at request_date_at user_country intent_category
count                                 258786                                258786               258786          258786       258786          258786
unique                                258786                                 11086               247495              59            3               4
top     d08aec8e-306c-41da-8ce0-5198d6330ba1  6f37f97f-523d-47b4-8e8e-547b7b4ae79d  2025-02-12 16:24:31      2025-02-26           US         Writing
freq                                       1                                    88                    4            6264       125127           90917

General exploration

The data can be trusted. Before looking at engagement or retention, get to know each variable.

Let's get to know the variables one by one: country, intent, answer length, generation speed, errors and dislikes. What chart would you draw for each, and what would you look for? ⏱ ~3 min

General exploration: the charts

Here is one chart per variable.

What do these charts tell you? What stands out, and what would you dig into next? ⏱ ~4 min

A. Country

B. Intent

C. Answer length

D. Generation time per character

E. Error rate per day

Data exploration: engagement

The data looks usable. Let's start with engagement.

How would you explore how frequently and how intensively users interact with Vibe Work? Which metrics and breakdowns would you look at, and what would you watch out for? ⏱ ~3 min

Engagement: the charts

Here are the main engagement charts.

What do these charts tell you about how users engage with Vibe Work? What stands out, and what would you dig into next? ⏱ ~3 min

A. Volume over time: requests, active users, requests per active user

B. How often users come back

C. Intent mix by country

Retention: definition and measurement

Leadership is worried about retention.

How would you define and measure retention with this data? ⏱ ~3 min

Retention: choosing the churn window

To say a user has churned, we first need to decide how long a silence is too long.

How would you choose the time to come back, i.e. how long without a request before you call a user churned? ⏱ ~2 min

Retention: picking the churn window

For each number of inactive days N, here is how often users came back afterwards.

Looking at this curve, which churn window would you choose, and why? ⏱ ~3 min

One point per active day followed by more than N days of inactivity. Only active days early enough to observe the inactivity plus the following 21 days are used.

Retention: reading the curves

Here are the weekly retention curves by cohort, excluding the first week (pre-existing users) and the partial last week.

Here are the weekly retention curves, one per first-seen cohort. What do you take from them? What would you tell leadership? ⏱ ~3 min

Share of each cohort active in week N after its first-seen week. Cohorts start on 8 Jan (the first week is excluded), and the partial last week (26–28 Feb) is excluded.

Retention: what drives it

We know how to measure retention. Now, what drives it?

What charts would you want to plot to start exploring what drives retention? ⏱ ~2 min

Retention: the charts

Here are the charts of churn after a request, by experience. All use the 14-day churn definition and exclude the last 14 days of data.

What do these charts tell you about what drives retention? What would you be careful about? ⏱ ~3 min

A. Churn by intent

B. Errors and dislikes

C. Country, at equal intent

Retention: causality

The charts show associations. How do we get to causes?

What are 3+ causal methods you could use to study the impact of a given experience (e.g. an error, an image request) on churn? ⏱ ~5 min

Retention: what to prioritise

A colleague regressed churn after each request on the request experience, comparing each user with themselves (user fixed effects). Coefficients are in probability units: 0.0147 = +1.47 percentage points of churn.

                                PanelOLS Estimation Summary                                 
============================================================================================
Dep. Variable:                     churn_14d   R-squared (Within):                    0.0018
Estimator:                          PanelOLS   No. Observations:                     185,600
Entity effects:                    user_uuid   Entities:                               9,358
Mean of dep. variable:                0.0185   Cov. Estimator:            Clustered (entity)
============================================================================================
                                 Parameter  Std. Err.  T-stat  P-value  Lower CI  Upper CI
--------------------------------------------------------------------------------------------
Intercept                           0.0156     0.0011   13.66    0.000    0.0134    0.0178
error                               0.0147     0.0026    5.60    0.000    0.0095    0.0198
dislike                            -0.0024     0.0020   -1.20    0.231   -0.0064    0.0016
length_k                           -0.0001     0.0004   -0.34    0.732   -0.0008    0.0006
latency_ms                         -0.0001     0.0002   -0.56    0.575   -0.0005    0.0003
C(intent)[T.Image_Generation]       0.0136     0.0012   11.74    0.000    0.0113    0.0159
C(intent)[T.Search]                 0.0011     0.0009    1.20    0.228   -0.0007    0.0028
C(intent)[T.Writing]                0.0008     0.0007    1.08    0.280   -0.0006    0.0021
============================================================================================
C(day_of_week) coefficients not shown. Intent baseline: Coding.
churn_14d = 1 if the user makes no further request within 14 days after the request.
Sample: requests at least 14 days before the end of the data.

Leadership can fund one retention project next quarter. Based on these results, what would you prioritise, and why? ⏱ ~5 min

A/B test: design

Following the retention analysis, the team built a new image generation model, "Imagine v2". Before rolling it out, they want to A/B test it.

Design the test: who is in it, what do you randomise, what is your primary metric, what are your guardrails, how many users do you need, and how long does it run? ⏱ ~5 min

A/B test: first readout Optional

The test ran for 8 weeks. Here is the dashboard the product team shared.

The PM says: "Retention +2.5 pp, engagement +18%. Let's ship." What do you check before agreeing? ⏱ ~4 min

A/B test: the rerun Optional

The logging was fixed (users are now logged when randomised) and the test was rerun for 8 weeks. The split is now balanced: 5,931 vs 5,902 users.

Effect by week since first exposure

The primary metric is significant (+1.5 pp, p = 0.04). What effect would you report to leadership, and why? ⏱ ~3 min

What we covered

Your turn

What questions do you have for us, about the team, the role, or how we work with data at Mistral? ⏱ ~5 min