Best way to evaluate cell label agreement with new clustering?

[X-Man]

I'm working on a single-cell dataset where I have a set of trusted annotations, and I've just run a new clustering on updated PCA/knn-graph embeddings. Some clusters align well with the known labels, some don't. Is there a way to score how well the known labels fit the new clustering? Bonus if it gives confidence metrics or stats.

 

[piscesmoon2530]

You could try Adjusted Rand Index (ARI) great for comparing clustering results to known labels. If you want per label insights, a confusion matrix with purity scores per class can help too. Some use bootstrapping or permutation tests to assess clustering stability.

 

[Josef_10]

I'm working on a single-cell dataset where I have a set of trusted annotations, and I've just run a new clustering on updated PCA/knn-graph embeddings. Some clusters align well with the known labels, some don't. Is there a way to score how well the known labels fit the new clustering? Bonus if it gives confidence metrics or stats.

Yeah, a common approach here is to use ARI like @piscesmoon2530 mentioned or NMI. I'd also recommend that you consider bootstrapping your new clustering multiple times and calculating the ARI / NMI for each iteration to get a distribution of scores. You could also look at individual cluster purity, which tells you how homogenous your new clusters are with respect to your known labels.

 

[TropicalMe]

You can use metrics like ARI or NMI to quantify how well your new clusters match the trusted labels or compute per cluster purity or silhouette scores using the known labels as ground truth.