我正在为面试官计算 Spearman 相关系数。它适用于 Interviewer_1...我不明白 Scipy 如何打断 interviewer_2 没有相关性/0/nan。
import pandas as pd
from pandas import DataFrame
import scipy.stats
df = pd.DataFrame({'Interviewer': ['Interviewer_1','Interviewer_1','Interviewer_1','Interviewer_1','Interviewer_1','Interviewer_1','Interviewer_1','Interviewer_1','Interviewer_1','Interviewer_1','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2','Interviewer_2'],
'Score_1': [-1,-1,-1,1,1,-1,-1,-1,1,1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,1,-1,-1,-1,-1,1,-1],
'Score_2': [1,-1,-1,-1,1,1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
})
df
样本数据 yield :
Interviewer Score_1 Score_2
0 Interviewer_1 -1 1
1 Interviewer_1 -1 -1
2 Interviewer_1 -1 -1
3 Interviewer_1 1 -1
4 Interviewer_1 1 1
5 Interviewer_1 -1 1
6 Interviewer_1 -1 -1
7 Interviewer_1 -1 -1
8 Interviewer_1 1 -1
9 Interviewer_1 1 -1
10 Interviewer_2 -1 -1
11 Interviewer_2 -1 -1
12 Interviewer_2 -1 -1
13 Interviewer_2 -1 -1
14 Interviewer_2 -1 -1
15 Interviewer_2 -1 -1
16 Interviewer_2 -1 -1
17 Interviewer_2 -1 -1
18 Interviewer_2 -1 -1
19 Interviewer_2 -1 -1
20 Interviewer_2 -1 -1
21 Interviewer_2 -1 -1
22 Interviewer_2 -1 -1
23 Interviewer_2 1 -1
24 Interviewer_2 -1 -1
25 Interviewer_2 -1 -1
26 Interviewer_2 -1 -1
27 Interviewer_2 -1 -1
28 Interviewer_2 1 -1
29 Interviewer_2 -1 -1
df.groupby('Interviewer').sum()
产生总和:
Score_1 Score_2
Interviewer
Interviewer_1 -2 -4
Interviewer_2 -16 -20
使用 Scipy:
def applyspearman(row):
row['Cor'] = scipy.stats.spearmanr(row['Score_1'], row['Score_2'])[0]
return row
df = df.groupby('Interviewer').apply(applyspearman)
df
Interviewer Score_1 Score_2 Cor
0 Interviewer_1 -1 1 -0.089087081
1 Interviewer_1 -1 -1 -0.089087081
2 Interviewer_1 -1 -1 -0.089087081
3 Interviewer_1 1 -1 -0.089087081
4 Interviewer_1 1 1 -0.089087081
5 Interviewer_1 -1 1 -0.089087081
6 Interviewer_1 -1 -1 -0.089087081
7 Interviewer_1 -1 -1 -0.089087081
8 Interviewer_1 1 -1 -0.089087081
9 Interviewer_1 1 -1 -0.089087081
10 Interviewer_2 -1 -1
11 Interviewer_2 -1 -1
12 Interviewer_2 -1 -1
13 Interviewer_2 -1 -1
14 Interviewer_2 -1 -1
15 Interviewer_2 -1 -1
16 Interviewer_2 -1 -1
17 Interviewer_2 -1 -1
18 Interviewer_2 -1 -1
19 Interviewer_2 -1 -1
20 Interviewer_2 -1 -1
21 Interviewer_2 -1 -1
22 Interviewer_2 -1 -1
23 Interviewer_2 1 -1
24 Interviewer_2 -1 -1
25 Interviewer_2 -1 -1
26 Interviewer_2 -1 -1
27 Interviewer_2 -1 -1
28 Interviewer_2 1 -1
29 Interviewer_2 -1 -1
我尝试在 Excel 中手动使用这个公式(排名函数、abs 差、d^2 和 d^ 之和,两个面试官得到了不同的结果: p = 1 - (6 Σ d^2i)/(n(n^2-1))
interviewer_1,p = 0.878788
interviewer_2,p = 0.993985
问题:
- 为什么 Interviewer_2 为空? NaN 问题是否与等级关系有关?
- 为什么 Scipy 的结果与我的手工结果不同?
请您参考如下方法:
不确定 source 中到底发生了什么但是您可以使用 pandas 的 Series.rank(method='dense') 定义自己的函数,这似乎可以解决问题:
def spearmanr(x, y):
""" `x`, `y` --> pd.Series"""
assert x.shape == y.shape
rx = x.rank(method='dense')
ry = y.rank(method='dense')
d = rx - ry
dsq = np.sum(np.square(d))
n = x.shape[0]
coef = 1. - (6. * dsq) / (n * (n**2 - 1.))
return coef
grouped.apply(lambda frame: spearmanr(frame['Score_1'], frame['Score_2']))
Interviewer_1 0.970
Interviewer_2 0.998
