我使用这个函数(见底部)从两个数据帧开始计算 Pearson 和 Pval，但我对 Pval 结果没有信心:似乎太多的负相关是显着的。
是否有更优雅的方法(如一行代码)来计算 Pval 和 Pearson？
这两个答案(pandas.DataFrame corrwith() method)和(correlation matrix of one dataframe with another)提供了优雅的解决方案，但缺少 P 值计算。
代码如下:
def pearson_cross_map(df1, df2):
"""Correlate each Mvar with each Nvar.
Parameters
----------
df1 : dataframe1
Shape Mobs X Mvar.
df2 : dataframe2
Shape Nobs X Nvar.
Returns
-------
DFcorr, dataframe Mvar x Nvar in which each element is a Pearson
correlation coefficient.
DFpval, dataframe Mvar x Nvar in which each element is a P value (one-tailed).
"""
intersection = (df1.index & df2.index).tolist()
df1 = df1.convert_objects(convert_numeric=True)
df1 = df1.T[intersection].T
df1 = df1.loc[:, (df1 != 0).any(axis=0)].sort().sort(axis=1)
df2 = df2.convert_objects(convert_numeric=True)
df2 = df2.T[intersection].T
df2 = df2.loc[:, (df2 != 0).any(axis=0)].sort().sort(axis=1)
x = df1.T.values
y = df2.T.values
mu_x = x.mean(1)
mu_y = y.mean(1)
n = x.shape[1]
s_x = x.std(1, ddof=n - 1)
s_y = y.std(1, ddof=n - 1)
cov = np.dot(x,y.T) - n * np.dot(mu_x[:, np.newaxis], mu_y[np.newaxis, :])
DFcoeff = pd.DataFrame(cov / np.dot(s_x[:, np.newaxis], s_y[np.newaxis, :]))
DFcoeff.index = df1.columns.tolist()
DFcoeff.columns = df2.columns.tolist()
n = len(intersection)
r = DFcoeff
t = r*np.sqrt((n-2)/(1-r*r))
DFpval = pd.DataFrame(stats.t.cdf(t, n-2))
DFpval.index = df1.columns.tolist()
DFpval.columns = df2.columns.tolist()
return DFcoeff, DFpval
谢谢!
请您参考如下方法:
您需要 Pearson 相关性测试，而不仅仅是相关性计算。因此，使用 scipy.stats.pearsonr返回估计的 Pearson 系数和 2 尾 pvalue 的方法。
由于该方法需要一系列输入，请考虑遍历两个数据帧的每一列以更新预先分配的矩阵。甚至转换为具有所需列和索引的数据框:
import numpy as np
import pandas as pd
from scipy.stats import pearsonr
df1 = pd.DataFrame(np.random.rand(10, 5), columns=['Col1', 'Col2', 'Col3', 'Col4', 'Col5'])
df2 = pd.DataFrame(np.random.rand(10, 5), columns=['Col1', 'Col2', 'Col3', 'Col4', 'Col5'])
coeffmat = np.zeros((df1.shape[1], df2.shape[1]))
pvalmat = np.zeros((df1.shape[1], df2.shape[1]))
for i in range(df1.shape[1]):
for j in range(df2.shape[1]):
corrtest = pearsonr(df1[df1.columns[i]], df2[df2.columns[j]])
coeffmat[i,j] = corrtest[0]
pvalmat[i,j] = corrtest[1]
dfcoeff = pd.DataFrame(coeffmat, columns=df2.columns, index=df1.columns)
print(dfcoeff)
# Col1 Col2 Col3 Col4 Col5
# Col1 -0.791083 0.459101 -0.488463 -0.289265 0.494897
# Col2 0.059446 -0.395072 0.310900 0.297532 0.201669
# Col3 -0.062592 0.391469 -0.450600 -0.136554 0.299579
# Col4 -0.470203 0.797971 -0.193561 -0.338896 -0.244132
# Col5 -0.057848 -0.037053 0.042798 0.176966 -0.157344
dfpvals = pd.DataFrame(pvalmat, columns=df2.columns, index=df1.columns)
print(dfpvals)
# Col1 Col2 Col3 Col4 Col5
# Col1 0.006421 0.181967 0.152007 0.417574 0.145871
# Col2 0.870421 0.258506 0.381919 0.403770 0.576357
# Col3 0.863615 0.263268 0.191245 0.706796 0.400385
# Col4 0.170260 0.005666 0.592096 0.338101 0.496668
# Col5 0.873881 0.919058 0.906551 0.624783 0.664206
