我想计算 57832 个向量的成对距离。每个向量有 200 个维度。我正在使用 pdist 来计算距离。
from scipy.spatial.distance import pdist
pairwise_distances = pdist(X, 'cosine')
# pdist is supposed to return a numpy array with shape (57832*57831,).
但是,这会导致内存错误。
Traceback (most recent call last):
File "/home/munichong/git/DomainClassification/NameSuggestion@Verisign/classification_DMOZ/main.py", line 101, in <module>
result_clustering = clf_clustering.getCVResult(shuffle)
File "/home/munichong/git/DomainClassification/NameSuggestion@Verisign/classification_DMOZ/ClusteringBasedClassification.py", line 158, in getCVResult
self.centroids_of_categories(X_train, y_train)
File "/home/munichong/git/DomainClassification/NameSuggestion@Verisign/classification_DMOZ/ClusteringBasedClassification.py", line 103, in centroids_of_categories
cat_centroids.append( self.dpc.centroids(X_in_this_cat) )
File "/home/munichong/git/DomainClassification/NameSuggestion@Verisign/classification_DMOZ/ClusteringBasedClassification.py", line 23, in centroids
distance_dict, rho_dict = self.compute_distances_and_rhos(X)
File "/home/munichong/git/DomainClassification/NameSuggestion@Verisign/classification_DMOZ/ClusteringBasedClassification.py", line 59, in compute_distances_and_rhos
pairwise_distances = pdist(X, 'cosine')
File "/usr/local/lib/python2.7/dist-packages/scipy/spatial/distance.py", line 1185, in pdist
dm = np.zeros((m * (m - 1)) // 2, dtype=np.double)
MemoryError
我笔记本电脑的内存是 16GB。我该如何解决?或者有什么更好的方法吗?
请您参考如下方法:
在大型数据集上进行基于矩阵的算法是令人望而却步的。
内存需求很容易估计。即使利用对称性,许多实现最多也将达到约 65000 个实例。但即使是 64 位实现和大型机器最终也会放弃。具有 double 和利用对称性的 1000000x1000000 矩阵需要 4 TB 的 RAM。
使用不需要 O(n^2) 内存和运行时间的更好算法。
