Skip to main content
 首页 » 编程设计

Python:计算成对距离会导致内存错误

2024年10月24日13sky-heaven

我想计算 57832 个向量的成对距离。每个向量有 200 个维度。我正在使用 pdist 来计算距离。

from scipy.spatial.distance import pdist 
pairwise_distances = pdist(X, 'cosine') 
# pdist is supposed to return a numpy array with shape (57832*57831,). 

但是,这会导致内存错误。

   Traceback (most recent call last): 
  File "/home/munichong/git/DomainClassification/NameSuggestion@Verisign/classification_DMOZ/main.py", line 101, in <module> 
    result_clustering = clf_clustering.getCVResult(shuffle) 
  File "/home/munichong/git/DomainClassification/NameSuggestion@Verisign/classification_DMOZ/ClusteringBasedClassification.py", line 158, in getCVResult 
    self.centroids_of_categories(X_train, y_train) 
  File "/home/munichong/git/DomainClassification/NameSuggestion@Verisign/classification_DMOZ/ClusteringBasedClassification.py", line 103, in centroids_of_categories 
    cat_centroids.append( self.dpc.centroids(X_in_this_cat) ) 
  File "/home/munichong/git/DomainClassification/NameSuggestion@Verisign/classification_DMOZ/ClusteringBasedClassification.py", line 23, in centroids 
    distance_dict, rho_dict = self.compute_distances_and_rhos(X) 
  File "/home/munichong/git/DomainClassification/NameSuggestion@Verisign/classification_DMOZ/ClusteringBasedClassification.py", line 59, in compute_distances_and_rhos 
    pairwise_distances = pdist(X, 'cosine') 
  File "/usr/local/lib/python2.7/dist-packages/scipy/spatial/distance.py", line 1185, in pdist 
    dm = np.zeros((m * (m - 1)) // 2, dtype=np.double) 
MemoryError 

我笔记本电脑的内存是 16GB。我该如何解决?或者有什么更好的方法吗?

请您参考如下方法:

在大型数据集上进行基于矩阵的算法是令人望而却步的。

内存需求很容易估计。即使利用对称性,许多实现最多也将达到约 65000 个实例。但即使是 64 位实现和大型机器最终也会放弃。具有 double 和利用对称性的 1000000x1000000 矩阵需要 4 TB 的 RAM。

使用不需要 O(n^2) 内存和运行时间的更好算法。