Skip to main content
 首页 » 编程设计

python之NLP之如何添加更多功能

2024年11月24日7pander-it

我想使用 sklearn 分类器训练模型,使用文本特征(内容)、数字特征(人口)和分类特征(位置)对数据条目(是,否)进行分类。

下面的模型仅使用文本数据对每个条目进行分类。文本在导入分类器之前使用 TF-IDF 转换为稀疏矩阵。

有没有办法添加/使用其他功能?这些特征不是稀疏矩阵格式,所以不确定如何将它们与文本稀疏矩阵结合起来。

 
    #import libraries 
    import string, re, nltk 
    import pandas as pd 
    from pandas import Series, DataFrame 
    from nltk.corpus import stopwords 
    from nltk.stem.porter import PorterStemmer 
    from sklearn.feature_extraction.text import CountVectorizer 
    from sklearn.feature_extraction.text import TfidfTransformer 
    from sklearn.model_selection import train_test_split 
    from sklearn.metrics import classification_report 
    from sklearn.pipeline import Pipeline 
 
    # read data and remove empty lines 
    dataset = pd.read_csv('sample_data.txt', 
                           sep='\t', 
                           names=['content','location','population','target']) 
                           .dropna(how='all') 
                           .dropna(subset=['target']) 
 
    df = dataset[1:] 
 
    #reset dataframe index 
    df.reset_index(inplace = True) 
 
    #add an extra column which is the length of text 
    df['length'] = df['content'].apply(len) 
 
    #create a dataframe that contains only two columns the text and the target class 
    df_cont = df.copy() 
    df_cont = df_cont.drop( 
        ['location','population','length'],axis = 1) 
 
    # function that takes in a string of text, removes all punctuation, stopwords and returns a list of cleaned text 
 
    def text_process(mess): 
        # lower case for string 
        mess = mess.lower() 
 
        # check characters and removes URLs 
       nourl = re.sub(r'http\S+', ' ', mess) 
 
        # check characters and removes punctuation 
        nopunc = [char for char in nourl if char not in string.punctuation] 
 
        # join the characters again to form the string and removes numbers 
        nopunc = ''.join([i for i in nopunc if not i.isdigit()]) 
 
        # remove stopwords 
        return [ps.stem(word) for word in nopunc.split() if word not in set(stopwords.words('english'))] 
 
    #split the data in train and test set and train/test the model 
 
    cont_train, cont_test, target_train, target_test = train_test_split(df_cont['content'],df_cont['target'],test_size = 0.2,shuffle = True, random_state = 1) 
 
 
    pipeline = Pipeline([('bag_of_words',CountVectorizer(analyzer=text_process)), 
                         ('tfidf',TfidfTransformer()), 
                         ('classifier',MultinomialNB())]) 
 
    pipeline.fit(cont_train,target_train) 
    predictions = pipeline.predict(cont_test) 
 
    print(classification_report(predictions,target_test)) 
 

模型预计返回以下内容:准确率、精确率、召回率、f1-score

请您参考如下方法:

我相信您需要为“位置”功能使用单热向量。 给定数据的单热向量是,

伦敦 - 100

曼彻斯特 - 010

爱丁堡 - 001

矢量长度是您所在城市的数量。请注意,这里的每一位都是一个特征。 在提供给机器学习算法之前,分类数据通常会转换为单热向量。

完成此操作后,您可以将整行连接到一维数组中,然后将其提供给分类器。