Skip to main content
 首页 » 编程设计

python re.compile 并用 ÆØÅ 字符拆分

2024年11月01日9zhwl

我是 Python 的新手。 我确实有一个包含单词列表的文件。它们包含丹麦字母 (ÆØÅ),但 re.compile 不理解这些字符。该函数按每个 ÆØÅ 拆分单词。文本是从 Twitter 和 Facebook 下载的,并不总是只包含字母。

text = "Rød grød med fløde.... !! :)" 
pattern_split = re.compile(r"\W+") 
words = pattern_split.split(text.lower()) 
words = ['r', 'd', 'gr', 'd', 'med', 'fl', 'de'] 

正确的结果应该是

    words = ['rød', 'grød', 'med', 'fløde'] 

如何获得正确的结果?

完整代码

#!/usr/bin/python  
# -*- coding: utf-8 -*- 
 
import math, re, sys, os 
reload(sys) 
sys.setdefaultencoding('utf-8') 
 
# AFINN-111 is as of June 2011 the most recent version of AFINN 
#filenameAFINN = 'AFINN/AFINN-111.txt' 
 
# Get location of file 
__location__ = os.path.realpath( 
    os.path.join(os.getcwd(), os.path.dirname(__file__))) 
 
 
filenameAFINN = __location__ + '/AFINN/AFINN-111DK.txt' 
afinn = dict(map(lambda (w, s): (w, int(s)), [  
            ws.strip().split('\t') for ws in open(filenameAFINN) ])) 
 
# Word splitter pattern 
pattern_split = re.compile(r"\W+") 
#pattern_split = re.compile('[ .,:();!?]+') 
 
def sentiment(text): 
    print(text) 
    words = pattern_split.split(text.lower().strip()) 
    print(words) 
    sentiments = map(lambda word: afinn.get(word, 0), words) 
    if sentiments: 
        sentiment = float(sum(sentiments))/math.sqrt(len(sentiments)) 
 
    else: 
        sentiment = 0 
    return sentiment 
 
 
# Print result 
text = "ånd ånd med fløde... :)asd " 
id = 999 
split = "###" 
print("%6.2f%s%s%s%s" % (sentiment(text), split, id, split, text)) 

请您参考如下方法:

修改您的脚本以使用最佳实践:

import csv 
import math 
import os 
import re 
 
LOCATION = os.path.dirname(os.path.abspath(__file__)) 
afinn_filename = os.path.join(LOCATION, '/AFINN/AFINN-111DK.txt') 
 
pattern_split = re.compile(r"\W+") 
 
with open(afinn_filename, encoding='utf8', newline='') as infile: 
    reader = csv.reader(infile, delimiter='\t') 
    afinn = {key: int(score) for key, score in reader} 
 
 
def sentiment(text): 
    words = pattern_split.split(text.lower().strip()) 
    if not words: 
        return 0 
    sentiments = [afinn.get(word, 0) for word in words] 
    return sum(sentiments) / math.sqrt(len(sentiments)) 
 
 
# Print result 
text = "ånd ånd med fløde... :)asd " 
id = 999 
split = "###" 
print('{sentiment:6.2f}{split}{id}{split}{text}'.format( 
    sentiment=sentiment(text), id=id, split=split, text=text)) 

用 Python 3 运行它意味着 text 是一个 Unicode 对象,正则表达式是用 re.UNICODE 集解释的。

在 Python 2 中,您将使用:

text = u"ånd ånd med fløde... :)asd " 

(注意字符串中的前导 u 前缀)和

pattern_split = re.compile(ur"\W+", re.UNICODE) 

您的 AFINN 文件仍将被读取为 CSV,但事后从 UTF8 解码 key,使用:

with open(afinn_filename, 'rb') as infile: 
    reader = csv.reader(infile, delimiter='\t') 
    afinn = {key.decode('utf8'): int(score) for key, score in reader}