Skip to main content
 首页 » 编程设计

python之使用未转义的换行符清理制表符分隔文件

2024年11月01日11me-sa

我有一个制表符分隔的文件,其中一列偶尔有未转义的换行符(用引号引起来):

   JOB  REF Comment V2  Other 
1   3   45  This was a small job    NULL    sdnsdf 
2   4   456 This was a large job and I have to go onto a new line,  
    but I didn't properly escape so it's on the next row whoops!    NULL    NULL         
3   7   354 NULL    NULL    NULL 
 
# dat <- readLines("the-Dirty-Tab-Delimited-File.txt") 
dat <- c("\tJOB\tREF\tComment\tV2\tOther", "1\t3\t45\tThis was a small job\tNULL\tsdnsdf",  
"2\t4\t456\tThis was a large job and I have\t\t", "\t\"to go onto a new line, but I didn't properly escape so it's on the next row whoops!\"\tNULL\tNULL\t\t",  
"3\t7\t354\tNULL\tNULL\tNULL") 

我知道这可能是不可能的,但这些错误的换行符只出现在一个字段(第 10 列)中。我对 R(首选)或 python 中的解决方案感兴趣。

我的想法是引入一个正则表达式,在 10 个和仅 10 个制表符后寻找换行符。我开始使用 readLines 并尝试删除出现在空格 + 单词末尾的所有换行符:

dat <- gsub("( [a-zA-Z]*)\t\n", "\\1", dat) 

但是readLines的行结构好像很难反转。我应该做什么?

编辑:有时会出现两个换行符(即用户在评论字段中的段落之间放置了一个空行。示例如下(期望的结果是应该将其制作成单个行)

140338  28855   WA  2   NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    1   NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    1000    NULL    NULL    NULL    NULL    NULL    NULL    YNNNNNNN    (Some text with two newlines) 
 
The remainder of the text beneath two newlines  NULL    NULL    NULL    3534a   NULL    email   NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL    NULL 

请您参考如下方法:

这是我用 Python 编写的答案。

import re 
 
# This pattern should match correct data lines and should not 
# match "continuation" lines (lines added by the unquoted newline). 
# This pattern means: start of line, then a number, then white space, 
# then another number, then more white space, then another number. 
 
# This program won't work right if this pattern isn't correct. 
pat = re.compile("^\d+\s+\d+\s+\d+") 
 
def collect_lines(iterable): 
    itr = iter(iterable)  # get an iterator 
 
    # First, loop until we find a valid line. 
    # This will skip the first line with the "header" info. 
    line = next(itr) 
    while True: 
        line = next(itr) 
        if pat.match(line): 
            # found a valid line; hold it as cur 
            cur = line 
            break 
    for line in itr: 
        # Look at the line after cur.  Is it a valid line? 
        if pat.match(line): 
            # Line after cur is valid! 
            yield cur  # output cur 
            cur = line  # hold new line as new cur 
        else: 
            # Line after cur is not valid; append to cur but do not output yet. 
            cur = cur.rstrip('\r\n') + line 
    yield cur 
 
data = """\ 
   JOB  REF Comment V2  Other 
@@@1   3   45  This was a small job    NULL    sdnsdf 
@@@2   4   456 This was a large job and I have to go onto a new line,  
@@@    but I didn't properly escape so it's on the next row whoops!    NULL    NULL         
@@@3   7   354 NULL    NULL    NULL 
""" 
 
lines = data.split('@@@') 
for line in collect_lines(lines): 
    print(">>>{}<<<".format(line)) 

对于你的真实程序:

with open("filename", "rt") as f: 
    for line in collect_lines(f): 
        # do something with each line 

编辑:我对此进行了修改并添加了更多评论。我想我也解决了您看到的问题。

当我加入一行到 cur 时,我没有先从 cur 的末尾去除换行符。所以,连接的线仍然是一条分割线,当它被写入文件时,这并没有真正解决问题。立即尝试。

我重新处理了测试数据,以便测试行上有换行符。我的原始测试在换行符上拆分输入,因此拆分行不包含任何换行符。现在每行都将以换行符结尾。