我有一个制表符分隔的文件,其中一列偶尔有未转义的换行符(用引号引起来):
JOB REF Comment V2 Other
1 3 45 This was a small job NULL sdnsdf
2 4 456 This was a large job and I have to go onto a new line,
but I didn't properly escape so it's on the next row whoops! NULL NULL
3 7 354 NULL NULL NULL
# dat <- readLines("the-Dirty-Tab-Delimited-File.txt")
dat <- c("\tJOB\tREF\tComment\tV2\tOther", "1\t3\t45\tThis was a small job\tNULL\tsdnsdf",
"2\t4\t456\tThis was a large job and I have\t\t", "\t\"to go onto a new line, but I didn't properly escape so it's on the next row whoops!\"\tNULL\tNULL\t\t",
"3\t7\t354\tNULL\tNULL\tNULL")
我知道这可能是不可能的,但这些错误的换行符只出现在一个字段(第 10 列)中。我对 R(首选)或 python 中的解决方案感兴趣。
我的想法是引入一个正则表达式,在 10 个和仅 10 个制表符后寻找换行符。我开始使用 readLines 并尝试删除出现在空格 + 单词末尾的所有换行符:
dat <- gsub("( [a-zA-Z]*)\t\n", "\\1", dat)
但是readLines的行结构好像很难反转。我应该做什么?
编辑:有时会出现两个换行符(即用户在评论字段中的段落之间放置了一个空行。示例如下(期望的结果是应该将其制作成单个行)
140338 28855 WA 2 NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL 1 NULL NULL NULL NULL NULL NULL NULL NULL 1000 NULL NULL NULL NULL NULL NULL YNNNNNNN (Some text with two newlines)
The remainder of the text beneath two newlines NULL NULL NULL 3534a NULL email NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL
请您参考如下方法:
这是我用 Python 编写的答案。
import re
# This pattern should match correct data lines and should not
# match "continuation" lines (lines added by the unquoted newline).
# This pattern means: start of line, then a number, then white space,
# then another number, then more white space, then another number.
# This program won't work right if this pattern isn't correct.
pat = re.compile("^\d+\s+\d+\s+\d+")
def collect_lines(iterable):
itr = iter(iterable) # get an iterator
# First, loop until we find a valid line.
# This will skip the first line with the "header" info.
line = next(itr)
while True:
line = next(itr)
if pat.match(line):
# found a valid line; hold it as cur
cur = line
break
for line in itr:
# Look at the line after cur. Is it a valid line?
if pat.match(line):
# Line after cur is valid!
yield cur # output cur
cur = line # hold new line as new cur
else:
# Line after cur is not valid; append to cur but do not output yet.
cur = cur.rstrip('\r\n') + line
yield cur
data = """\
JOB REF Comment V2 Other
@@@1 3 45 This was a small job NULL sdnsdf
@@@2 4 456 This was a large job and I have to go onto a new line,
@@@ but I didn't properly escape so it's on the next row whoops! NULL NULL
@@@3 7 354 NULL NULL NULL
"""
lines = data.split('@@@')
for line in collect_lines(lines):
print(">>>{}<<<".format(line))
对于你的真实程序:
with open("filename", "rt") as f:
for line in collect_lines(f):
# do something with each line
编辑:我对此进行了修改并添加了更多评论。我想我也解决了您看到的问题。
当我加入一行到 cur 时,我没有先从 cur 的末尾去除换行符。所以,连接的线仍然是一条分割线,当它被写入文件时,这并没有真正解决问题。立即尝试。
我重新处理了测试数据,以便测试行上有换行符。我的原始测试在换行符上拆分输入,因此拆分行不包含任何换行符。现在每行都将以换行符结尾。
