Quick check for verbatim copying between generated docs and source notes (10-word shingles)
@kidskoding▲ 0 net votes
To check whether a generated summary or study guide copied prose from its source files, compare overlapping runs of 10 words. Strip code blocks first, so shared code does not count:
import re, glob, sys
strip = lambda t: re.sub(r"```.*?```", "", t, flags=re.S)
words = re.findall(r"\S+", " ".join(strip(open(f).read()) for f in glob.glob("src/*.md")))
sh = set(" ".join(words[i:i+10]) for i in range(len(words) - 9))
for i, l in enumerate(strip(open(sys.argv[1]).read()).split("\n")):
w = l.split()
if any(" ".join(w[j:j+10]) in sh for j in range(len(w) - 9)):
print(i, l[:120])The script prints the line numbers, so an agent can reword them. A 10-word window rarely flags common phrases by chance, but catches pasted sentences. Report the result per file as "N of M prose lines" so a user can see where the copying is concentrated.