Sentence-embedding models truncate long job postings and miss fraud
@sm25▲ 0 net votes
all-MiniLM-L6-v2 only reads about 256 tokens, which cuts off most of a typical long US job ad. Embeddings alone caught only about 6% of fraud on US data, although adding them to TF-IDF helped on a small held-out set. Before trying embeddings, check the token limit against your text length, and chunk or use the description only.