Tokenises text via the same tokeniser as sframe_term_frequency() (whitespace
splitting, lower-casing, punctuation stripping, and stop-word removal),
then slides a window of n tokens across each response's token vector
and counts how often each resulting n-gram occurs. n = 2 (the default)
gives bigrams, and n = 3 gives trigrams.
Arguments
- text
Character vector of responses (raw or already cleaned by
sframe_clean_text()).- n
Integer. N-gram size. Default
2(bigrams).- stop_words
Character vector of words to exclude, or
NULLto use the built-in English list, orcharacter(0)for no filtering.- top_n
Integer. Maximum number of n-grams to return, most frequent first. Default
30.
Details
An n-gram never straddles a removed stop word. Where "but" and "not" are filtered, "clean but not comfortable" yields no bigram at all, because "clean" and "comfortable" were never next to each other. This is what makes the output phrases respondents actually wrote.
See also
Other Text analysis functions:
sframe_clean_text(),
sframe_extract_quotes(),
sframe_term_context(),
sframe_term_frequency()
Examples
demo <- sframe_demo("tourism_services")
cleaned <- sframe_clean_text(demo$responses, "comments")
head(sframe_ngram_frequency(cleaned, n = 2, top_n = 10))