Natural Language Processing
Abstract
This chapter starts with several generic sections about natural language processing (NLP) datasets and big-picture concepts. It focuses on the core NLP concepts, and quick-and-dirty techniques to more complicated ones. The most basic concept in NLP is that of a “bag-of-words”, also called a frequency distribution. It's a way to turn a piece of free text (a tweet, a Word document, or whatever else) into a numerical vector that we can plug into a machine learning algorithm. The first part of any NLP process is simply breaking a piece of text into its constituent parts (usually words). This process is called “tokenization”, and it is complicated by issues such as punctuation markers, contractions, and a host of other things. The chapter is about extensions and refinements of the basic idea of bag-of-words, and discusses some of the more intricate (and error-prone) sentence parsing toward the end.