Natural Language Processing: How AI Works With Text

Before you begin

Know classification, learned representations, and the distinction between a language model and its application.

• Supervised, Unsupervised, and Reinforcement Learning
• Neural Networks for Beginners
• What Are Large Language Models?

Language contains more than words in a list

A library search for books about machines that learn may need to find an entry titled Introduction to Machine Learning. Exact word matching helps when terms coincide; it can miss related meaning when they differ. NLP, or natural language processing, studies computational methods for working with language.

Tasks include sentiment classification, translation, information extraction, document search, speech-related text processing, and conversation. Some use simple rules, some use statistical models, and some use large neural networks. NLP is not a synonym for a chatbot.

Language is ambiguous. Fine can mean acceptable, a monetary penalty, or a description of size. Context, genre, and user intent affect interpretation. A system's errors often reflect what its representation or training data fails to capture.

Represent text as numbers

A basic representation counts words or short word sequences. A document becomes a vector, a list of numbers with defined positions. These methods are inspectable and can work well for some classification tasks, but counts alone lose much information about order and context.

An embedding is a learned numerical representation intended to capture useful relationships. Similarity between embeddings can help retrieve related passages. Similarity is not identity, truth, or agreement: two passages discussing opposite conclusions can share much vocabulary and context.

Tokenization divides text into units for processing. Choices affect handling of languages, punctuation, and rare names. Preserve original text so errors in preparation can be investigated rather than hidden by irreversible cleaning.

Work through library search

Store three short entries: one about training models, one about fixing bicycles, and one about study habits. A query about computers learning from examples should ideally retrieve the model-training entry. A keyword baseline might search for learn or training; an embedding-based approach may capture broader relationships.

Retrieval returns candidate source material. A language model can then generate an answer using that material. This is often called retrieval-augmented generation. It does not eliminate error: retrieval can miss the right source, and generation can misread or add unsupported claims.

Evaluate those stages separately. Did the correct passage appear among retrieved results? Did the answer preserve its meaning? Did citations support specific statements? One overall pleasant answer can conceal a failure at either stage.

Query → representation → retrieve passages → optional generated answer
                         ↓                         ↓
                    relevance check          support check

Other tasks need different targets

Sentiment analysis might classify a review as positive, negative, or mixed. The sentence The screen is lovely, but the battery is terrible contains conflicting evaluations. A single label may discard useful detail. Define whether your task concerns overall sentiment or a particular aspect.

Translation aims to preserve meaning across languages, but idioms, names, register, and domain terms complicate evaluation. A fluent translation can still reverse a negation or alter a number. For learning, compare important details with a trusted speaker or reference instead of judging only smoothness.

A chatbot is an interface and system behavior. It may use intent classification, retrieval, a language model, or combinations. Identify what evidence and tools support its answers rather than assuming that conversation implies factual reliability.

Build tests that probe meaning

Test a sentiment tool with I liked it and I did not like it. Then add I thought I would dislike it, but I enjoyed it. These reveal whether the system handles negation and contrast rather than recognizing isolated positive words.

For search, include paraphrases, ambiguous requests, and questions whose answers are absent. For generated responses, require support from supplied passages and inspect omissions. Measure the task you actually need rather than choosing a metric only because a library provides it.

scikit-learn documents text feature extraction methods such as token counts and TF-IDF, which weights terms using their occurrence across documents. These provide useful baselines before more complex systems. Text feature extraction.

Important terms

NLP

Computational methods for processing language.

Vector

An ordered collection of numerical values.

Embedding

A learned numerical representation.

Retrieval

Finding candidate information relevant to a query.

Sentiment

Expressed attitude toward something.

RAG

Generation supplied with retrieved source material.

TF-IDF

A text weighting method using term frequency and document frequency.

Mini project: Build a six-case language test

1. Write two positive, two negative, and two mixed or ambiguous reviews.
2. Record the intended meaning and acceptable answers before testing.
3. Include at least one negation and one contrast.
4. Compare predictions with your reference and classify the errors.
5. Finish by explaining whether a single sentiment label serves your actual purpose.

Common mistakes and debugging

• Equating similar embeddings with agreement: inspect source meaning.
• Treating retrieved material as a verified answer: check relevance and support separately.
• Removing punctuation and negation blindly: cleaning can destroy useful meaning.
• Calling every chatbot an LLM: inspect the system components.

Independent challenge

Create two passages with similar vocabulary but opposite advice. Design a question and check whether retrieval plus generation preserves the difference.

Check your understanding: 10 questions

1. Is NLP limited to chatbots?

2. What is an embedding?

3. Does embedding similarity imply agreement?

4. Which two stages should a RAG evaluation separate?

5. Why test negation?

6. In your own words, what does “NLP” mean?

7. In your own words, what does “Vector” mean?

8. In your own words, what does “Embedding” mean?

9. In your own words, what does “Retrieval” mean?

10. In your own words, what does “Sentiment” mean?

Quiz answers

Summary

NLP uses representations to classify, compare, retrieve, and generate language. Ambiguity and context make task-specific tests and source checks essential.

Continue learning

ML09 combines task design, data preparation, modeling, and evaluation into one improvement process.

• What Are Large Language Models?
• How to Use AI for Research Without Losing Accuracy
• Build a Raspberry Pi AI Assistant
• Learn a Language With AI Conversation and Feedback

Sources and further reading

• scikit-learn: Text feature extraction

Prepared 2026-09-18. Draft — text-processing API concepts; no model benchmark claimed