Natural Language Processing (NLP)

Natural language processing is the AI discipline that enables machines to read, interpret, and generate human language. It powers search engines, voice assistants, machine translation, and the text-to-SQL interfaces that let business users query data without writing code.

For data-driven organizations, NLP transforms how people interact with their data infrastructure. Instead of browsing a catalog manually or asking an analyst to write a query, a product manager types "show me customer churn by region for Q3" and gets a chart. That interaction depends on NLP parsing intent, mapping terms to governed definitions, and generating the right query — a process that breaks down the moment the underlying metadata is ambiguous or missing.

How NLP Works

An NLP pipeline transforms raw text into structured meaning through a series of stages. Each stage builds on the previous one, and errors at any stage propagate forward.

Tokenization splits text into units the model can process — words, subwords, or characters. When a user types "Show me quarterly revenue by region," the tokenizer breaks it into tokens that map to the model's vocabulary. Modern tokenizers like BPE (byte pair encoding) handle unknown words by splitting them into known subword pieces rather than discarding them.

Embedding converts each token into a dense numerical vector that captures its semantic meaning. Words with similar meanings cluster near each other in this vector space: "revenue" sits close to "income" and "sales," while "region" sits near "territory" and "geography." These embeddings are what make semantic search possible — matching meaning rather than exact strings.

Model processing reads the full sequence of embeddings and produces an output that depends on the task. For classification, it predicts a label. For generation, it produces the next token. For text-to-SQL, it translates the natural language input into a database query. The model needs context beyond the words themselves: when the user says "revenue," the model must know which table holds the canonical revenue metric and how it's calculated — information that comes from a business glossary, not from the text itself.

Output varies by task: a translated sentence, a SQL query, a classification label, a generated paragraph, or extracted entities like company names and dates. The quality of this output depends entirely on the quality of every preceding stage.

The Transformer Revolution

Before 2017, NLP models processed text sequentially — reading one word at a time, left to right. This limited their ability to capture relationships between distant words in a sentence. The transformer architecture changed that by introducing self-attention: a mechanism that lets the model weigh every word against every other word simultaneously. A sentence like "The bank by the river was eroding" requires understanding that "bank" means a riverbank, not a financial institution — context that depends on words appearing later in the sentence.

This single architectural change enabled the jump from narrow NLP tools to general-purpose language models. BERT (Google, 2018) reads text bidirectionally and excels at understanding tasks: classification, entity recognition, semantic search. GPT (OpenAI, from 2018 onward) generates text autoregressively and powers conversational AI, code generation, and document summarization. Both architectures pre-train on massive text corpora, then fine-tune for specific tasks — or, in the case of large language models like GPT-4 and Claude, perform tasks with in-context instructions alone.

For enterprise data teams, the practical consequence is that NLP went from a specialist tool requiring months of custom model training to a capability available through APIs. A data team can deploy natural language search over their catalog without training a model from scratch — they connect a pre-trained language model to their governed metadata and let the model handle the language understanding.

NLP in the Enterprise

Enterprise NLP applications cluster around five use cases where language understanding creates measurable value.

Natural language data catalog search. Users type "customer churn data for EMEA" into a search bar instead of browsing folder hierarchies or guessing table names. NLP parses the query, identifies the intent (find datasets), the domain (customer churn), and the filter (EMEA region), then matches against catalog metadata. This works only when the catalog contains rich, governed descriptions — a search engine over empty metadata fields returns nothing useful.

Automated metadata tagging and classification. NLP models scan column names, sample data, and existing descriptions to auto-classify datasets: PII detection, domain assignment (finance, HR, marketing), and sensitivity labeling. A column named "ssn" gets flagged as personally identifiable information; a table with columns like "diagnosis_code" and "patient_id" gets classified as healthcare data. This automation scales metadata enrichment beyond what manual curation can achieve.

Conversational BI and text-to-SQL. Business users ask "What drove the revenue drop in March?" and the system generates a SQL query, runs it, and returns a chart with commentary. AI-powered BI tools use NLP to translate natural language into database queries, mapping business terms to table and column names. The quality depends on the semantic layer providing correct term-to-column mappings.

Document extraction for compliance. Banks process thousands of loan documents, regulatory filings, and contracts daily. NLP extracts key terms, dates, obligations, and named entities from unstructured text — work that previously required manual review. Named entity recognition identifies parties and amounts; relation extraction maps obligations to responsible entities.

Sentiment analysis for customer data. NLP classifies customer feedback, support tickets, and social media mentions by sentiment and topic. A spike in negative sentiment around "billing" in the support queue triggers automatic escalation. The analysis runs continuously across millions of text records — a scale impossible for human reviewers.

NLP and Data Catalogs

NLP transforms data discovery from a manual browsing exercise into a conversational search experience. The difference is not cosmetic — it changes who can find data and how fast.

Traditional data discovery requires knowing where to look. A data engineer searching for churn data must know the right database, schema, and table naming convention. With NLP-powered search, any business user types "customer churn rate" and the system uses semantic matching to find related datasets — even when the actual table is named "cust_attrition_monthly" and the column is called "churn_flag." Semantic search finds conceptual matches, not just string matches.

NLP also powers automated glossary enrichment. When new datasets land in the catalog, NLP models analyze column names, data types, and sample values to suggest business term mappings. A column named "ltv_12m" in a marketing table gets matched to the glossary entry "Customer Lifetime Value (12-month)." These suggestions route to data stewards for approval, combining machine speed with human judgment.

Entity extraction auto-tags datasets with domain labels, owner suggestions, and compliance classifications. A RAG pipeline connected to the catalog can answer questions like "Which datasets contain revenue data for the APAC region?" by retrieving catalog entries, glossary definitions, and lineage records — then synthesizing a grounded answer that cites specific tables and their owners.

Challenges and Limitations

NLP has made enormous progress, but production systems still hit consistent failure modes.

Ambiguity and context. "Apple earnings" could mean the company's financial results or the revenue from selling fruit — disambiguation depends on context that may not be present in a short query. In data catalogs, the same word often means different things in different departments: "customer" in marketing includes leads, while "customer" in finance means paying accounts only. NLP systems need a business glossary to resolve this kind of domain-specific ambiguity, not just general-purpose language understanding.

Bias in language models. Language models trained on internet text absorb the biases in that text. A sentiment classifier trained on product reviews may systematically rate reviews in non-standard English dialects as more negative. For enterprise applications, this means NLP outputs need human review in high-stakes contexts — hiring, credit, compliance — where biased language processing produces biased business decisions.

Domain vocabulary. Generic NLP models don't understand industry jargon. "CDO" means Chief Data Officer in a governance conversation and Collateralized Debt Obligation in a finance conversation. Without fine-tuning or RAG-style grounding in a business glossary, NLP systems make systematic errors on domain-specific terminology. This is particularly acute in data catalogs where column names, table names, and metric definitions use organization-specific shorthand.

Compute cost at scale. Running transformer models across millions of documents or queries requires significant GPU resources. Organizations must balance between hosting models internally (higher cost, more control) and using cloud APIs (lower upfront cost, data leaves the perimeter). For metadata enrichment tasks that run across the entire catalog, batch processing with smaller specialized models often outperforms real-time inference with large general-purpose models.

Tools and Platforms

Open source. spaCy provides production-grade NLP pipelines for entity recognition, classification, and dependency parsing — fast and well-suited for batch processing. Hugging Face Transformers gives access to thousands of pre-trained models (BERT, RoBERTa, T5, Llama) with a unified API for fine-tuning and inference. Both libraries run on-premises, which matters for organizations with data residency requirements.

Cloud APIs. Azure Text Analytics, Google Cloud Natural Language API, and AWS Comprehend offer managed NLP services that handle scaling, model updates, and infrastructure. They trade customization for speed of deployment: a team can add sentiment analysis to an application in an afternoon, but fine-tuning the underlying model for domain-specific vocabulary requires different tooling.

LLM providers. OpenAI (GPT-4), Anthropic (Claude), and open-source alternatives (Llama, Mistral) provide general-purpose language understanding through APIs. These models handle a wide range of NLP tasks without task-specific training, making them versatile for prototype-to-production workflows. The tradeoff is cost per token and the need for prompt engineering to get consistent results.

How Dawiso Uses NLP

Dawiso applies NLP across its data catalog to make data discovery conversational rather than manual. Natural language search lets users find datasets, glossary terms, and lineage records by describing what they need in plain English — the system handles the mapping from business language to catalog metadata.

AI-powered metadata enrichment auto-generates descriptions from column names, data types, and sample data. When a new dataset arrives with columns like "cltv_3yr" and "acq_channel," NLP models suggest human-readable descriptions and match the columns to existing business glossary terms. Data stewards review and approve — combining machine speed with human accuracy.

Semantic matching links related business terms across the catalog. "Customer attrition" and "churn rate" get connected as synonyms, so a search for either term returns the same governed datasets. Through the Context Layer and MCP, NLP-powered AI agents can query the catalog in natural language and receive governed, contextual answers — grounded in catalog metadata rather than general training data.

Conclusion

NLP turned human language from an input humans could understand into an input machines can process. For enterprise data teams, this means the interface between people and data infrastructure no longer requires SQL fluency or catalog-browsing expertise. But NLP-powered search and enrichment are only as good as the metadata they operate on. A semantic search engine over an empty catalog returns nothing. A text-to-SQL system without governed term mappings generates wrong queries confidently. NLP is the interface layer; data governance is the substance beneath it.