Natural Language Processing (NLP)

Natural language processing is the AI discipline that enables machines to read, interpret, and generate human language. It powers search engines, voice assistants, machine translation, and the text-to-SQL interfaces that let business users query data without writing code.

For data-driven organizations, NLP transforms how people interact with their data infrastructure. Instead of browsing a catalog manually or asking an analyst to write a query, a product manager types "show me customer churn by region for Q3" and gets a chart. That interaction depends on NLP parsing intent, mapping terms to governed definitions, and generating the right query - a process that breaks down the moment the underlying metadata is ambiguous or missing.

How NLP Works

An NLP pipeline transforms raw text into structured meaning through a series of stages. Each stage builds on the previous one, and errors at any stage propagate forward.

Tokenization splits text into units the model can process - words, subwords, or characters. When a user types "Show me quarterly revenue by region," the tokenizer breaks it into tokens that map to the model's vocabulary. Modern tokenizers like BPE (byte pair encoding) handle unknown words by splitting them into known subword pieces rather than discarding them.

Embedding converts each token into a dense numerical vector that captures its semantic meaning. Words with similar meanings cluster near each other in this vector space: "revenue" sits close to "income" and "sales," while "region" sits near "territory" and "geography." These embeddings are what make semantic search possible - matching meaning rather than exact strings.

Model processing reads the full sequence of embeddings and produces an output that depends on the task. For classification, it predicts a label. For generation, it produces the next token. For text-to-SQL, it translates the natural language input into a database query. The model needs context beyond the words themselves: when the user says "revenue," the model must know which table holds the canonical revenue metric and how it's calculated - information that comes from a business glossary, not from the text itself.

Output varies by task: a translated sentence, a SQL query, a classification label, a generated paragraph, or extracted entities like company names and dates. The quality of this output depends entirely on the quality of every preceding stage.

The Transformer Revolution

Before 2017, the dominant NLP models (RNNs and LSTMs) processed text sequentially - reading one word at a time, left to right. This limited their ability to capture relationships between distant words in a sentence. The transformer architecture changed that by introducing self-attention: a mechanism that lets the model weigh every word against every other word simultaneously. A sentence like "The bank by the river was eroding" requires understanding that "bank" means a riverbank, not a financial institution - context that depends on words appearing later in the sentence.

This single architectural change enabled the jump from narrow NLP tools to general-purpose language models. BERT (Google, 2018) reads text bidirectionally and excels at understanding tasks: classification, entity recognition, semantic search. GPT (OpenAI, from 2018 onward) generates text autoregressively and powers conversational AI, code generation, and document summarization. Both architectures pre-train on massive text corpora, then fine-tune for specific tasks - or, in the case of large language models such as OpenAI's GPT models and Anthropic's Claude, perform tasks with in-context instructions alone.

For enterprise data teams, the practical consequence is that NLP went from a specialist tool requiring months of custom model training to a capability available through APIs. A data team can deploy natural language search over their catalog without training a model from scratch - they connect a pre-trained language model to their governed metadata and let the model handle the language understanding.

NLP in the Enterprise

Enterprise NLP applications cluster around five use cases where language understanding creates measurable value.

Natural language data catalog search. Users type "customer churn data for EMEA" into a search bar instead of browsing folder hierarchies or guessing table names. NLP parses the query, identifies the intent (find datasets), the domain (customer churn), and the filter (EMEA region), then matches against catalog metadata. This works only when the catalog contains rich, governed descriptions - a search engine over empty metadata fields returns nothing useful.

Automated metadata tagging and classification. Some catalog and classification tools use NLP classifiers to scan column names, sample data, and existing descriptions and suggest dataset classifications: PII detection, domain assignment (finance, HR, marketing), and sensitivity labeling. In such a tool, a column named "ssn" gets flagged as personally identifiable information; a table with columns like "diagnosis_code" and "patient_id" gets classified as healthcare data. This kind of automation scales metadata enrichment beyond what manual curation can achieve.

Conversational BI and text-to-SQL. Business users ask "What drove the revenue drop in March?" and the system generates a SQL query, runs it, and returns a chart with commentary. AI-powered BI tools use NLP to translate natural language into database queries, mapping business terms to table and column names. The quality depends on the semantic layer providing correct term-to-column mappings.

Document extraction for compliance. Banks process thousands of loan documents, regulatory filings, and contracts daily. NLP extracts key terms, dates, obligations, and named entities from unstructured text - work that previously required manual review. Named entity recognition identifies parties and amounts; relation extraction maps obligations to responsible entities.

Sentiment analysis for customer data. NLP classifies customer feedback, support tickets, and social media mentions by sentiment and topic. A spike in negative sentiment around "billing" in the support queue triggers automatic escalation. The analysis runs continuously across millions of text records - a scale impossible for human reviewers.

NLP and Data Catalogs

NLP transforms data discovery from a manual browsing exercise into a conversational search experience. The difference is not cosmetic - it changes who can find data and how fast.

Traditional data discovery requires knowing where to look. A data engineer searching for churn data must know the right database, schema, and table naming convention. With NLP-powered search, any business user types "customer churn rate" and the system uses semantic matching to find related datasets - even when the actual table is named "cust_attrition_monthly" and the column is called "churn_flag." Semantic search finds conceptual matches, not just string matches.

NLP tools can also power automated glossary enrichment. When new datasets land in a catalog that offers it, NLP models analyze column names, data types, and sample values to suggest business term mappings. A column named "ltv_12m" in a marketing table gets matched to the glossary entry "Customer Lifetime Value (12-month)." These suggestions route to data stewards for approval, combining machine speed with human judgment.

Some tools also use entity extraction to auto-tag datasets with domain labels, owner suggestions, and compliance classifications. A RAG pipeline connected to the catalog can answer questions like "Which datasets contain revenue data for the APAC region?" by retrieving catalog entries, glossary definitions, and lineage records - then synthesizing a grounded answer that cites specific tables and their owners.

Challenges and Limitations

NLP has made enormous progress, but production systems still hit consistent failure modes.

Ambiguity and context. "Apple earnings" could mean the company's financial results or the revenue from selling fruit - disambiguation depends on context that may not be present in a short query. In data catalogs, the same word often means different things in different departments: "customer" in marketing includes leads, while "customer" in finance means paying accounts only. NLP systems need a business glossary to resolve this kind of domain-specific ambiguity, not just general-purpose language understanding.

Bias in language models. Language models trained on internet text absorb the biases in that text. A sentiment classifier trained on product reviews may systematically rate reviews in non-standard English dialects as more negative. For enterprise applications, this means NLP outputs need human review in high-stakes contexts - hiring, credit, compliance - where biased language processing produces biased business decisions.

Domain vocabulary. Generic NLP models don't understand industry jargon. "CDO" means Chief Data Officer in a governance conversation and Collateralized Debt Obligation in a finance conversation. Without fine-tuning or RAG-style grounding in a business glossary, NLP systems make systematic errors on domain-specific terminology. This is particularly acute in data catalogs where column names, table names, and metric definitions use organization-specific shorthand.

Compute cost at scale. Running transformer models across millions of documents or queries requires significant GPU resources. Organizations must balance between hosting models internally (higher cost, more control) and using cloud APIs (lower upfront cost, data leaves the perimeter). For metadata enrichment tasks that run across the entire catalog, batch processing with smaller specialized models often outperforms real-time inference with large general-purpose models.

Tools and Platforms

Open source. spaCy provides production-grade NLP pipelines for entity recognition, classification, and dependency parsing - fast and well-suited for batch processing. Hugging Face Transformers gives access to more than three million pre-trained models on the Hugging Face Hub (BERT, RoBERTa, T5, Llama) with a unified API for fine-tuning and inference. Both libraries run on-premises, which matters for organizations with data residency requirements.

Cloud APIs. Azure Language in Foundry Tools (formerly Azure AI Language), Google Cloud Natural Language API, and Amazon Comprehend offer managed NLP services that handle scaling, model updates, and infrastructure. They trade customization for speed of deployment: a team can add entity recognition to an application in an afternoon, but fine-tuning the underlying model for domain-specific vocabulary requires different tooling.

LLM providers. OpenAI's GPT models, Anthropic's Claude, and open-source alternatives (Llama, Mistral) provide general-purpose language understanding through APIs. These models handle a wide range of NLP tasks without task-specific training, making them versatile for prototype-to-production workflows. The tradeoff is cost per token and the need for prompt engineering to get consistent results.

How Dawiso Uses NLP

Dawiso applies language models at a few documented points in its data catalog, each grounded in catalog metadata. Ask Dawiso AI lets users ask questions in plain language and get answers based on the catalog's metadata and documentation, instead of first working out where the right asset lives.

AI-generated descriptions produce a draft that the user reviews and saves, during metadata ingestion at database, schema, view, and table level. The prompt is configurable per attribute type. In Dawiso, PII and confidentiality classifications are set by data stewards.

The AI Dictionary groups attributes into shared business and attribute domains, so related assets carry a common vocabulary. Through the Context Layer and MCP, external AI tools and agents can access Dawiso information and work from governed catalog metadata rather than general training data.

Conclusion

NLP turned human language from an input humans could understand into an input machines can process. For enterprise data teams, this means the interface between people and data infrastructure no longer requires SQL fluency or catalog-browsing expertise. But NLP-powered search and enrichment are only as good as the metadata they operate on. A semantic search engine over an empty catalog returns nothing. A text-to-SQL system without governed term mappings generates wrong queries confidently. NLP is the interface layer; data governance is the substance beneath it.

Sources

• Vaswani et al., Attention Is All You Need (NIPS 2017, now NeurIPS).
• Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018).
• OpenAI, Improving language understanding with unsupervised learning (2018).
• Sennrich et al., Neural Machine Translation of Rare Words with Subword Units (ACL 2016; byte pair encoding for NLP).
• MIT Sloan School of Management, Tapping the power of unstructured data (2021; analyst estimates that 80% to 90% of data is unstructured).
• Microsoft Learn, What is Azure Language in Foundry Tools
• Hugging Face, Models (model count as of October 2026).
• Dawiso Help, AI features and integrations in Dawiso
• Dawiso Help, PII Level and Confidentiality