Agile Development

Agile development is an iterative approach to software delivery where teams ship working increments in short cycles - in Scrum, sprints of one month or less, often two weeks - gather feedback, and adapt. It replaced waterfall's long planning-then-building sequence with continuous delivery and continuous learning. For data teams, agile methods shape how data products, governance policies, and catalog features are prioritized and delivered.

Core Principles

The Manifesto for Agile Software Development was written in February 2001 by seventeen practitioners who met at Snowbird, Utah (history). It defines four values:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

The manifesto adds that while there is value in the items on the right, its authors value the items on the left more. Twelve supporting principles follow from these values. The practical ones: deliver working software frequently, from a couple of weeks to a couple of months, with a preference for the shorter timescale. Welcome changing requirements, even late in development. Business people and developers must work together daily throughout the project. Build projects around motivated individuals, give them the environment and support they need, and trust them. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

These principles are not aspirational slogans. They are design constraints. A team that delivers every two weeks but ignores feedback is not agile. A team that holds daily standups but ships once a quarter is performing ceremony without substance.

Scrum, Kanban, and XP

Scrum is one of the most widely used agile frameworks. The 2020 Scrum Guide by Ken Schwaber and Jeff Sutherland defines it. Work is organized into Sprints, fixed-length events of one month or less (two weeks is a common choice). Three accountabilities make up the Scrum Team: the Product Owner orders the Product Backlog and maximizes the value of the product, the Scrum Master establishes Scrum and helps the team improve, and the Developers create the Increment. Five events structure the work: the Sprint itself, which contains Sprint Planning (what can be done and how), the 15-minute Daily Scrum (inspect progress toward the Sprint Goal), the Sprint Review (inspect the outcome with stakeholders) and the Sprint Retrospective (improve quality and effectiveness). Three artifacts carry the work: the Product Backlog, the Sprint Backlog and the Increment.

Kanban drops the time-box. Work flows continuously through a visual board with columns (To Do, In Progress, Done). The key constraint is WIP limits - capping how many items can be in progress simultaneously. This prevents overcommitting and forces the team to finish work before starting new work. Kanban works well for teams handling unpredictable work like support requests or data incident response.

Extreme Programming (XP), developed by Kent Beck in the late 1990s, focuses on engineering practices rather than project management. Its core: test-driven development (write the test before the code), pair programming (two developers, one screen), continuous integration (merge and test code multiple times per day), and relentless refactoring. XP practices complement Scrum - many teams run Scrum ceremonies with XP engineering discipline underneath.

Agile for Data Teams

Software teams have worked with agile methods since the manifesto of 2001. Many data teams adopted them later. The principles transfer directly, but the implementation looks different.

Data product backlog. Instead of user stories for application features, the backlog contains data deliverables: datasets to publish, pipelines to build, quality rules to implement, glossary terms to define. Each item has acceptance criteria - "the customer dimension table includes all active accounts, refreshes daily by 6am, and passes five quality checks" - not vague descriptions like "improve data quality."

Sprint-based governance rollout. Rather than deploying a governance framework across all domains at once, teams govern one domain per sprint. An illustrative plan: Sprint 1, define and document the 30 most-used business terms in finance. Sprint 2: link those terms to source tables and add quality rules. Sprint 3: move to the next domain. Each sprint delivers measurable progress.

Cross-functional data squads. Data engineers, analysts, data stewards, and business users working in the same sprint toward a shared goal. The business user defines what "revenue" means, the engineer builds the pipeline, the steward documents it in the catalog, and the analyst validates the output. No handoffs across team boundaries.

Iterative catalog development. Ship catalog features incrementally - lineage for the top 10 pipelines first, then expand. A glossary with 50 well-defined terms is more useful than a glossary with 500 placeholder entries. BI consumers start getting value immediately instead of waiting for a "complete" solution.

Common Pitfalls

Agile theater. The team holds daily standups, runs sprints, and uses a board - but never actually adapts based on feedback. Requirements are locked at sprint start and never change. Retrospectives produce action items that nobody follows up on. The ceremonies happen; the mindset does not.

Backlog grooming neglect. An unmanaged backlog of 500+ items is not agile - it is a graveyard of good intentions. If no one has looked at a backlog item in three months, it should be deleted or archived. The backlog is a prioritized list of what to build next, not a wish list.

Velocity as a target. Velocity measures how much work a team completes per sprint. When management treats velocity as a performance metric ("increase velocity by 20%"), teams inflate story points. A team that completed 40 points last sprint now completes 50 - not because they delivered more, but because they re-estimated.

Missing technical practices. Adopting Scrum events while neglecting the internal quality of the software is what Martin Fowler called "Flaccid Scrum" in 2009: progress slows because the code base becomes hard to change. Scrum itself prescribes no engineering practices; continuous integration, automated testing and refactoring make the work shippable. Without both, sprints end with "almost done" increments that pile up.

Measuring Agile Effectiveness

The DORA metrics (from the DevOps Research and Assessment program) are a widely used way to measure software delivery performance, which shows whether agile practices are paying off in delivery. DORA groups them into throughput and instability:

Change lead time - the time from when a change is committed to version control until it is running in production.

Deployment frequency - how often the team deploys to production. Higher frequency usually means smaller batches, which means lower risk per release.

Failed deployment recovery time - how long it takes to fix a failed deployment that needs urgent, hands-on intervention.

Change fail rate - the proportion of deployments that need urgent follow-up, such as a rollback or hotfix. Shipping fast only matters if what you ship works.

Deployment rework rate - the proportion of deployments that were unplanned and happened because of a production incident.

Some teams also track cycle time, the time from work started to work done, which is not a DORA metric but shows how fast work moves once it is picked up. These metrics function as KPIs for engineering teams. They are measurable, owned (by the team), tied to outcomes (not activity), and actionable. Tracking throughput and instability together helps teams avoid the trap of optimizing speed at the expense of quality, or vice versa. A/B testing process changes against these metrics helps teams distinguish real improvements from noise.

Agile and Data Governance

Traditional data governance programs often followed waterfall-style rollouts: long planning cycles and "big bang" deployments of governance frameworks. A typical failure pattern is a governance team that spends months writing a comprehensive policy document, then finds that few people follow it.

Agile governance starts with one data domain, proves value, and expands. Instead of abstract deliverables ("governance framework document v3.2"), governance sprints deliver measurable outcomes, for example: "100 business terms defined and linked to source tables," "data quality rules covering 95% of the finance pipeline," or "lineage mapped for the top 20 dashboards."

The sprint structure forces prioritization. A governance team cannot govern everything simultaneously, so they start with the data domain that causes the most pain - usually the one where conflicting definitions lead to conflicting dashboards. Fixing that domain first generates visible value, which builds organizational support for expanding to the next domain.

Retrospectives matter here too. After each governance sprint, the team asks: Did the business terms we defined actually get used? Did the quality rules catch real issues? Are the documented owners actually responding to alerts? If not, the team adjusts before repeating the same approach across five more domains.

How Dawiso Supports Agile Data Teams

Dawiso does not run sprints, backlogs or delivery metrics; teams keep those in their own planning tools. What Dawiso provides is the data catalog and business glossary that governance sprints produce and improve, and it supports incremental adoption. Teams can start with one data domain, add business terms, indicators (with description, calculation, owner and steward) and rules sprint by sprint, and move each definition through a draft, review and approved workflow. Lineage coverage can grow the same way: it comes from connectors, dbt and SQL parsing, which is not switched on by default, or from manually added relations.

The catalog also shows a steward what is still missing, such as objects without a description or an owner, which the team can turn into backlog items in its own tracker.

Through Dawiso's Model Context Protocol (MCP) server, an AI agent can search the catalog, read object details, lineage and samples, and use write tools to update metadata. That lets an agent help list tables without descriptions or columns without business terms; prioritizing the resulting work stays with the team.

Conclusion

Agile development is not a set of ceremonies. It is a commitment to short feedback loops, continuous delivery, and adaptation based on evidence. The frameworks - Scrum, Kanban, XP - provide structure, but the principles matter more than the process. For data teams, agile methods offer an alternative to big-bang governance rollouts whose value only shows at the end. Start with one domain, deliver something real every sprint, and expand based on what works.

Sources

• Kent Beck et al., Manifesto for Agile Software Development, 2001.
• Agile Manifesto authors, Principles behind the Agile Manifesto, 2001.
• Jim Highsmith, History: The Agile Manifesto, agilemanifesto.org.
• Ken Schwaber and Jeff Sutherland, The Scrum Guide, November 2020.
• Martin Fowler, Flaccid Scrum, 2009.
• DORA, DORA's software delivery performance metrics, dora.dev.
• Dawiso Help, Business glossary attributes and components, help.dawiso.com.
• Dawiso Help, Workflows overview, help.dawiso.com.