Your Regression Suite Is Probably Testing the Wrong Things

Home > Blog > Your Regression Suite Is Probably Testing the Wrong Things

A few weeks ago we asked where D365 teams go after RSAT.

The conversations that came back surfaced a more uncomfortable question — one most teams haven’t answered yet:

What are you actually testing right now, and does it match what your business cannot afford to break?


Imagine This

Your team executes:

✅ 2,000 test cases

✅ 98% pass rate

✅ Release approved

Three days later:

❌ Customer orders fail

❌ Finance posting breaks

❌ Warehouse integrations stop working

What happened?

The answer is uncomfortable.

Your regression suite was working.

It was just testing the wrong things.


Quick Self-Check

How many of these can you confidently say yes to?

☐ We know which test cases protect revenue

☐ We know which test cases protect compliance

☐ We know which test cases protect customer experience

☐ We know which test cases protect integrations

If the answer is fewer than three — your suite is probably measuring volume, not risk.


Test Cases ≠ Coverage

The pattern we see most often in D365 environments:

Article content

What This Looks Like In Practice

A team we spoke with recently reported thousands of automated test cases. Strong number on a slide. Real confidence going into a release.

During that release, one business process failed.

A single end-to-end workflow — the one that generated revenue.

Why?

Because the suite heavily validated screens and field-level behaviour, but barely touched the workflow itself. Lots of clicks tested. Almost no business outcomes.

The issue wasn’t test volume.

It was test prioritisation.


The 3-Layer Coverage Model

A simple way to re-sort what you’re testing:

Layer 1 — Revenue Processes

Order-to-cash. Procure-to-pay. Invoice processing.

If these break, money stops moving. Always test. Always automate. Always validate after every Wave update.


Layer 2 — Operational Processes

Approvals. Inventory. Planning. Reporting.

Critical but not catastrophic if delayed. Test thoroughly, but the bar is lower than Layer 1.


Layer 3 — Everything Else

Useful. Worth testing. But not equally critical.

The mistake most D365 teams make is testing all three layers with the same intensity — which means Layer 1 gets the same attention as Layer 3, and the suite produces confidence that doesn’t match real risk.


The Question That Should Drive Your Strategy

Don’t ask:

How many test cases do we have?

Ask:

Which business processes can we not afford to break?

Those are rarely the same conversation.

And once you’ve answered the second question, the first one starts to matter less.


Where Crestech Sits in This

We work with D365 F&O teams on exactly this — re-sorting regression coverage so it maps to business risk, not screen count.

If your team is reviewing what to test (and what to stop testing) ahead of the next Wave, we’d happily spend 30 minutes walking through how other D365 teams are structuring it.

No pitch. Just a working conversation.


Found this useful? Share it with your QA lead, ERP manager or anyone reviewing Wave readiness. The “test count vs business risk” conversation is overdue across most D365 environments.


#Dynamics365 #D365FO #ERPTesting #RegressionTesting #QualityAssurance #MicrosoftDynamics #D365Community #WaveRelease #TestStrategy #CIO #CFO

Like the blog? Spread the word

Latest Insights

Most d365 teams dont know what theyre replacing

Read More >>

Microsoft called rsat feature complete what does that mean for d365 teams

Read More >>

Wave 1 changed retail d365 has your qa plan kept up

Read More >>

Most d365 regression suites are growing very few are becoming smarter

Read More >>