Case Study
A Learning Platform
The blocker and critical defects from one release, ranked by how quickly customers would notice each kind.
- Workflow and Navigation: people got stuck and couldn’t move forward.
- Configuration and Live Updates: changes to live setups didn’t apply.
- Performance and Scalability: things slowed or failed with large amounts of data.
- Interruption Handling: work was lost or wouldn’t save when paused.
- Progress and Status Tracking: status didn’t update after work was completed.
- Access and Security: access that should have been removed still worked.
The riskiest changes are the ones that affect how people move through a workflow. When those break, people get stuck straight away, so they are the first thing customers notice. That is where most of the hand testing goes.
Case Study
An Email Security Platform
From the QA lead who has tested it for 16 years.
The defects customers find first come from changes to the core detection engines, antivirus, and spam. A new engine version can change the spam score of a message, so mail that was quarantined last week is delivered this week. An antivirus update can change the verdict on a file type.
Those changes touch every customer’s traffic at once, so every engine update gets focused regression before it ships.
Most production issues on the product are not in the core logic. They are in customer-specific configurations and in upgrades from older versions, where a mismatch can let a policy bypass through even though the fix itself works.
That is why every release on this product is also tested against customer-specific configurations and upgrades from older versions.