Manufacturing Software Testing: A Practical Guide for MES, ERP and QMS Teams

Monday, 06:00. The new manufacturing execution system went live over the weekend. The first work order of the shift arrives from ERP, the MES rejects it – the unit of measure is "KAR", a carton code nobody mapped – and line 3 stands still while a planner searches for the conversion table. Nothing crashed. Every test passed. The plant is losing money by the minute anyway.

How much money is no longer a guess. Siemens' True Cost of Downtime 2024 puts an unplanned hour in automotive at $2.3 million, and unplanned downtime overall at 11% of revenue for the world's 500 largest companies – roughly $1.4 trillion a year. A growing share of those hours are caused not by a bearing or a motor but by software that sits on the critical path of the line.

Manufacturing software testing is mostly about the seams. ERP talks to MES, MES talks to SCADA, SCADA talks to PLCs, and engineering data arrives from CAD and PLM upstream of all of it. Each layer is tested by its vendor; the hand-offs between them are tested by nobody until go-live.

This guide walks down the ISA-95 automation pyramid from the planning layer to the machine, tests each layer, and spends most of its attention on the arrows between them. We have been doing this work – engineering design platforms, quality management systems, telecom and connectivity software, planning tools – since 2010; what we offer as a service is on our manufacturing application testing services page. This article is the engineering behind it.

Ready to take your software quality to the next level?

Schedule a call

Why software testing in the manufacturing industry is a different job

Testing manufacturing software differs from testing a web product in six ways, and each one changes what the test plan should contain.

Our rule after fifteen years of these projects: test the master data and the interfaces first; the screens were never the problem.

Level 4 — ERP and MRP: where the plan is made

The planning layer decides what to make, buy and when; every defect here becomes a defect on the floor a few days later.

What breaks

The industry's track record here is poor. Gartner expects that more than 70% of recently implemented ERP initiatives will fail to fully meet their business goals by 2027, and up to 25% will fail catastrophically; Panorama Consulting's 2024 ERP Report found a third of projects over budget, most often because the work was underestimated. Testing is a large part of the underestimated work.

Testing ERP for manufacturers

• Regress the processes as the plant runs them, not as the vendor demos them. Order-to-cash and procure-to-pay flows with your products, your routings, your scrap factors and your intercompany transfers, scripted end to end.

• Build master-data validation suites. BOM completeness and effectivity dates, routing–work-centre consistency, UoM conversion round-trips (carton → piece → carton must return the same quantity), lot and serial rules per material type.

• Inventory the customisations and regress them explicitly. Custom objects, extensions and integrations are where upgrades break; if the list does not exist, the first job is to make it.

• Treat an upgrade as a release of every integration, because that is what it is for the layers below.

The general method for ERP regression – scope, data, automation, cut-over – is in our ERP testing guide; Oracle-specific suites have their own quirks, covered in our practical guide to testing Oracle applications.

Material requirements planning: testing the explosion

Material requirements planning software is a calculation engine, and the right way to test a calculation engine is with golden answers.

• Golden-plan testing. A fixed data set – demand, on-hand inventory, open orders, BOMs, lead times – with the expected planned orders computed and approved once. Every release re-runs the plan and diffs it against the golden result. Any change is either a defect or a deliberate rule change that updates the golden file.

• A data-driven matrix for multi-level BOMs, phantoms and alternates, scrap and yield factors, safety stock, lot-sizing rules (fixed, lot-for-lot, period) and minimum/multiple order quantities.

• Calendar and capacity. Shift patterns, plant holidays, maintenance windows; the plan must respect them, and the tests must include the week with the public holiday.

What we cover for MRP and planning modules specifically is described under MRP testing services.

From the field – the UoM table. The pattern we see most often at Level 4 is not a calculation bug but a mapping gap: a unit, a plant code or a material type that exists in ERP and has no counterpart in the system below. It passes every unit test because each system is internally consistent. We now generate the cross-system master-data reconciliation as the very first automated suite on any manufacturing project – before a single screen is tested – and run it against every environment refresh.

Level 3 — MES and QMS: where the plan meets the line

The execution and quality layer turns planned orders into work, records what actually happened, and decides whether it was good.

What breaks

Testing a manufacturing execution system

• Work-order lifecycle with every exception. Release → dispatch → start → pause → complete → close is the happy path; the tests that matter are hold, rework, split, merge, scrap, partial completion and reversal.

• Traceability as a round trip. Build a product in the test environment from receiving through packing, then trace forward (which finished units contain lot X?) and backward (which lots are in serial Y?) and assert both are complete. Run it on every release.

• Operator UI on the real terminals. Touch targets with gloves, barcode-wedge input timing, screen-lock and re-authentication policies, stability across a 12-hour shift with no restart.

• Electronic signatures and audit trail. Who did what, when, and why – unalterable, time-synchronised, exportable. Where 21 CFR Part 11 applies, this is a validation deliverable, not a feature test.

Testing a quality management system

QMS software testing is workflow testing with a combinatorial twist: roles × workflow states × document types multiply fast, and the regulator cares about every combination.

• Workflow engine. Non-conformance → CAPA → change control → training assignment, with role-based approvals, delegation, escalation and SLA timers. Script the state machine, then fuzz the transitions.

• Document control. Versioning, effective dates, controlled and uncontrolled copies, obsolescence, watermarks – tested as data rules, not as a UI.

• Inspection plans and sampling. Acceptance-quality-limit (AQL) tables as data-driven tests; statistical process control (SPC) calculations compared against reference values; out-of-spec and out-of-trend triggers.

• Supplier quality and audits. Scorecards, audit schedules, findings → CAPA linkage.

• Mobile and offline. Inspections increasingly happen on a tablet in a place without Wi-Fi; the sync-conflict scenarios need the same rigour as the workflow ones.

Our scope for quality platforms – workflows, document control, inspections, validation evidence – is on the QMS testing services page.

From our projects – a quality management platform for construction. The product handles checklist inspections, punch lists, daily progress reports and QA/QC plans for construction companies and inspectors – the QMS pattern applied to a site instead of a plant, on mobile, often offline. When we joined there were no automated tests and regression was slow and redundant. Over three years the team added 1,500+ test cases, built the automation suite from scratch – 500+ end-to-end scripts covering 100% of the main user flows – and introduced daily runs in production and development. User complaints fell by 90% within five months. The hard part was the role × workflow-state combinatorics, which is exactly what a QMS is. Read the case study.

Levels 2–0 — SCADA, PLCs and the machines: hardware in the loop

Below the MES, the software talks to physical equipment through supervisory control (SCADA), human-machine interfaces (HMI) and programmable logic controllers (PLC), and the failure modes become physical.

What breaks

Integration testing between MES and the plant floor

• Tag-level contract tests. For every OPC UA, MQTT or Modbus mapping: name, data type, engineering units, scaling, and behaviour on bad quality or stale values. A simulated PLC drives every state – including nonsense values – and the test asserts what the MES records.

• Clock-sync and ordering tests. Assert NTP/PTP synchronisation across layers and that event streams are ordered by source time, not arrival time.

• A device matrix on real hardware. Scanners, scales, printers, HMIs, with firmware versions inventoried and tracked like operating-system versions. Test the oldest unit still in production as well as the newest.

• Alarm and exception handling. Flood scenarios, acknowledgement and shelving rules, priority filtering, and what the operator sees when 200 alarms arrive in ten seconds.

• OT security (IEC 62443). Network segmentation between IT and OT zones, least-privilege service accounts, no default credentials on devices, patched remote-access paths – tested, not assumed from a diagram.

Firmware, protocol and device-level techniques – bench setups, simulators, timing analysis – are covered in our guide on how to test embedded systems; the plant-floor seam adds the MES side of every one of those tests.

Upstream of the pyramid — CAD, CAM and engineering data

Before any work order exists, a model does, and defects in design software reach the shop floor as scrap. In a CoLab survey, 90% of engineering leaders said they had delayed product launches because of late-stage design changes – many of them discovered when the design met manufacturing.

What breaks

Testing CAD and CAM software

CAD quality assurance services have one central technique – golden-model regression – and everything else supports it.

• Golden-model regression. A library of reference models and assemblies that is opened, regenerated and compared on every build: geometry hashes, mass properties, bounding boxes, feature counts, drawing diffs. This is what "automated CAD checking" means in practice, and it catches kernel and feature regressions that no manual test would.

• A format matrix. STEP AP203/AP214/AP242, IGES, DXF/DWG, STL, native formats, across software versions and in both directions (import and export round trips).

• CAM regression. Toolpaths compared against golden G-code; post-processor outputs diffed per machine; simulation compared against a physical test cut on a reference part at least once per major release.

• Performance. Large-assembly load and regeneration time, memory under long sessions, GPU and driver matrix for rendering.

• Host compatibility. For plug-ins and add-ins: AutoCAD, SolidWorks, Inventor, Revit and the rest, each new host release treated as a regression trigger.

The specifics of how we structure golden-model, format and host-compatibility testing are described under CAD application testing.

From our projects – Graitec. Graitec builds software for the architecture, engineering and construction industry across design, simulation, fabrication and data management. For a new cloud-based structural steel design platform we provided combined manual and automated QA embedded in the Agile teams and the CI/CD pipeline: 900+ manual test cases, 250+ automated scenarios, 80%+ test coverage and 280+ defects tracked to resolution. The lesson from design software: the expensive bugs are the ones that produce a plausible wrong answer – a member size, a connection detail – so every automated check compares against a known-good result, never just "it rendered". Read the Graitec case study.

The network the pyramid runs on — telecom platforms, BSS and private 5G

Plants increasingly run on private cellular networks and industrial IoT connectivity: ABI Research's 2024 survey found 44% of manufacturers deploying private 4G and 75% evaluating private 5G. The vendors that supply and operate those networks run business and operations support stacks (BSS/OSS) that need the same discipline as the plant software they connect.

What breaks

Testing BSS platforms: rating, charging, orchestration

BSS testing is a calculation-and-orchestration problem, like MRP with money attached.

• Rating and charging matrices against the tariff specification: every plan, bundle, add-on, discount and proration rule, including mid-cycle changes and roaming.

• Mediation integrity under load: CDR counts in versus records rated, with deliberate duplicates and malformed records injected.

• Order-to-activate orchestration across CRM, provisioning, network and billing, with rollback and compensation when any step fails.

• Catalogue-driven regression: a catalogue change triggers the suite for every product that references the changed component.

• TM Forum Open API contract tests on every interface that partners or internal systems consume.

Our scope for rating, charging and orchestration platforms is described under BSS platform testing. For network operations, provisioning and field-service products – the software that plans, builds and maintains the networks factories now depend on – see our telecom software testing page.

From our projects – NEXSYS-ONE. NEXSYS-ONE develops cloud-based enterprise software for telecommunications operations: network audits, asset management, fibre-optic and transmission deployments, network maintenance, plus a project-management and resource-optimisation platform. When we joined in 2021 there was no regression testing, no CI environment and an unstable product. Two full-stack QA engineers built automated regression for two applications – 1,300 automated scenarios in total, running in 1.5 hours on 15 parallel threads in a Bitbucket pipeline – and now find 70% of blocker, critical and major defects at the development stage. The lesson for asset-heavy operations software: the data model (sites, assets, work packages) is the test oracle; the UI is only the way in. Read the NEXSYS-ONE case study.

The same stack outdoors — agritech and precision agriculture

Agritech is the manufacturing pyramid with worse connectivity and seasons instead of shifts. Sensors in fields replace sensors on lines; tractors and sprayers with telematics replace PLCs; weather and satellite feeds replace the ERP forecast; the farm management system is the MES.

What to test: sensor-data simulation with noise, gaps and drift (a soil-moisture probe that reads 0 for an hour must not trigger irrigation); offline-first field apps that sync without loss or duplication after a day without signal; equipment integrations over ISOBUS and manufacturer APIs; agronomic calculations (nutrient, spray and irrigation recommendations) compared against reference models; and seasonal peak load – planting and harvest are the plant's two shutdown weekends, in reverse. Our approach to farm management, sensor and equipment software is described under agritech software testing.

Validation, go-live and the two release windows a year

Testing proves the software works; validation proves it, in writing, to a regulator – and go-live is where both are judged.

Computer system validation without the paperwork theatre

Validation does not have to mean a room full of binders. The GAMP 5 (second edition, 2022) risk-based approach, and the FDA's computer software assurance guidance, both encourage focusing effort where the patient or product risk is and accepting automated test evidence.

• Start from the URS and a risk assessment: which functions affect product quality, patient safety or data integrity? Those get the rigour; the rest gets ordinary testing.

• Map installation, operational and performance qualification (IQ/OQ/PQ) to automated and manual tests with a traceability matrix that lives in the repository, not in a spreadsheet someone updates before the audit.

• Treat the automated regression suite's signed report as validation evidence: each test ID traces to a requirement, each run is time-stamped and attributable.

• Where 21 CFR Part 11 or EU Annex 11 applies, test electronic signatures, audit trails and access controls as requirements with evidence, not as features.

Requirements that cannot be tested cannot be validated; writing them so they can is the subject of our article on why software requirements testing matters.

UAT with operators, not analysts

User acceptance testing on a manufacturing project is only acceptance if the users are the operators, supervisors and quality technicians who will run the system – on the actual terminals, with the actual scanners, during a realistic shift pattern. Scripts are written from the line's perspective ("start the next job on press 4 after a tool change") rather than the system's ("navigate to Work Orders"). How to structure and run UAT so it produces decisions rather than opinions is covered in our user acceptance testing guide.

Cut-over rehearsal

• Data migration reconciliation. Open orders by status, inventory by lot and location, work in progress (WIP) by operation, open non-conformances – counts and quantities reconciled to the unit between source and target.

• Dress rehearsals on a shutdown weekend, timed end to end, with the people who will do the real cut-over.

• Rollback criteria decided before, not during. What result at what time triggers a return to the old system, and who makes the call.

Manufacturing test software and automation: what we actually use

The tooling for manufacturing test software follows the pyramid: a different kind of tool at every layer, tied together by traceability.

What does not automate well, and should not be forced: operator ergonomics on the terminal, the physical behaviour of a scanner in a cold store, and the first production run of a genuinely new product. Those stay human.

How we set up QA for a manufacturing software project: our playbook

Fifteen years of design platforms, quality systems, planning tools and connectivity software have reduced our approach to five moves.

Two lessons we learned the expensive way. On one planning project we trusted the vendor's "standard" ERP interface specification – and discovered during the cut-over rehearsal that each of the client's plants had customised it over a decade; the rehearsal, not the production go-live, found it, which is what rehearsals are for, but we now inventory interfaces per plant, never per vendor. And on a quality platform we treated the mobile offline path as an edge case; in the field it was the main case, and the first release lost inspection photos when devices reconnected. Offline sync has been a first-class scenario in every field-app suite since.

Manufacturing software testing checklist

Level 4 — ERP and MRP

• Order-to-cash and procure-to-pay are regressed with the plant's own products, routings and transfers.

• Master-data suites validate BOM effectivity, routing consistency, UoM round trips and lot/serial rules.

• Customisations are inventoried and explicitly regressed on every upgrade.

• MRP runs are compared against a version-controlled golden plan.

• Lot-sizing, scrap, safety-stock and calendar rules are covered by a data-driven matrix.

Level 3 — MES and QMS

• Work-order lifecycle tests include hold, rework, split, scrap, partial completion and reversal.

• Forward and backward traceability round trips run on every release.

• Operator UI is tested on the real terminals across a full shift.

• Electronic signatures and audit trails are tested as requirements with evidence.

• NC → CAPA → change-control workflows are scripted for every role and state.

• Document control rules (versions, effective dates, obsolescence) are tested as data rules.

• AQL sampling and SPC calculations are compared to reference values.

• Mobile inspection apps are tested offline with sync-conflict scenarios.

Levels 2–0 — Control and devices

• Every tag mapping has a contract test, including bad-quality and stale values.

• Clock synchronisation and event ordering are asserted across layers.

• The device lab covers the oldest firmware still in production.

• Alarm floods, acknowledgement and shelving are tested.

• IEC 62443 controls (segmentation, least privilege, credentials) are verified.

Upstream — CAD/CAM

• Golden-model regression runs on every build with geometry and mass-property comparison.

• Import/export round trips are tested across the format matrix.

• Toolpaths and post-processor outputs are diffed against golden G-code.

• Plug-ins are regressed on every host release.

Network and outdoors

• Rating/charging matrices, mediation integrity and orchestration rollback are tested.

• Sensor-data simulation covers noise, gaps and drift; field apps survive a day offline.

Validation and go-live

• A traceability matrix in the repository links requirements to tests to results.

• UAT is run by operators on real terminals with line-perspective scripts.

• Migration is reconciled to the unit; cut-over is rehearsed; rollback criteria are agreed in advance.

Conclusion

Three seams cause most of the damage in manufacturing software: master data crossing from ERP to MES, tag mappings crossing from MES to the floor, and models crossing from CAD to toolpath. Each is a calculation or a translation, and each is best tested against a known-good answer – a golden plan, a golden batch record, a golden model – kept under version control and run on every build. Add a device lab that tracks firmware like an operating system, and traceability that turns test runs into validation evidence, and the two release windows a year stop being a gamble.

If you are planning an MES rollout, an ERP upgrade or a QMS replacement and want to see how this works on your systems, a manufacturing app testing company should be willing to prove it on one seam first. Talk to us about a free proof of concept: an ERP-to-MES work-order flow, or a CAD golden-model regression set on your own models.

Book a strategic QA consultation

Schedule a call

FAQ

What is manufacturing software testing?

The testing and, in regulated sectors, validation of the software that plans, executes and controls production — ERP and MRP, MES, QMS, SCADA/HMI and PLC integrations, and the CAD/CAM and PLM systems upstream — with particular emphasis on master-data integrity, the interfaces between layers, hardware in the loop and evidence for audits.

Who offers end-to-end test case management for industrial products?

Look for a partner who can cover every layer of the pyramid (planning, execution, control, design), test on real devices, produce validation-grade traceability from requirements to results, and show manufacturing or engineering-software case studies rather than generic web work. DeviQA provides this as a managed service — strategy, automation, device lab, validation evidence — with a free proof of concept on one integration flow before any commitment.

How do you test an MES before go-live?

Reconcile master data across ERP and MES first; script the full work-order lifecycle including every exception; run forward and backward traceability round trips; test the operator UI on the actual terminals; contract-test every tag mapping to the floor with a PLC simulator; then rehearse the cut-over on a shutdown weekend with real operators and agreed rollback criteria.

What is the difference between testing and validation in manufacturing?

Testing finds defects and shows the software works. Validation proves, with documented and attributable evidence traced to user requirements and a risk assessment, that the system is fit for its intended use — what regulators in pharma, medical devices, food and automotive require. Well-structured automated tests can serve as validation evidence; ad-hoc testing cannot.

How do you test CAD software for regressions?

Maintain a library of golden reference models; on every build open, regenerate and compare them on geometry hashes, mass properties and drawing output; run import/export round trips across the format matrix; for CAM, diff toolpaths and post-processor output against golden G-code; and regress plug-ins against every host application release.

What is the best software testing service for telecom platforms?

The best fit tests the calculations and the orchestration, not just the portals: rating and charging against tariff specifications, mediation integrity under load, order-to-activate flows with rollback, catalogue-driven regression and TM Forum API contracts — and can show telecom operations software in its case history. DeviQA's work for NEXSYS-ONE, with 1,300 automated scenarios running in 1.5 hours, is the kind of evidence to look for.

About the author

Mykhailo Ralduhin

Senior QA engineer

Mykhailo Ralduhin is a Senior QA Engineer at DeviQA, specializing in building stable, well-structured testing processes for complex software products.