In order to ensure secure and reliable API delivery, a MuleSoft testing strategy should be employed. It is vital that MuleSoft applications and APIs are thoroughly tested before being released into Production. Insufficient testing could lead to systems not being kept in sync, delays in synchronisation or incorrect data being written to critical systems. In a worst case scenario, records could be overwritten, removed or corrupted if the integrations are not behaving as expected. This blog series walks through a testing framework and provides guidance on the tools available that can be used to automate testing MuleSoft Applications and APIs.
The approach outlined is a baseline framework that is used across several projects and can be adapted to specific project needs (different environments, specific CI/CD requirements, different teams, and different levels of scope for testing based on criticality). Some stages may also overlap and take place at the same time.
API Unit Testing
The first stage of the testing cycle is Unit Testing and is tightly coupled with the MuleSoft API being tested. The Unit Tests will be packaged with the source code and run any time a build takes place (as part of the Maven lifecycle). Each Unit Test should test an individual component such as transforms (DataWeave), choice routers, error handling or sub-flows and mock any external dependencies and systems.
Responsibility: Developers will write the Unit Tests at the same time as writing the code.
Automated: Yes
Tools Available: MUnit & JUnit (for custom Java code which should be minimal)
Environment: N/A the tests run as part of the build and mock any external connections.
Benefits:
- Automated and can be baked into the CI/CD Pipelines.
- Coverage reports can be generated and enforced (>80% coverage recommended minimum).
- Allows developers to catch errors early.
- Acts as a regression test pack giving developers confidence to make changes to existing code.
- Often used as part of a TDD approach.
The next post in the series will provide more in depth detail on MUnit and how it can be used.
Developer and Technical Testing
The next stage is Technical Testing and takes place once the developer is happy with their Unit tests and has deployed the API into the Dev environment. The developer will test the API or application by sending requests and triggering the flows validating; connections, data transformations, error handling and logic against the requirements. The developer may also enable Debug logging to see additional details and monitor that an application is working as expected. This level of testing isn’t usually end-to-end at this stage but multiple APIs may be called e.g. a Process API may call a System API during the testing.
Responsibility: Developers
Automated: No
Tools Available: Manual with Postman, Anypoint MQ, Anypoint Platform to trigger requests.
Environment: Dev
Benefits:
- Provides an initial sanity check that the design & implementation are correct.
- Ensures connectivity between MuleSoft and other systems.
- Ensures configurations and properties have been provided correctly.
Systems Integration Testing (SIT) and Functional Testing
Now the developers are satisfied that the implementation is ready and meets the requirements, the APIs and applications will be deployed into the QA or SIT environment ready for a QA team to start testing. The testing team would have a list of test cases to go through covering the requirements and edge cases. The team can test individual APIs with similar tools as in the Technical testing and can also test end-to-end by updating in a source system and ensuring the update is made to the target system ensuring that all fields and records have been mapped correctly. Automation can also be put in place at this stage.
Responsibility: Testing / QA Team
Automated: Automation is possible depending on the type of Mule application
Tools Available: Postman, SoapUI, BAT CLI, Anypoint MQ, end systems.
Environment: QA / SIT
Benefits:
- Proves the deployment process from one environment (dev) to another (qa) is successful.
- Thoroughly validates the implementation meets the business requirements.
- Validates systems are talking to each other end-to-end correctly.
- Validates edge cases are handled.
- Provides some validation that the given requirements are correct and will meet the business’ needs.
User Acceptance Testing (UAT)
User Acceptance Testing is where the business gets to use and interact with the integrations end-to-end. The MuleSoft APIs and Applications will be deployed into a UAT or Pre-prod environment and business users would interact with the end systems as they would in Production. This gives the business a chance to see how the requirements have been implemented and also identify additional requirements or changes required to requirements. It is also a chance to see business processes working with the new system and validate that the goals of the implementation are being met.
Responsibility: Business / End Users
Automated: No
Tools Available: End User Systems
Environment: UAT / Pre-Prod
Benefits:
- Provides business users the chance to interact with the system before production.
- Can be used for training at the same time as testing for business users to understand how the systems and processes will work in Production.
- Identifies any further edge cases not found in previous testing stages.
- Identifies any additional or changes to requirements needed.
Performance Testing
Performance Testing involves sending many requests to the MuleSoft APIs replicating the volumes expected in Production and ensuring response times are sufficient. The Performance Testing can help identify bottlenecks and whether the hardware sizing is sufficient. This testing can be done in parallel to UAT but we would recommend a separate environment to the UAT testing to avoid impact to users.
Responsibility: Testing / QA Team
Automated: Yes
Tools Available: JMeter, LoadRunner
Environment: Pre-Prod (the environment should be identical to Production)
Benefits:
- Ensure the integrations can handle the volumes expected in Production.
- Evaluate what loads the integrations can handle to ensure sufficient headroom for growth in Production.
- Identify bottlenecks, memory or thread leaks that can be resolved.
- Assess the number of workers and vCores that APIs need to be deployed to.
- Ensure the APIs & Applications can be scaled appropriately.
Regression Testing
Regression Testing is important when an API has been delivered and tested previously but changes have been made. If manual, a regression test would be scoped based on the area that the change was made to ensure the change hasn’t had other unintended consequences to the API. Regression testing can often be automated after an initial release of an API and new tests added when changes are made. The testing can be baked into the CI/CD pipeline or run in parallel to SIT or UAT.
Responsibility: Testing / QA Team
Automated: Yes
Tools Available: Postman, BAT CLI, MUnit
Environment: SIT / UAT / Pre-Prod (the environment should be identical to Production)
Benefits:
- Ensures that any changes haven’t had adverse effects on other requirements.
Smoke Test
A Smoke Test is done after an initial deployment to ensure that all connections are working and configurations are correct. It is often done after the first release into Production as it is the first chance to test the connections. When doing a Smoke Test in Production only non-state changing requests (e.g. read requests) should be made to ensure that live data isn’t updated incorrectly.
Responsibility: Release Team / Developers / QA Team
Automated: No
Tools Available: Postman or End Systems
Environment: Production
Benefits:
- Sanity test a deployment into Production has been done correctly.
- Validate connections are working.
- Validate configurations are correct.
Penetration Testing
Penetration testing may be required if it is the first time APIs have been deployed in the organisation. This testing would usually be ran in a Pre-Prod or Production Environment to ensure malicious actors can’t use the APIs for malicious purposes. Since Mule is often used to create, read and update data in back-end systems it is vital that the APIs cannot be mis-used and only access is provided to consumers who require the access.
Responsibility: InfoSec Team
Automated: No
Tools Available: Postman, Anypoint Platform
Environment: Pre-Prod / Production
Benefits:
- Validate APIs are properly secured and only accessible internally (if applicable).
- Validate Anypoint Platform has been configured correctly.
- Reduce attack vectors for malicious actors by ensuring vulnerabilities are identified and patched.
Conclusion
Following these stages of testing in your MuleSoft projects will help ensure that your APIs are delivered in a secure and reliable manner reducing the number of defects in the Production. Not all stages may be relevant and some may be larger in scope than others depending on their integration criticality as well as the potential impact defects could cause in Production.
The following posts will go into more detail on tools that can be used for the different stages to automate the tests including MUnit, Postman and BAT CLI.