Gartner defines system integration as the process of creating a complex information system that may include designing or building a customised architecture or application, integrating it with new or existing hardware, packaged and custom software, and communications. A system integration solution essentially creates a bridge that allows multiple systems to communicate and transfer data.
An integration project size can range from a simple “system A to system B” to “large enterprise multi-directional integrations.” But regardless of size, there are a few things to keep front of mind to ensure a smooth integration project and more robust deliverables.
Here are the top 6 things we think you should consider before you start your integration.
1. Make sure your mapping is correct
Good mapping is the foundation of successful system integration or data migration. Put simply, data mapping is defined as translating between a source and destination message format. It’s a means to an end to help the systems communicate. Failing to apply the correct mapping from the outset of a project can cause rework in your integration solution and introduce more technical debt.
Here are a couple of tips for getting your mapping right:
- When working on data mapping, it’s crucial to understand with what standards the source system expects the mapped data to comply. For example, some systems expect to receive date-time data in the format of MM/DD/YYYY, whereas others expect MM/DD/YYYY HH: mm. The same applies to phone numbers. While some systems expect a phone number in E164 format, others don’t perform any validation and will accept any format.
- Consider introducing a canonical format when integrating the same data type between more than two systems. This is a common data model accepted within your organisation. The benefit of using a canonical format is that it reduces the coupling between different applications/systems and the number of translations/mappings required.
- Ensure you have excellent mapping documentation. For example, source field on the left, data type, mapping rules, format, comments, and sample inputs/outputs. Also outlines any specific business or transformation rules on the documentation. This provides the development team with all the detail they need to implement accurately and efficiently.
2. Make sure you know what information to track for logging instrumentation
When your project team is wholly focused on delivering the solution, it’s not uncommon for logging to be overlooked. As a result, post-go-live support becomes more complex. In the modern cloud computing era, logging and instrumentation are vital when integration processes can run on multiple compute nodes, even in different regions.
- Most PaaS offerings have out-of-the-box options for looking at logs/run histories. However, using additional tracked properties in the source system (e.g. in Microsoft logic apps), including relevant information such as the data’s unique ID, makes it easier to search run histories and troubleshoot an issue. Dell Boomi supports document tracking out-of-the-box using tracked fields. Also, consider visualising the logged data for easy monitoring by building a dashboard to support the solution. Check out Microsoft’s log analytics workspace, or leverage a third-party tool such as Sumo Logic.
3. Make sure you handle exceptions/errors
Exception handling is the process of reacting to unexcepted events or errors when an integration solution runs. When errors are mishandled, it can lead to red herrings, which might mask more significant issues presented in the logs and require manual clean-ups.
Well-designed exception handling in a solution should be able to:
- Handle transient errors by retrying. For transient faults, such as a network blip in the services/databases, the solution should have a retry policy to successfully deliver the message after the error resolves itself. However, watch for nested retries, outage scenarios, and parent processes retrying processes already embedded, as this can lead to N factorial retries depending on the nesting level.
- Utilise a dead letter queue for failures. The solution must be able to correctly route error messages that cannot be processed and may require resubmission in the future, either manually or by an automated clean-up process.
- Handle known errors gracefully and notify as appropriate. In specific scenarios, the solution should inform an appropriate person or group of the error instead of only sending to a log. For example, when integrating an invoice payment, the detail of the invoice to which the payment will apply is a mandatory field. If this information is absent, the system users need to be notified so they can correct the data and resubmit as required.
4. Understanding the current environment states
When planning and designing your integration solution, it’s a good idea to get in touch with the appropriate subject matter experts (SMEs) and owners of the systems with which you are integrating. This allows you to obtain as much information as possible about the environments and to carry out connection testing in advance. While some systems will have only one environment, others will have multiple test environments, which you can leverage for dev testing to mitigate connection issues. If there is not enough environment to perform testing on, consider setting up mocking for integration testing, where it allows us to generate mock API response message without having to call the actual API endpoint.
- The choice of security credentials plays an integral part in building a secure solution. Where possible, use the latest methods, such as OAuth, in your APIs/integration solution. Also, apply the principle of least privilege to accounts that access the systems. For example, if a solution doesn’t need to write to a system, only grant “read” access to the account. On the other hand, if it only needs to access accounting information, then don’t allow access to customer information.
- Remember that security doesn’t only apply to the solution but also your account. Apply the usual security practices such as having strong passwords and enable MFA.
5. Make sure you have well-documented and articulated support scenarios
It’s never too soon to start thinking about life post-implementation. When designing the solution, remember that it should be as supportable as possible. Ask yourself if it can be easily handed over to non-developer personnel. Without great documentation, support personnel are in the dark about the purpose (while it is advisable to involve them in the development possible, this is not always an option in the projects)
- Ideally, your support documentation should include a list of known issues (unexpected behaviour in a system that’s been documented but not yet fixed), what causes them, how to resolve them, and whom to notify. When known issues recur, it’s worth weighing up whether time should be invested in finding and applying a permanent fix.
- Also, keep in mind that you need to factor in maintenance and support costs (internal or external) of keeping the solution running smoothly. Few solutions are set and forget, so make sure you minimise (or raise awareness of) the potential of future issues from the outset.
6. Capture those non-functional requirements
Don’t forget to capture the non-functional requirements for integrations. These significantly impact how you approach integration design and ignoring non-functionals has caught out many teams. Here are a few examples to capture.
- Message size, frequency, and throughput – these attributes are vital if messages are extremely large or volume excessive and often require a change in approach to leverage, say ETL or claim check patterns.
- Backload requirements – there is nothing worse than finding out as you’re about to go live that you need to find a solution to backload seven years’ worth of data, and teams are looking to your integration to do the heavy lifting. Finding out early means you can find alternative methods to do these one-off activities.
- Transaction per second – when developing APIs, especially for front-end applications, always capture the product owners’ desired performance targets. This way, you can wrap integrations with automated performance tests that can assert this you can design for speed up front. Nothing is worse than finding out in UAT that your API is too slow for the boss!
At Adaptiv, we live and breathe integration. So, if you’re in need of some help or advice, call.
