in Software development

Thoughts on integrating external services

This week I was working on the backend of a new application. One of the requirements was to convert addresses to coordinates. This process is also known as forward geocoding. Reverse geocoding is exactly the opposite, where you convert coordinates to addresses. Since we don’t have that geographical data ourselves, we decided to look at an external service. We choose for a  competitor of Google Maps, but that does not really matter. What I want to share with you here, is the solution that we came up with for safely integrating that external service.

About integrating external services.

In my first years as a developer I would not think long about implementing the connection with this service. “How hard can it be? Just call the web service and use the response. What could go wrong?” Nowadays I tend to take a step back and consider a few things:

What happens when the external service is not available? Will this kill the process? What will the customer see? Should we just keep retrying until their web service is back? Should we cache the responses on our end?

The solution that we came up with.

When we receive a new address in our system, it needs to be forward geocoded immediately. In an event driven architecture this is easy to accomplish. We just listen to the AddressWasAdded event and dispatch a GeoCodeAddress command on the async command bus. This command gets picked up by one of the workers and is handled by the correct handler. The handler tries to connect with the external service and saves the coordinates in the database. Because the external web service is billing us per request, the GeoCoderService caches the given coordinates for about one day. So if the same address is requested, it will hit the cache and not the actual service. The real implementation is more nuanced but globally this is how it looks:

Diagram of the solution

Diagram of the solution

Takeaways.

The main takeaway of this solution is the introduction of the Async Queue. This is solving most of our problems. If an error happens in the Geo Coder Service, the command is requeued and will be retried at a later time until it’s handled successfully.

Because we work with an event driven architecture and CQRS, processes are triggered via events and commands. But this solution can be implemented in any architecture. Just make sure that you application publishes a message on an async bus and let a consumer handle the rest.

I am very interested in your thoughts on this solution and if I am missing important questions about integrating external services. Please leave a comment to share your thoughts.

Write a Comment

Comment

  1. Hi Joost, interesting topic!
    What happens if you need the location but the Web Service is still unavailable?
    I understand that the command is requeued and executed later but I guess it would be nice if there is some feedback about what is going on. How do you handle that?

    • Hi Richard,
      Thanks for your question!

      In your code, you can identify what is going wrong and throw a specific error with a detailed error message. Make sure that these errors are written to a log file and that you have monitoring in place so you can get notifications when errors happen. For example, if you called the service with wrong authentication details, the service returns a 401 HTTP status code. You can they throw an AuthenticatedFailed error with a specific message so that you know you have to change the authentication details. In the case that the service is not responding at all, you can set a time-out and when the service does not respond in 2 seconds, you can throw a ServiceNotAvailable error. When you see what errors are happening, you will know if you have to take immediate action or just let the system try it again in a few minutes.

      I hope that makes sense.