Rethink your e-commerce infrastructure with microservices

District5

• 10 min read
Rethink your e-commerce infrastructure with microservices
Photo by Pickawood / Unsplash

Microservices! a super hot buzz word at the moment which seems to be thrown around by every developer.  If implemented properly, they can provide a huge benefit in both reliability and reduced maintenance.

The microservice structure revolves around keeping the code base as small and optimised as possible.  Keeping things small makes it easier and faster to identify an issue, whilst releasing updates regularly and being strict about preventing tech-debt (we'll cover tech-debt in a different post).

For this article, we'll take a look at a general e-commerce example.

When you think of a typical e-commerce business, there are a few key components:

  1. User front end (what a user sees)
  2. Order handling system
  3. Supplier management
  4. Complaints handling
  5. Returns processing

We're not going to talk about all of these, but let's look at number 2, 'Order handling system' on the list and just see how we can improve the process.  What we're going to discuss can be implemented in every system, but it'd be far too much to read and write about.

Internal order handling system information...

The internal order handing systems come after a user has placed an order.  So they've been to your website, added things to their cart, gone through the checkout process and payment has been made.

Like any system, this in turn can be broken down to some key components:

  1. Order retrieval - fetches (or receives) the orders from the database that supports the website and stores it in the internal database
  2. Internal order management interface (change it, cancel it, etc)
  3. Maybe a confirmation email, saying that the order was received
  4. Integrations with shipping providers to create shipment labels and postage
  5. An interface for employees in the distribution department to view and update the status of an order
  6. A confirmation email that the order has shipped
  7. Possibly another email with a tracking number
  8. Some businesses send yet another email after delivery, just to check how the user is enjoying the product or service.

It's quite easy to look at the above list and think to yourself, 'well, it can just be a single system', and you'd be absolutely right, it can be; but should it?

Your priority list...

When you have a system which has been written by your development partner, or internal team of developers, your priorities should be in this order:

  1. Does it function reliably
  2. How prone is this to errors?
  3. What's the business impact if this system crashes?
  4. Have we minimised complexity enough to make it easy to change or identify issues?
  5. Are developers complaining about it being 'legacy' or unreliable?

The system may function and you've never had an issue with it to date. But one day, it will inevitably crash.

There's something almost supernatural about the way systems crash.  They prefer not to crash during appropriate times.  It's generally in the middle of the night, just after everyone's gone to sleep, or worse, at 1am on Black Friday (yep, I've been there too).

De-risking the code...

It's common knowledge that the less complicated something is, the more reliable. This is where a reliability plan and de-risking comes into play.

One of our employees has a microwave from the 1980's. Every time his new fancy microwave breaks, he'll grab the old one from the garage and plug it in. It's been so reliable that is was handed down from his grandparents to his parents and now in turn to him. Microwaves weren't complicated back then. They had a specific task. Heat the food for X amount of time. That's it. No fancy buttons for baked potatoes or popcorn. It was either on or off.

When you remove complexities, you inevitably improve reliability.

In our above list of the core features, there are a few that jump out as 'quick wins'. My personal preferred starting place is emails.

Pretend you had a microservice, which could receive an API notification containing an email type and some data for the email (for example, email type could be orderConfirmed and the data could be the name, address and ordered items). Depending on the email type received, the service could easily handle sending all of the emails you needed.  More importantly, the email code could be entirely removed from the larger (legacy) system.  That's already 50% of the components removed for this particular microservice.

You could take this even further, and depending on your business needs, it might be a good idea have a microservice per email type.  In practice, you could shrink the size of your email sending code base, to fit inside a tiny little cloud function that costs you an insignificant amount to run.

The next thing to look at would be the data.  We're not going to talk about sharding the database, let's just call it 'the database'.  If you moved this database to its own instance, within a cloud hosted environment, you wouldn't have to worry about it. The common push back here is that you always need it available and questions inevitably come up around reliability and uptime if your office connection drops out.  Actually though, we're yet to find a business who can actually still operate when their internet connection goes down, and that, unfortunately is every businesses weakest link.  The concept of the cloud is now so ingrained in our society that if a cloud was to experience complete downtime, there would be bigger problems.  Card processing providers would be offline, the global availability of a search engine could be affected. So, if you think about it practically, and you're honest with yourself, there's actually nothing preventing you from making this database change and the only impact felt would in fact be positive by way of increased reliability and less worry for your tech teams.  Obviously you don't 'have to'.  Either way, we've worked with customers who've both migrated it off to the cloud or who've kept it in their server cabinet downstairs.

Made it this far? Amazing. Bear with us. This isn't something you can easily cover quickly.

So at this point we've moved 50% of the internal order handling system and moved the database to the cloud.  As a rough guess, probably 30% of the complexity has gone. The pressure is starting to ease. Here's what we're left with:

  1. Order retrieval - fetches (or receives) the orders from the database that supports the website and stores it in the internal database
  2. Internal order management interface (change it, cancel it, etc)
  3. Integrations with shipping providers to create shipment labels and postage
  4. An interface for employees in the warehouse to view and update the status of an order

Now that our database has been moved, it's already looking more manageable. The next logical step is the order retrieval.  This step retrieves (or receives) the data about an order and in turn writes it to your database.

We've seen businesses go about this in many different ways, but we feel that the most reliable is to fetch and import the order from the website.  Pushing the order to a system can in itself be a point of failure, and remember, this is what we're trying to prevent.  We can easily implement this in a tiny containerised daemon. That daemon can sit on a machine somewhere simply calling the website and asking "Hey, have you had any orders since order number 123456789?".  If the site has received some, it serves them back and the import processes the data, inserting it into the internal (well, cloud) database.

We're getting there! We may as well tackle the remaining 3 items in order!

With the 'Internal order management interface', it's usually just a user interface.  We've seen web based interfaces, and we've seen some Windows and Mac apps that have been painstakingly put together that get installed on a users machine.  Regardless of the route you choose here, this step has been removed from the behemoth system and is now a single project. Since moving the database, there's really no need for a back end to it. It's 99% user interface. A perfect project for a new starter, or even a graduate developer who would want to sink their teeth into it.  The system uses API's to interact with the database and manipulate data with no heavy lifting involved, and it really is the perfect project to give to someone; they'll have pride and take ownership of the entirety of it.

There's not a lot to write about with the management interface, it may seem complicated to the untrained eye, but it's not.

The next one seems daunting, but it's not. 'Integrations with shipping providers' is exactly how it sounds.  When you've taped up the box ready to ship to the customer, you need a label for the package and probably need to tell Royal Mail, DPD, or FedEx that you are ready to send a package (otherwise they might not show up to collect it).

Thankfully, all of these providers have amazing API's that you can speak to in order to produce a label.  It might sound like we're repeating ourselves here, but we're going to move this logic into it's own microservice. You can have a microservice per shipping provider, but for most businesses that's overkill. Instead of having all of the shipping providers in the original system, you can move this into a headless system and use an exposed API to communicate with them. In pseudo code, it could look like this:

connection = http.client.HTTPConnection('the-shipping-api.local')
connection.request('POST', '/make-label', {"address": addr, "provider": comp, "method": "24hour"})
label_data = connection.getresponse().read()

printed = print_label(label_aata)

Wow. We've just removed thousands of lines of code and replaced them with that!

The new shipping provider API is admittedly more thorough than the internal ordering system, and it doesn't have a front end that looks pretty, but an experienced developer should be able to get this put together in a relatively short amount of time.  The shipping companies (from our experience) are more than willing to be on hand to answer technical queries about integrations.

We now only have a single piece of the original system left. 'An interface for employees in the warehouse to view and update the status of an order'.

This is very similar to the internal order management interface in skillset, but instead of having the ability to edit an order, the requirement of this is to provide employees in your warehouse with a way of viewing the order, marking that they've placed an item from the order into a box, and clicking 'Generate label' to print the shipping label off of the printer.

And that's it?

Well, yes. That's all it takes. The only caveat we'd add here is that different businesses have different requirements.  We've worked with a number of e-commerce businesses who all have vastly different systems but there is always a way of making them more streamlined.

What we've outlined basically takes your legacy system, splits it into small and easily manageable systems and gives you an entirely new level of reliability.

If you've read the entire article, you may feel overwhelmed, or think to yourself "great, but my developers would never get onboard with doing it". You're probably right. One of the most common issues with internal systems is they turn legacy quickly. The idea of "if it ain't broke, don't fix it" rings loud and clear with developers, but it's both grammatically and practically incorrect.  Mainly because your developers have (throughout their employment with you) only worked on your current systems, they may not feel they have the skills or experience required to undertake such a huge transformation.  Such a proposal is somewhat overwhelming for inexperienced development teams.

Imagine if the first ever car (the 'Motorwagen', by Carl Benz in 1886) hadn't been innovated and modernised as the years went on. We'd live in a very different world.

Other unexpected benefits...

Developers are very good at what they do.  So good in fact that when one of your developers makes a suggestion for a change, you should adopt it. Why? Developers aren't known for taking the initiative to make huge improvement suggestions. Not because they're lazy, but because when they're working on a big legacy system full of bugs from 5 years ago, they have no interest in potentially creating more problems that need fixing.

Once you've moved to the microservice architecture, the momentum shouldn't stop there; it's important to keep things up to date.

A new version of the programming language is released? Upgrade it! A security fix was issued for a library you use? Upgrade it!

By keeping things up to date and fresh, your developers will constantly be learning the latest and greatest.  From a quick poll within our team here, our developers left their last job because of antiquated systems.  It makes no sense to hire a developer straight from university and have them working on a language that EOL'd (end of life'd) when they were 12 years old.  Most upgrades are painless. In fact, if you experience pain during an upgrade, it's indicative of a hole in your test plan.

With a stable upgrade route and a solid test plan, no upgrade is too much.  You invest such a huge amount of money training your developers on your systems to lose them at the 18 month 'itch'.  Worse still is that developers won't raise issues with you if they are unhappy; they just find a new role and move on. It makes more sense if you're the company they want to interview with. Asked for the reason why they are looking to leave their job, the answer is typically: "To work on a more up to date technology stack and further my knowledge. I just feel held back where I am".

District5 can help...

Our involvement with your business is dictated entirely by you.  District5 can either take complete ownership, or act as an extended part of development team.

We've undertaken huge legacy system migrations without too much interaction with your developers and after completion, we work with your team to train and teach them exactly how the new systems work, how to make changes, how to minimise their workload, and overall improve their job satisfaction. Regardless of our involvement, you'll be more than satisfied with the results.

It's commonplace knowledge that developers won't start changing huge parts of codebases and if you're in the position of having an internal team, you'll be more than aware of the mumblings between the team about how they feel with the state of the code; it doesn't have to be like that. Once your system has started to be called 'legacy' it's because the developers know there are better solutions. Speak to them, get their ideas, listen to them.  Unfortunately, getting out of them exactly what they'd change may feel like pulling teeth, but they are the best placed people to tell you what's wrong with the systems (as long as they also accept they could have a lot of work lined up after the conversation).

We'd love to discuss transforming your business. Undertaking huge transformation projects across businesses of all sizes, District5 are converse with the common issues businesses encounter.

Get in touch with us by emailing solutions@district5.co.uk and we'll direct you to an industry specialist within our solution team to discuss your business.

Your e-commerce doesn't sleep. So why should your systems?