#AEM and PIM systems integration

Contents

Intro

When it comes to showcasing products on the website it’s always a challenging thing to achieve.

Either if we are talking about a large multi-market multi-division global company holding dozens and dozens to even hundreds of websites, or if we are talking about a small company managing a single website, or anything in between (ranging from e-commerce websites providing things like: checkout & payment capabilities, personalized shopping experience, user profile tracking, loyalty programs, wishlist, ratings and many more; to simple product presentation websites with no complex user interaction features) in all these situations we need to design a product system architecture that provides product data management for the client and supports product content syndication for all required channels and viewports.

On top of this, having the website, which contains our product pages, hosted on Adobe Experience Manager (AEM) makes no exception when it comes to extra challenges. Au contraire, we might face additional particularities that we need to address and pay attention to.

The situation

For the sake of the article, we will be targeting a complicated situation when the company runs on a large digital ecosystem, and that is because on this way we can analyze more cases.

But what does that mean, complicated situation ? First of all let me just say that digital ecosystems basically means three things: People, Business and Systems.

And with that in mind, complexity increases when several things happen:

1. One of them is when company has some legacy systems, which usually are maintained by same old teams that only they know what the hack is going on in there. And ya, zero to none chances when we even remotely think to dare asking for documentation.
(The People)

2. Other is when company grew based on an aggressive acquisition strategy, without having the time to naturally digest and incorporate all the processes, and still having each local business team with enough power to make decisions on their own or make demands to high level leadership. And this may in the end have a direct effect on how products are being managed and displayed on the website, cause’ after all, this is what we need to narrow down our speech to.
(The Businesses)

3. Another is when product data is stored in different Product Information Systems (shortly PIMs). I would mention here some of the most know PIM software out there: Akeneo, Pimcore, Acquia, but many more are available on the market. I found this nice list on Gartner. Therefore data most probably is not respecting a unique structure across all company’s PIMs, nor is easy or feasible to make it be, either from financial reasons or because other platforms that company owns may already depend on them.
(The Systems)

And these are only few to mention, those most common I might say, at least from what I have encountered in my experience so far. But I am sure there are many more situations, lots of them specific to the given context in which a company finds itself.

The business need

That being said, assuming is not much we can efficiently change in terms of People and Business, let’s focus on what we can do on the Systems ares.

As you probably already guess by now, the business need is simple. I am referring to ease of verbally articulating this need or drafting it on the paper, not actually achieve it. If only it would have been that easy. So, in simple words the business need is to leverage existing products data that the company owns and display them on their AEM hosted websites. The challenge is to common up with a technical solution for doing that, and every actual real technical work starts with the designing the architecture.

Architecture Overview

I assume the AEM website(s) are already in place, properly developed and running (hopefully). I won’t get into topics about how to develop an AEM site and things like that, as it is not the purpose of this article.

Now the question is, how do we take that product data and drop it on the website for the end users to check it out ? Well, we can start with a simple high-level Architecture Overview diagram. When designing such diagram, we can ask ourselves several questions.

1. What do we want to achieve ? Broadly speaking, we want to access the products data, do whatever we need to do with it, then display it wherever we need for whoever we want.

2. Where is the product data stored ? Data is spread across multiple PIMs. This also means most probably different businesses/departments/teams may be involved in their ownership. So trying to replace/sunset the PIMs or bring any changes, is a low chance game. We will thus take this out of discussion.

3. Is the data having same structure in all PIMs ? We assume no. Remember we took on a complex situation, not a peachy one. So we will need a way to aggregate the data from different PIMs and define a common schema with unified structure.

4. Do we want to have control over the data ? Yes we want. Assuming we cannot control direct data within the PIMs, we will then think of a way to import/pull the data and store it somewhere closer to our systems, within easy reach, where we can validate it, enrich it, amend it, curate it, transform it, enable/disable it and so on.

5. Do we need smart processes ? Yes we do. We want to be able to control the process of importing the data. Meaning we want to trigger import process whenever we want to, or even automate the task. We also want to report and log all the imports, to monitor every step and troubleshoot any issue.

6. Do we need to address performance ? This is a no-brainer. We always need to think of the performance. We will most probably need to put some caching strategies into the solution. We might need to consider data push vs data import strategy, so that only when data gets. We can try to import deltas only, meaning only data that gets changed from last import (if we can have this info about the data).

More and more question may rise during discovery or solutioning (weird word, I know) phases, depending on the specific of each situation, but I believe based on these 6 questions we have enough to draft our Architecture Overview diagram.

Few notes to make here though. Designing the Architecture Overview we took into account possibilities that data may not be consumed only by AEM websites. As we can see we’ve put in there also mobile apps, in-store kiosks or even brochure printings for example. Why ? Simply because our solution may become a product data hub that powers company’s omnichannel strategy. So we decided to pave our path at early stage. Also, you can put in other services, or channels or types of end-users. As I was keep saying, every company and project has its own uniqueness. The main goal here though is to take us through the thinking process of how to unveil things and end up with a final architecture of our own.

Ok, now that we have an overall picture of our solution, we can start going deep and design a more in-detail architecture. The area that interest us the most is the Services layer. So we put this layer under microscope and try to image what could happen in there, how everything should interact with everything, what are the inputs and the outputs, what are the limitations vs benefits, and so forth. And below I will exemplify few such architectures, with a focus on the Services layer, as I said, cause’ there is where our implementation will happen.

Solution Architecture no. 1

Usually we would want not to use AEM as a database, specially for large sets of data. Afterall, AEM is a CMS, and it has not been designed for that, even if it does a pretty decent job at it, specially on the cloud version. But sometimes the conjuncture forces us to take this solution path. Maybe is the client who puts some pressure and limits our technical team choices (or worst, decides in their place). Maybe is the short time window we have for development and delivery. Maybe we don’t have other systems in place to support other architectures. Or simply maybe because we don’t have extra skills in the team.

But considering this is the context we find ourselves in, within this first proposed solution, we decided that most of our services will reside in AEM and that the entire product data will be stored in JCR as well.

1. Import Service

This service is responsible to initiate the import process. It will trigger an import round. This service will create an Import Context object which will be passed along the execution flow in order to instruct the actors involved of the import behavior they need to respect. Later on we will see what the Import Context Object can hold.

2. Import Round job

This is a parent job that encapsulates the concept of an import round. An import round is the whole import process from start to end, no matter how many PIMs are being queries, how many catalogs or products are being imported. It is important to have separate knowledge about import rounds when you need to differentiate between imports. This job in turn will trigger in cascade imports relative to PIMs, to catalogs or to products, based on the import context configs. Also it can abort or not and entire import round based on the same configs. In the same time it can stop any jobs execution if needed.

3. PIM / Catalog / Product import job

This a job triggered specifically for a unit of work (either PIM or Catalog or Product). It will create a REST Command object and pass it on to the Pull Service to do the actual network call to target PIM systems. It will notify back of the import status, either fail or success, to both parent Import Round Service and Monitor Service. Also, in case the import was success, it sends the data to the Content Manager Service to be processed.

4. Pull Service

It is the last service in the execution flow which makes the actual network call to external PIMs and gets the data. It then sends the data and/or response status to the caller import job.

5. Content Manager

It is responsible to handle and process the imported data.
It calls Generator Service in case from configurations was instructed to do so.
It notifies the Monitor Service of content storing status.
It uses Enrichment Service for amending the original raw data.
It uses Replication Service to send data to Publish tier/instance(s), if needed.
It calls Cache Service to invalidate and refresh in-memory stored data.
It stores the data intro JCR, in best way possible to serve company’s organizational model (brands, markets, languages etc)
It queries and retrieved data from JCR for the consumers to use it. Such consumers can be: Sling Models, Servlets etc

6. Enrichment Service

It enhances the raw data with extra information. Examples of such new data would be: internal IDs, light catalogs structures, timestamps etc.
It can create lightweight taxonomies objects of the catalogs.

7. Generator Service

This service is responsible to generate AEM specific content based on the data imported. It should be able to create Content Fragments, or actual Product Detail Pages (PDP) / Category Detail Pages (CDP), or PDF files for printing .
It also makes use of Replication Service to activate content and uses Cache Service to invalidate Dispatcher & CDN Caches.
It stores in JCR the generated content.

8. Replication service

All the programmatic replication events will be triggered from this service.

9. Cache Service

This service is responsible for managing caches.

10. Monitor Service

This is a service to which most services make use of.
It acts as an observer for the jobs, so that it can be notified back ok any event (fail or success)

11. Reporting Service

It generates import reports and logs, which can be either stored in JCR if needed, or sent out to notification channels via the Notification Service.

12. Notification Service

This is he service that actually integrates with any communication channel where the notifications need to be posted. It for example contain a Slack webhook and send messages to it.

13. Push Service

As a separate service, we can have a push service, which can accept data sent by PIM systems and store that data into JCR following the same flow via Content Manager Service.

14. Import Context object

Simple object, usually a POJO, holding configurations, flags, start date, user or process who triggered the import etc, for current import.
It is passed along the execution flow among services to instruct them how to behave.
It collects data until the end of the import, like finish date, exec time, statuses etc.

For example, this context object can contain configurations and flags like:

15. REST Command object

A generic object that is able to form a final PIM url to be called to import data. It uses endpoints coming from configurations, to which it may append different paths based on the exposed PIM APIs.

16. AEM JCR

17. External PIM systems

Solution Architecture no. 2

In this second proposed solution, we decided that some the services will reside outside AEM, and we will need to implement them with different technology. Most import service to build outside AEM is the Import service. Also this implies that products data will also be stored outside AEM but later be consumed by AEM.