I just completed an interesting assignment on data-driven decision-making in sales, for a retail client (one of the leading brands in high-end commodities with sales in over 8 countries, henceforth referred to as the ABC group). In this blog series, I will share my on-the-ground lessons from this assignment.

What is data-driven decision-making (DDDM)?

DDDM is a disciplined approach of making strategic business decisions using insights derived from business data rather than gut instincts or guesswork. It involves exploring patterns and trends in data to shape business strategies that can lead to long-term success.

In today’s digital age, businesses generate and access massive volumes of data, and if you’re not leveraging this precious asset to make better business decisions, your competitors are. DDDM has become critical for businesses seeking to remain competitive in today’s data-rich business world. Your business must make DDDM the norm, cultivating a culture that encourages critical thinking at all levels.

DDDM assists the ABC group develop sales strategies, forecast market trends, optimise operations, uncover hidden patterns in consumer behaviour, and much more. It has changed the entire perspective of their sales team. It has empowered business stakeholders at all levels to ask and answer questions, resulting in more efficient and effective decision-making.

Benefits of data-driven decision-making

Here are some significant advantages of DDDM learned while establishing the DDDM process for the ABC group.

Eliminates emotions and biases

The person with the loudest voice in the room should not be the one driving decisions. DDDM helps eliminate emotions and biases from decision-making. It ensures that decisions are made based on facts rather than gut instinct or guesswork, resulting in more accurate and reliable outcomes.

One evident shift I’ve witnessed at the ABC group is the gradual building of a “data culture”, which encourages individuals at all levels to make their own business decisions and back them up with facts.

Quick and agile decisions

Quick and agile decisions are essential in today’s fast-paced business world. DDDM could assist stakeholders at all levels swiftly explore patterns and trends in real time using self-serving tools that would otherwise go undetected. This allows stakeholders to quickly respond to changing market conditions or customer preferences over time.

For example, knowing which products are fast or slow moving in which stores or regions, helped the ABC group in analysing trends and customer preferences, to have appropriate actions (introduce new designs, discontinue old designs, customise designs, change market tactics, and so forth) to improve overall business performance.

Optimise resources and operations

DDDM enables businesses to find areas of improvements in order to make better informed decisions about demand forecasting, resource prioritisation, resource allocation, production-line optimisation, customer service, and more. This helps businesses to optimise operations, and cut costs while preventing several future risks.

For example, after establishing the DDDM process, among other things, the ABC group could plan efficient stock management across stores, and improve customer services.

Improved customer experience

One of the most significant benefits of DDDM is the relationship formed with customers. DDDM is an excellent approach to demonstrate that you’re paying attention to them and respect their time and interest.

For example, analysing customer behaviour, and feedback, both good and bad, enabled the ABC group to improve the customer experience. Customers benefited from more personalised recommendations and tailored experiences — which could lead to increased loyalty.

Businesses may utilise these relationships to create relevant, personalised marketing campaigns (Netflix, Amazon and Coca-Cola are well-known examples).

Competitive edge

Businesses that embrace the power of data gain a significant competitive advantage. DDDM empowers your business to create pro-active opportunities for growth in order to remain competitive in a data-driven marketplace.

The ABC group analysed historical sales data, purchasing patterns, and customer feedback to be proactive in designing new products, improving existing ones and so on. This resulted in attracting more customers and so revenue.

These benefits highlight how DDDM can reshape businesses. The world is getting more data-driven, and refusing to make data-driven decision-making is dumb.

DDDM Process

Building an organisation that can benefit from DDDM necessitates a combination of people, process and technology. Let us focus on the process.

In a nutshell, the DDDM process consists of 5 steps, though each organisation can have their own customisation aligned to its business objectives.

1. Define business objectives and KPIs

First and foremost, you must understand your business objectives. What are you aiming to achieve? This might be as specific as increasing sales revenue, lowering operating expenses, or improving customer satisfaction. Organisations must take the effort to establish specific objectives with a clear purpose.

Once you have a clear understanding of your business objectives, you can identify KPIs that are relevant to those objectives. KPIs are metrics that businesses use to track progress towards the objectives.

The ABC group identified around 20+ metrics like sales growth, sales per product, sales per region, sales per store, online vs stores sales, sales per CSR, customer ratings, market penetration, and much more to track progress towards the objective of increasing sales. It is essential to focus on actionable metrics that stakeholders can influence, for ex. customer ratings which can be influenced by the ABC sales and services team.

2. Collect relevant data

The first step in collecting data is deciding what data to collect based on identified metrics. This stage involves examining accessible data sources and evaluating which are more relevant for the metrics at hand. One must think creatively about what facts may bring better results.

Data can be collected from both internal as well as external sources. The idea is to collect enough data to uncover patterns and trends that will aid in decision making. The sources can include internal databases, customer feedback, surveys, interviews, market publications and so on, over time. Businesses can use various technologies to collect data, including point of sale systems, CRM software, external APIs, and so on.

This should ideally be done in collaboration between business users and data teams. This guarantees that business objectives are prioritised from the start, and that data teams significantly improve the outcomes that business users want to target.

In the case of the ABC group, we used their worldwide sales and CRM data to gain insights. These insights could help in identification of crucial areas of improvements, such as customising region-specific offerings, pricing strategies, product designs, product mix, stock management, future investments etc.

3. Cleaning and organising data

Data from multiple sources is often unorganised, and may contain errors, missing values, or duplicates. Data cleaning and preparation involves transforming and structuring the raw data into a format suitable for analysis. This step guarantees that the data is appropriate and reliable for analysis.

This step includes data cleaning tasks such as verifying data consistency and accuracy by eliminating outliers or irrelevant data, filling missing values, standardising data types, and denormalising data for analysis, among others. Organising your data for better data visualisation is critical for making concrete business decisions. If you can’t view all your relevant data in one place and understand how it connects, it’s tough to make the best decisions.

In the instance of the ABC group, data visualisation tools were quite helpful in detecting good cases of inconsistent data formats, missing values, outliers, and other issues.

In my experience at the ABC group, collecting, cleaning, organising and validating data takes up around 60% of the whole process time. Steps 2 and 3 are critical because data completeness and correctness have a greater impact on the analysis.

4. Analysing data

You’re now ready to start analysing data to uncover patterns, trends, and correlations that can be used to make decisions. Look for connections between variables, cause-and-effect linkages, and statistical models (such as decision trees, linear regression) that might help in predicting outcomes.

You will also need to decide how to interpret and present the information in order to answer questions at hand. This is accomplished through data mining and data visualisation. Data mining is about identifying relationships, hidden patterns, and trends, while data visualisation is essential for communicating insights effectively. Using data visualisation tools, statistical analysis software, or machine learning algorithms, enables the data to speak for itself.

Organisations should empower business users, not just data specialists, to interact with the information. Doing so early in the process, allows business users to find potential data gaps, new opportunities, and other elements that will be important in driving decisions. This necessitates the usage of tools suitable for non-technical users, like visualisations such as charts and graphs, which make complex data easier to interpret.

In another blog, I will go over the various approaches and tools we used at the ABC group for data analysis. It’s another lengthy and engaging topic. During this step we learned about sales growth, top selling products, which products are being purchased together, customer behaviours, and much more across regions to build strategies to drive sales.

5. Evaluate insights and make decisions

The data analysis step leads to the actionable strategies that can be put into action. This is the time to make analysis-driven decisions, and develop decision-driven strategies to achieve the business objectives.

Try to create a connected story through analysis. Keep digging into the identified metrics until you find a “why” for the problem you’re attempting to solve. Leverage your initial business stakeholders to help sell the insights and decisions to a larger audience. The report should provide action-driven recommendations based on findings.

Some of the decisions taken at the ABC group with the 2 main objectives of maximising sales and enhancing customer experience are

• The insights around products that sell well during a certain time period, are utilised to optimise region-specific product offerings like discounts to maximise sales. These insights also helped to plan the right stocks levels across regions while minimising over stocking.
• The products being purchased together as per customer data are given special offerings, to boost the chances of additional purchases.
• Send personalised marketing emails featuring new products based on customer behaviour and preferences which lead to increased sales.
• Do customer segmentation based on their behaviour in order to target them with suitable offers and promotions via targeted marketing efforts.

It is vital to continue evaluating insights and decisions so that they remain relevant and effective in the ever-changing data landscape. This can also help reveal patterns and trends that may not be immediately apparent (ex. KPIs that need data over a longer period of time).

Businesses can track KPIs to understand how their efforts have met certain business objectives as well as how this has influenced the overall organisation. It is important to evaluate the effectiveness of efforts to improve the data-driven decision-making process. Businesses can ensure long-term success by using this process to utilize data to its fullest potential and make well-informed decisions.

Once the process has been established, the focus should be on automating data collecting, data cleaning and data analysis steps. It is also required to build a data culture inside an organisation in order to increase data literacy, collaboration and sharing, as well as enabling data analysis for everyone within the business.

In the following post, I will discuss the challenges and best practices in implementing DDDM.

Architects are made, not born, and so am I (one among you).

There is a point in the career of a software developer where one has the choice of pursuing a career either on the management path or on the technical path as an architect. I chose to walk the technical path as an architect, as I am passionate about technology.

The transition from one who codes the design to one who leads the design requires a shift in mindset and far more than just technical skills, as I have learned. In this blog, I’ll share what I’ve learned with developers who want to be architects. I’ll break it down into two parts:
I. The first step: Understanding and preparing for this role
II. You are in a different world: On-the-ground learnings on getting started in this role

Part I: The First Step
The first step is to understand this role and prepare yourself.

There is no one recipe.
There is no one way to progress as an architect, and each organisation may have its own titles and career paths to follow.

I’ve seen different titles for the same set of responsibilities, as well as different responsibilities for the same title, across organisations. Note that these titles are used within the context of the organisation and could mean different things for different organisations. At times, you may hold two titles, one at the organisation level and another at the project level (ex. one of the three SAFe architect roles if the project is practising SAFe).

On top of this, there are different types of architects (what type of architect you want to be) without any standardisation.

So, when you choose this path, it’s critical to understand what type of architect you want to be, the career path within your organisation, what responsibilities come with a specific title, and whether it aligns with your aspirations.

Understand yourself.
“The only way to do great work is to love what you do” said Steve Jobs.

Take the time to think about your personality, your interests, and the core needs of this path to be successful as an architect. If your answer is “no” to any of the following points, then it is most likely not a good career path for you.

1. You have a passion for technology and building solutions for businesses in challenging environments.
2. You have the willingness to be technical but to go broader than deeper on technologies.
3. You have an interest in developing strong people skills.
4. You have the desire to coach and lead teams.

I’ve met great tech geeks who helped build complex technical solutions or solve complex technical problems, but they were perceived as individual contributors and struggled to advance as they lacked the 2–4 skills listed above.

Are you prepared?
While there are no hard and fast rules about the number of years of experience needed to become an architect, I believe that at least 8–10 years of experience driving complex designs and implementations on a diverse range of projects and working in high-pressure environments will help in setting the foundation.

Keep in mind that 1 year of experience repeated 8–10 times or just completing some training & certification won’t cut it here. The more practical experience you have, the more valuable you will become. In my experience, depth of knowledge in a specific business domain adds a great deal of credibility.

Of course, it is a journey that takes time and experience to master. You should have the desire to learn new things every day.

V-shape skills will make you stand out
You’re either an I-shaped developer (having deep skills on specific technology) or you’re growing as a T-shaped developer (gaining broad level of skills and knowledge on multiple technologies combined with deep skills on specific technology).

In today’s fast-paced digital business world, the architect is at the centre of contribution in an enterprise, so deep skills alone in technology discipline won’t be enough. The future architect must be V-shaped, versatilist (Gartner coined this term. Call it a Swiss army knife), and play a role in other adjacent disciplines that extends beyond technology discipline.

The bottom of the V represents the deep expertise in technology discipline, while the two sides of the V represents their broader but well-rounded expertise in other adjacent disciplines (ex. strategic thinking, planning, team building, people skills etc.). V-shaped architect will grow in spiral fashion both horizontally and vertically (unlike T-shaped).

I hope this helps you get started. In the second part, which should be more interesting, I’ll share my first-hand experiences as an architect.

I would appreciate your comments, which will allow me to tailor the content in Part 2 to any specific needs or questions you may have.

I talked about Trends in Integration Technologies, let us now explore topics around each of these trends.

The API-led integration approach has become a synonym for modern integration, to power the digital transformation. You can find tons of papers on the internet on the API-led integration approach, and its benefits through reusable & built-for-purpose APIs (https://www.mulesoft.com/lp/whitepaper/api/api-led-connectivity).

But is this the best approach we shall consider in all integration scenarios? Let us see integration scenarios where the API-led approach may not fit and result in a lower ROI for your business.

· Unpredictable future integration requirements

One of the primary advantages of the API-led approach is the high rate of re-use of integration APIs. But if a business is not predictable enough to forecast the business processes and, therefore, the right design of APIs, then this approach can result in similar challenges as SOA, in terms of lower re-use and higher re-work. This will result in a lower ROI.

· Limited scope of integration

The traditional point-to-point integration approach would be a perfectly acceptable solution for a small organisation with a small number of applications to be connected and a focus on short-term benefits.

There is no point in creating multiple layers of APIs for no business reason other than to follow the guidelines, which results in a lot of code to maintain, network traffic, and run-time cost & performance. If there is no business value creation, then there is no point in going for the API-led design for unintended business needs that might not pop up in the future.

· Limited time and budget

An overemphasis on creating multiple layers of APIs and laying a solid foundation necessitates a significant amount of initial effort and is costly both during the build and run times. Organizations with limited time and budget should reconsider investing in the API-led approach because it can result in a heavy foundation, which complicates and impacts the overall schedule and budget.

In such scenarios, an organization can focus on reusing built-in components and native APIs available on their integration platform to achieve results in a reasonable amount of time and money instead of the API-led approach.

· Readiness of the organisation for the API-led approach

Changing gears to think in the API-led way takes effort and time, and culture can eat both. The API-led approach needs close collaboration across business units and partners. To drive this change, an appropriate organisation and governance structure must be in place to maximise the ROI.

In addition to the structure, as we build APIs, it is also important to have the right tools and infrastructure (API developer portal, CICD pipeline, service repository, service registry etc.) in place for the API lifecycle management in a decentralised manner to support a large user base. Otherwise, this can end up being an inefficient approach with no real value to the business.

· One-time data synchronisation

We come across a lot of one-time data synchronisation scenarios during eco-system modernization or migration assignments. In such scenarios, it is perfect to have simple point-to-point integration or use native features of the end-point applications, to synchronise the data between data stores or applications. You will be efficient and effective on both time and budget.

So, if there is no future for an integration outside of the specific tasks, there is no point in investing in the API-led integration approach where benefits are more in the long term.

· The large-volume batch integrations

Large volume batch data integrations using ETL, and other methodologies are often different from traditional real-time application integrations. This type of integration need features like scheduling, run-time monitoring, performance-centric processing, an interface to schedule and manage these batches, etc. Most integration tools provide a built-in approach specific to building large-volume batch integrations, and I would recommend using these purpose-specific built-in tools than the API-led approach, to maximise gains.

Though we can call APIs within batch flows, I would recommend not building layered APIs for such performance-oriented integrations, as it can impact performance (more network calls) as well as high run-time cost (increased cores).

Summary

In reality, the IT world is infamous for its tighter deadlines and resource constraints; on top of that, the agile business models and technologies make it more complicated. Also, there can be a lack of long-term vision, budget constraints, and pressure to demonstrate immediate results. The API-led approach definitely helps to achieve speed and agility but may not always be the best fit.

So, architects should think twice before making integration design decisions to arrest the true value of the approach, and the approach should be well aligned with the business goals and roadmap to arrest the true value for the business.

I would appreciate your comments, and please also share your experience on the API-led approach.

• API-first Approach

We have seen exponential growth in API-led integrations since 2020, and this approach will continue to grow in upcoming years. The API-led approach has helped organisations to support business agility as well as maximise re-use of business functions. In addition, with more and more APIs being developed, there will be a focus on enhancing API management tools for API developers and API consumers. With this approach, we see that most middleware platforms no longer call them Enterprise Service Bus (ESB).

• Hybrid Model

Overall, organisations are shifting towards hybrid and multi-cloud models for their business applications. The universal integration approach to managing such a diverse set of environments will be needed in the future.

• Embedded Integration Layer

Many organisations are on packaged or SaaS products, and these products will continue to enhance their integration capabilities to uncover core business and connect to the outside world. Some examples are SAP, Salesforce, Workday, or Maximo.

• Domain Centric Decentralization

In the future, we will see decentralised integrations due to more and more adoption of domain-driven and micro-services-driven architectures. Instead of a centralised monolithic integration platform, each domain will have different integration approaches and even tools, based on the volume, velocity, and variety of data being integrated. And these domain-specific integration solutions will be governed by respective business units, rather than a centralised integration team who knows little about business data.

This trend is also making organisations think about server-less integrations instead of using traditional integration platforms to achieve agility and scale.

• Smart Automation

Integration platforms are now focusing more on approaches to achieve automation in building integrations and business processes through AI&ML on their platforms. This reduces reliance on integration developers as well as overall time to market. Platform vendors are working on auto-mapping, mapping suggestions, auto-flow completion, etc. through data insights using AI&ML. This will not only minimise human intervention but will also enhance the accuracy and speed of the data integrations.

• Autonomous Learning

ML supported data integration platforms can induce autonomous learning to provide insights on patterns and trends in data being integrated, which can help businesses to monitor business behaviour and so the decision making.

In addition, there are new entrants like Data-as-a-Product, Data Mesh, Data Fabric, Knowledge Graphs, etc. that accelerate real-time analytics and insights, especially when dealing with big data, and can change the overall approach to integrations.

Now, the bigger question is which approach or toolset is suitable for my enterprise. We live in a noisy and complicated world, so the answer is — it depends!. It depends on the applications being connected, the style of integration, size of data, interface contract, mapping complexity, non-functional requirements, and many other parameters.

I would appreciate your comments on the trends that I see, and please also share the trends that you see in Integration Technologies.