Making smart integrations in scaling organizations

As companies grow and scale, start-ups acquisitions become very common and necessary for the success of the company to build a complete solution. It’s not a secret that the market doesn’t wait and other competitors are not sitting still…

Also, when companies grow, the set of products and solutions are expended and more engineers are being hired to support this goal. Different groups of engineers develop different sets of products or components, sometimes in different offices and also in different locations around the world.

Similarly, to a well-trained and oiled orchestration, companies need to integrate their products to work together and create a complete, holistic and winning solution for their customers.

In this article I would like to deal with the challenges of products’ integrations:

  • Why is products integration considered to be so cumbersome and why do we tend to exceed the original planned project’s timeline?
  • What are the technical gaps as well as the cultural gaps we need to overcome in order to get integrations done?
  • How can we “spice up” integrations with innovation and make sure employees have meaning (especially engineers), so that they don’t try to avoid these kinds of projects?

Let’s dive into the main areas that some small required tweaks can lead to a great change:

Cultural Differences

During my career, I had the opportunity to manage and work in cooperation with teams located all over the world — EU (Romania, Bulgaria, Ukraina), US, IL, India and more. The inevitable conclusion so far is that even globalization is growing with time and we are becoming more connected, the cultural differences are still immense in relation to a variety of areas. Examples for that can be seen with employees’ attitude towards management, intonation and velocity during meetings, approach to rewrite products/components from scratch vs “don’t touch whatever works” and even mail responding patterns. Cultural differences can be also “local” and be seen when acquiring start-ups. For example, usually, companies’ DNA is different and the start up’s employees are being perceived as creating “quick & dirty” solutions and “don’t understand about supportability and serving huge amount of customers” while, on the other hand, “the others” are being perceived as doing “over engineering” as well as being slow and cumbersome.

Can we overcome cultural gaps?

The simple answer is, usually, we can’t … BUT there are simple tricks to lighten the cultural issues:

Embrace the openness culture — In order to reduce any kind of antagonism, there should be a defined set of “ground rules” to increase transparency related to areas of responsibility, timelines, duties and expectations from each team. These rules don’t necessarily have to be written, but it’s important that they will be understood and orally consented by all. For example: What is the ownership of each team vs the complete solution? What is the definition of done when delivering any kind of content? What are the integration milestones and dates? What are the expectations from each team in relation to design coverage? testing ? support? or even meetings attendance or responding to request patterns.

More than that — I believe that the teams should be aligned to set of processes all agreed on (see relevant section below: “Facilities & Processes”) which define basic guidelines related to communication, knowledge sharing and more.

Lastly, openness culture also means that debriefs are being embraced by the organization, done on a regular basis and employees can raise concerns and ideas for improvements while managers and decision makers should be open minded to assimilate them if they make sense.

Good competition — apparently, there is something like that and it can be considered as “an art” to turn competition to a good one and for driving the teams and its employees. Competition works as a motivator because it pushes us to exceed. Not only are we driven to exceed our own capabilities, but also those of our teammates and coworkers as well. Having friendly competition related to the project encourages people to work together and spend time together, creating social bonds which strengthen teamwork. Taking it a step further, competition fosters cooperation, which is the backbone of good teamwork (For example, which team delivers with “no bugs”, fixes maximum issues in X hours , brings innovative ideas to moving project forward in X weeks or expanding it). Teams can be related to the same component/product or mixed to increase collaboration.

F2F interaction — Lastly, there is nothing that can be compared to F2F interaction to increase collaboration and overcome cultural differences — from my experience, there is a huge difference with the level of cooperation, transparency and openness culture after working F2F and actually connecting the face to the person you are working with remotely (even if it is for a limited time). So the trick is — JUST GO THERE! meet, work and solve problems together. Also, having an offsite activity (even just drinking a beer together) can be a cure for complex disagreements and elimination of antagonism.

Facilities & Processes

Some of the main obstacles that makes it harder to integrate is the cumbersomeness of the ongoing processes and availability of common facilities which involve several parties, making sure that all stakeholders are in sync with status/milestones and getting to a quick decision when disagreement arises.

What can we do to eliminate these obstacles?

Minimize the dependency to increase velocity — as different components/products of the unified solution require joint efforts (like E2E testing, debugging and bug fixing etc.), joint processes (like design or project timeline definition etc) or common facilities (like Labs or licenses), we need to try and minimize the dependencies so the inevitable joint parts will be completed as quickly and smoothly as possible. It is understandable that a complete solution cannot be developed by a single team. Nevertheless, there should be a clear understanding and agreement for the responsibilities and expectations from each party, including the integration points. Also, it would be better to define each component it in a way so each can function by its own as well as part of the unified solution (for example: UI & FE vs BE, agent vs server, services vs consumers and more).

Definition of done (DOD) is a list of requirements that must adhere for the team to call something “complete”. DOD for delivery of any kind should be clear whether we are dealing with design doc, tested components or the needed support after the products are delivered to production. Example for that can be completeness of automation tests coverage for component’s sanity and performance testing before delivering a product or security evaluation analysis as part of delivering a design doc.

Automation…automation… automation — I could dedicate a full article related to test automation foundations, how automation is important and which techniques exist to build a good automation infra (maybe next time :-) ). In a nutshell: no code will scale, no quality may preserve and no products integration can be sustained without automation. In order to deal with integration of several products which are being changed on a regular basis (each has a variety of supported versions) there is a strong need to have a solid infrastructure as well as automated sets of tests running 24/7 on several set of labs to indicate system health. Another benefit that comes with good automation is debugging issues and understanding quickly “who broke the main branch”. Also, in order to remove dependency, the different products should not rely on each other for testing/quality approval, so teams should consider developing tools and simulators to test their parts functionally as well as under performance.

Sharing information — Sharing is caring not just because it helps maintain the knowledge and increase the velocity, but it is also a good way to build credibility and trust between the teams:

  • Documentation — Documentation can be developed in many ways and there are a variety of tools that exists to store them. From my experience, when documentation is being done as part of the code base (for example, for RESTful API — Swagger, GraphQL/gRPC API — schema definition) it is usually easier to maintain updates and preserve consistency.
  • Code repository — there should be access to all relevant employees of the solution to the code repository. This would be beneficial for knowledge sharing/backup, getting constructive feedback and also reducing the bus factor.
  • Common Labs — Common labs are needed for E2E testing and actually approving the integration and solution before delivery to production. My two cents here are: there is a need to maximize the use of automation tests in these labs while running the latest and greatest code and there should be 24/7 support when issues arise to avoid blocking teams.

Team interactions — As already mentioned, working together F2F can create miracles, but in case this cannot be done on a regular basis, there should be recurring meetings which are needed to result in a clear syncing about what is going on, debating on solutions, coming to an agreements and making common definitions. Nevertheless, we should keep in mind not to exaggerate with meetings and make sure they don’t waste peoples’ time (especially when time differences are involved). There are two kinds of meetings I recommend keeping on the calendar:

  • Design meetings — A design review is one of the most crucial tools for ensuring good quality, scalability, and security. Remember — Prevention is cheaper and better than the need to rewrite (especially after issues from production arise). On top of that — design meetings are an important part of knowledge sharing and teams’ collaboration. If you are concerned about velocity, I can assure you that in the long run it is worth the effort. Plus — there are a set of recommendations that can ease the process, like having a common template or a defined decision maker when there is a disagreement.
  • Project status meetings — To understand overall project status and making sure all are in sync, where we are standing vs the defined project’s milestones (planned vs actual). These meetings should be short, focus on blockers and can be better facilitated by a great project manager.

On top of meetings, the on-going communication can be done by a variety of tools. From my experience Slack tool (real time messaging, file sharing and powerful search) is doing a great job. It is a great interface that can easily be integrated with other tools and ease our day 2-day work: dedicated groups can be created per theme and you can integrate the automation system and even bug system into this tool to be notified about system health and issues.

Technical

Design things right — here are a few technical points to keep in mind:

  • First and foremost — there is a need to define a standard (quick & light) design process, design template and make sure it is visible for all. Good design process does not impact velocity, on the contrary — it saves precious time (see more details in “Facilities & Processes”).
  • Components communication — The basic guideline designing a product that can integrate easily is having an API layer which means a set of functions and procedures allowing the communication with other products and applications by creating access its features, data and services. API should be simple and easy to reason about, yet ready for extension with the growing features and functionalities. Also we should keep in mind backward compatibility, as at any point in time might have different products of different versions deployed.
  • Security assessment — Products are being written by human beings, hence tend to have vulnerabilities. Threat modeling process is a repeatable process to find and address all threats to the solution by which potential threats can be identified, enumerated and prioritized, or in other words: fixing it before it gets worse. This process it crucial as part of the design to find security threats in the system as early as possible (when there is time to fix), prevent re-factoring and re-design throughout the product security life-cycle and deliver more secure products. From my experience — STRIDE model is very helpful so you might want to check it out.
  • Performance and Scalability — In every system performance and system scalability need to be taken in account as part of the design for understanding if there are design trades we need to make, ensuring the components meet well under perf — each can scale while no one becomes a blocker, specific technology we need to consider for better resource utilization and more.
  • Resources vs Cost — on top of system performance and scale, design should make clear assessment about required resources (especially under performance), expected load model and estimated price. We should also take in account that cost is not always linear with the growth of the load model and can turn exponential at a certain point.
  • Supportability — Lastly, as already mentioned, right documentation, clear logs, metrics, KPIs and visualization should be used especially when integrating products for better supportability, collaboration and of course — increasing velocity.

Where is the innovation?

Employees desire to be inspired from the products they are creating and feel that the solution they are building solves real customer’s problems while involving the top edge of technologies. The problem with integrations is that innovation is not always the case. At the end of the day, we need to make sure that an orchestration involving several components is working well and sadly, innovation is not always involved.

It’s all about the people… we want to hire the best and keeping them happy is always a challenge — the secret to keep them happy and inspired during such times is giving them MEANING! Making sure that employees have meaning is one of the biggest challenges in today’s competitive workplace. So how can we create meaning when innovation is not involved?

  • Connecting employees to customers to better understand the necessity of the holistic solution and how it will solve real customers problems better and quicker. This can be done by creating direct interaction (Yes! Also sending engineers to customers and hearing what they have to say).
  • Encouraging employees to contribute to the solution and expend it by creating a road-map. For example: What other products/feature can we build for this solution, how can we create unified E2E testing system to simulate customers’ env and increase product quality? Can we create a unified pain of glass UI for better look & feel?

— — — — —

Thanks for your time! I hope you enjoyed reading this article and got some tips.

I would love to hear from you if you encountered some of these challenges, how did you solve them and what best practices you find useful.

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Initial insights from our first trial of Self-Organising Teams

How To Double Down On Culture As A Leader During COVID-19

Leaders need to shine a light through culture in COVID19 darkness

The World Needs Your Leadership

Find Your List of Leadership Qualities

What does a business coach do?

“Five Steps One Can Take to Become More Resilient”, with Toby Gabriner and Fotis Georgiadis

Why You Can’t Motivate Your Way to Success Part 2

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Cohavit Almagor

Cohavit Almagor

More from Medium

2022 Digital Workplace Trends

Enterprise software should be enable employees to access routine tasks and information wherever they are

Starting Employee Journey Mapping at Work with Jeannie Walters

OrgDesign — Signal Processing

Continuous Improvement via Operational Awareness