The Evolution of DevOps - The Golden Age

Let's wind the clock back a couple of decades to the late 90s and early 2000s, a time when most organisations had a single gatekeeper: the system or IT administrator. If engineering teams needed to run their applications on production servers, they had to ask for help, creating silos that led to poor experiences on both sides: miscommunications, a decrease in engineering velocity, and slower time-to-market.

In 2006, the cloud emerged as a trend with AWS launching its first few services. This provided the foundation for DevOps concepts to establish themselves as the new golden standard for engineering teams. Since then, various tools and solutions have emerged to help alleviate problems such as scalability, operability, and availability. Engineering teams became more agile, and were able to solve various technical problems:

• Monolithic systems are broken down into tiny pieces that interconnect (Microservices, Microfrontends, Mono-repos and Multi-repos)

• Infrastructure as Code has taken centre stage in promoting GitOps. By leveraging Git and third-party software tools, developers can write less code and focus more on business logic.

• Containerisation has become popular because it allows you to ship your tech stack to any server of any type and scale it on demand to handle exponential numbers that are inexplicable.

• CI/CD tools have become the automatons for performing all of the heavy lifting required to build and ship your code to production.

Developers can now deploy and run their applications and services end-to-end with confidence. It's a "You build it, you ship it" approach. The engineering teams were thriving. Productivity increased, and innovations were being shipped out faster.

However, it got to a point where it got over-sophisticated. Let's consider the perspective of developers who need to interact with and utilize various tools, methodologies, technologies, and processes. The problem is staring us right in the face: cognitive load. This refers to the amount of information that working memory can hold at any given time.

Developers often get overwhelmed with the task of mastering multiple different tools simultaneously, including CI/CD pipelines, Terraform modules, and Helm charts, just to create a new environment and deploy resources. Conversely, when implementing a new feature, there are a few things a developer must know beforehand:

• Which microservice APIs do they need to interact with?

• Who owns those microservices?

• How do they roll out new cloud resources with Terraform?

• What environments are involved in this entire process?

In a typical organisation that uses DevOps innovation and tools, finding information can be a cumbersome process, even with the right documentation available. To address these issues, a new practice has emerged - platform engineering.

What is Platform Engineering?

In simple terms, platform engineering is a discipline for designing and building internal platforms. These platforms are designed to enable self-service capabilities for software engineering organisations for rapid development and deployment of software products. Platform engineering aims to optimise the developer experience and accelerate engineering teams' delivery of customer value. This is achieved by providing common frameworks and components as part of the platform offering to engineering teams.

From the Ashes of DevOps: the rise of Internal Developer Platforms

In today's tech industry, platform engineering is becoming increasingly important. This new discipline is responsible for reducing developers' cognitive load while interacting and delivering software end-to-end. In other words, platform engineers will collaborate to create an Internal Developer Platform (IDP). This platform will allow developers to seamlessly bridge the gap between development and operations.

Developers can interact with Identity Providers (IDPs) and choose the appropriate level of abstraction to run their applications and services, based on their requirements. For example, they can configure a specific set of cloud resources using Terraform modules. Alternatively, junior front-end developers can self-serve an environment that comes fully provisioned with everything they need to deploy and test their code without worrying about the underlying infrastructure (such as whether their applications are running on a Kubernetes cluster.) Other possible abstractions include:

• Rolling back and debugging

• Enforcing role-based access control policies

• Adding environment variables and changing configurations for microservices

• Adding dependencies to services (sidecar monitoring tools, agents)

This list can go on, but an Internal Developer Platform is essentially the combination of all the technology and tools that a platform engineering team brings together to create a seamless experience for developers, paving a golden path. This experience allows developers to self-serve with minimal cognitive load, while platform engineers provide the necessary glue to bind everything together into that experience. This approach avoids the need for everyone in the engineering team to operate and understand the entire toolchain, making it more efficient for everyone involved.

To create a robust and easily comprehensible self-service platform, platform engineers must approach the platform as a product. And just like building any product, there are several criteria to consider during the development process:

1. Self-service: The platform should be designed with developers in mind, enabling them to easily self-serve the resources and tools they need.

2. Automation: Platform engineers should strive to automate as much of the platform as possible, reducing manual intervention and increasing efficiency.

3. Scalability: The platform should be designed with scalability in mind, providing the ability to easily scale resources up or down as needed to meet the needs of the organisation.

4. Flexibility: The platform should be flexible enough to accommodate a wide range of use cases and technologies along with different teams’ needs and preferences.

5. Security: Security should be a top priority for the platform, with role-based access control policies and other security measures in place to ensure the system does not get accidentally deleted or expose any sensitive information to the public, thus protecting data and intellectual property.

6. Standardisation: The platform should provide a consistent experience across all teams and projects, promoting collaboration and standardisation among teams.

7. Ease of maintenance: The platform should be easy to maintain and update, with clear documentation and processes in place and not require extensive manual work to maintain if needed.

8. Interoperability: The platform should be interoperable with other tools and systems, making it easy for developers to integrate with other technologies in the organisation’s tech stack.

9. Ease of use: The platform should be easy to use and not require extensive training or learning, providing an intuitive experience for developers.

10. Cost-effectiveness: The platform should be cost-effective and not require excessive resources to maintain, providing a high return on investment for the organisation.

By considering these criteria, engineering teams can build a robust and easily comprehensible internal developer platform that promotes collaboration and standardisation across software development.

Platform engineering is here to stay

A common misconception is that platform engineering only makes sense in large teams. However, even if your team is composed of just a few developers, having a platform team and IDP may not be overkill. As soon as your organisation starts growing past the 50-developer mark, an Internal Developer Platform (IDP) is probably something you should consider sooner rather than later.

Adopting platform engineering early on can help engineering teams avoid unnecessary hardships, such as DevOps engineers leaving and your organisation is unable to deploy for weeks at a time. It's a necessary investment that needs to be considered today. With platform engineering, organisations can benefit from the following:

• Speed up the onboarding process for new developers by providing visual representations of systems and their lifecycles.

• Enhance the developer experience in building bespoke systems.

• Drive adaptive governance for business needs.

• Resolve incidents faster with overall visibility of services and their owners.

• Accelerate time-to-market for products, allowing your organisation to stay ahead of its competitors.

Conclusion

The rise of platform engineering is a key trend in the tech industry. Engineering teams that adopt these practices can stay competitive and deliver high-quality software systems. By doing so, they can benefit from increased efficiency, scalability, interoperability, consistency, flexibility, and more, all while maintaining a competitive advantage.

Platform engineers are a fundamental and crucial building block for every organisation. More importantly, they are enablers of DevOps innovation, allowing the business to remain state-of-the-art without compromising on the initial promise of DevOps.