In this article, we will explore the development of Internal Developer Platforms (IDPs), the emerging concept of “Golden Paths”, and how IDPs can enable Golden Paths within enterprises.
Highlights
- Enterprises are beginning to invest in Internal Developer Platforms (IDP) to reduce cognitive load and developer toil in the cloud Native World.
- Well-architected platforms are built and treated following the “Platform as a Product” approach.
- Successful organizations leverage IDPs to build “Golden Paths” for their developers.
Problems of a post-DevOps World
In a decade, we have seen growing adoption of DevOps within enterprises. The advent of the cloud fueled this trend and companies saw the advantage of using cloud-enabled services or at least building similar services on-prem. Alongside came the trend of moving to microservices architectures, and the adoption of containers and infrastructure as code practices. Public Cloud providers made available to us an ever-increasing number of services on their catalogs. Open-source platforms like Kubernetes matured, and there has been explosive growth in the amount of cloud-native tools and technology on the market. The cloud-native era was born.
These tools made it possible to tear down the proverbial wall between Dev and Ops, which is supposedly a good thing. However, reality panned out a little differently. The proliferation of tools, technology, and thought processes increased cognitive overload, especially when it came to developer productivity. Developers suddenly need to understand complex cloud-native toolchains, just to roll out a small fix or change to their application.
The advent of the Platform engineer
While most enterprises were (and still are) catching up to the promise of DevOps, some leading tech companies began building internal platform teams to alleviate the resulting developer toil. Platform engineering as a discipline was pioneered by internet-scale companies such as Netflix and Google. Platform engineers design and build toolchains and workflows that enable self-service capabilities for software developers in the cloud-native era.
Puppet’s State of DevOps report from 2021 clearly shows a correlation between team performance and the adoption of internal platforms. It states, “Platform teams are key to success at scale”, adding that “The existence of a platform team does not inherently unlock higher evolution DevOps; however, great platform teams scale out the benefits of DevOps initiatives.”
What is an IDP?
An Internal Developer Platform (IDP) is the complete collection of tools and technology glued together and maintained by a platform engineering team, to facilitate a Golden Path for developers. The platform is built in such a way to enable self-service by developers, without abstracting away context or making the underlying technology inaccessible, with the intent to lower the cognitive load for the developers. The key qualifier for an IDP is that it is internal, ie, not external or customer-facing. Its customer is internal: the application developer. Well-architected IDPs would follow a Platform as a Product approach, where the platform team builds and improves the IDP, following best practices with product management principles.
Off-the-shelf DevOps platforms, PaaS solutions like Heroku, developer portals, or similar solutions that cater to a particular subset of the delivery lifecycle cannot be considered IDP. IDPs, on the other hand, are custom-built by a platform team through combining different tools and technologies (services, open-source, self-developed, etc.), with the intent to increase developer productivity and velocity.
IDPs help enterprises to scale their DevOps initiatives, easing out the complexities of modern software development. The consensus is that an IDP supports the following aspects of software engineering:
- Application configuration management
- Environment provisioning by code
- Infrastructure orchestration
- Role-based access control
For instance, an IDP could be composed of a managed Kubernetes platform like AKS (Azure Kubernetes Service), Terraform for automated provisioning, Argo CD for declarative Gitops, Helm for package management, Keycloak for access management, and so on.
However, every organization is different and has unique cultures and challenges. This means that the platform team needs to build an IDP that addresses the requirements of the particular environment, thus making it unique to that organization. It is for this reason that I believe that products off the market shelf, claiming to be an end-to-end DevOps solution, are not likely going to address your pains satisfactorily. However, enterprises may enlist the services of external integrators and experts to help set up their internal platforms.
Treating the Platform as a Product
Putting a platform team into action and having a platform built is only the start. How does this platform team differ from a traditional DevOps team or Infrastructure team? To be truly useful, the platform needs to solve a problem. It has to be built with this intent and has to evolve according to the consumer’s needs. This is similar to building a product for a customer, who in this case happens to be your internal development team.
In practice, Platform as a Product means the platform team has to be highly customer-focused. This starts with doing proper user research. The platform team listens to feedback from the entire organization and applies that feedback to constantly evolve the platform. Organizations considering to establish an IDP should be careful to avoid creating a silo-ed DevOps or Operations team or relabeling their existing platform to one. Instead, they should focus on developer workflows and self-service enablement.
Golden Paths a.k.a. DevOps on steroids
With an IDP established, developers can finally take full ownership of their services. Mature organizations do not stop at this point. After establishing developer autonomy, the platform team focuses on providing practical best practices and guidance to do things that are often repeated across the organization. The concept of “Golden Paths” was first invented by Spotify’s engineering department, to be quickly picked up by other early adopters who have called it by different names.
The general idea behind the concept of Golden Paths is that the IDP offers custom-built and supported approaches to building and deploying a particular kind of software. The development team that ‘stays’ in this Golden Path can leverage it to its advantage to make the quick road to production, with support from the platform team included. A mature platform team will create sophisticated Golden Paths that are well-architected and fine-tuned to recurrent workflows, establishing best practices and uniformity across the application portfolio.
Industry outlook
Golden Paths will accelerate application development and delivery, a sort of DevOps on steroids. Golden Paths are a relatively new development that industry leaders are investing heavily into, but should certainly mature resulting in enterprise adoption shortly. Industry expert Charity Majors discusses Golden Paths in her seminal blog post. She states:
Platform engineering teams bring a product development process to infrastructure, and a design/UX ethos to developer ergonomics. Their job is to provide patterns and Golden Paths that make the right thing easy and the wrong things harder.
Platform engineering and IDPs are revolutionizing the enterprise software development world. Industry analysts Gartner acknowledge the importance of developer self-service and the impact of platform engineering in the cloud-native World. Platform Engineering appears on the Gartner Emerging Technology Hype Cycle 2022, and is defined as “the discipline of building and operating self-service internal developer platforms (IDPs) for software delivery and life cycle management“.
IDPs as a concept is an emerging trend, but their value is already evident and this is spurring enterprise adoption. These enterprises shall be looking to leverage their existing Ops or Infrastructure teams to build out IDPs, and to get their first Golden Path right.
Backstage and Project Unox
IDPs as a concept is an emerging trend, but their value is already evident and this is spurring enterprise adoption. These enterprises shall be looking to leverage their existing Ops or Infrastructure teams to build out IDPs, and to get their first Golden path right.
At Devoteam Netherlands we recognized this early on. In the summer of 2022, we envisaged an internal project (codenamed Unox; nothing to do with the Dutch food brand), where we set out to build a solution showcase for a modern platform. The project includes all the building blocks for an IDP, that is built with our opinionated set of technologies. The centerpiece of this platform is a developer portal, built using Backstage. Backstage is an open-source platform to build developer portals, and a software catalog for all the infrastructure in an organization. We will be sharing more about Project Unox and Backstage in subsequent blogs soon.