Welcome to my blog page.
You will find all my blogs here, related to Technology, ranging from Platform Engineering, DevOps, SRE and much more. In the future, I aim to also add guides and how-tos revolving around these topics.
Happy reading!
Starting Your Internal Developer Platform Journey
If you work in the Tech industry, whether you provide a service or deliver a product, you have probably heard of the new Internal Developer Platform or Platform Engineering trend. Platform Engineering revolves around enabling self-service for developers by providing tools and workflows and integrating them into an IDP. You might be one of the lucky ones who is adopting the trend in your organization and seeing first-hand the results of that adoption.
But what if you are not? How do you start? And is it necessary to adopt this new trend? This post aims to shed some light on how to start this journey and whether it makes sense to you and your organization.
What are IDPs?
I explained what constitutes an IDP in my previous blog post, so please read that for more information. But in short, an IDP (Internal Developer Platform) is a simplified interface that offers a specific set of tools and resources with ease and efficiency.
This trend, like any other in the tech world, has advantages and disadvantages, usually depending on your use case. This post can assist you in finding a path toward adopting an IDP — especially if you have no clear guardrails for development teams, high operational costs, and other pain points.
Know the problem before the solution
Since the rise of DevOps teams within organizations, many issues related to the rate of deployment, build automation, and time to market have been solved (to a certain extent). Organizations then transitioned to end-to-end development teams fully responsible for their applications from ideation to production.
This transition introduced a host of new problems that might resonate with you: the lack of unified tooling, no defined standards for security, monitoring, logging, documentation, and compliance. This has created a fragmented landscape, resulting in varied experiences for developers, higher barriers to entry for newcomers, elevated operational costs, and an underperforming organization that becomes slow to respond to any compliance or governance change.
If you face this at your organization, then an IDP might be the light at the end of the tunnel.
Start with the people, not the tools
Using and adopting new tools is usually filled with excitement. It is the same feeling you have when your favourite game drops. You are excited to try out all the latest features and different scenarios and progress as much as possible.
However, when starting with IDPs, do your research. Are you looking to build your own from scratch or buy a SaaS solution? With all that, you tend to forget the most important aspect of this change: the people. They should be your starting point. Your developers, who are your primary stakeholders, will shape your IDP.
Listen to them, collect their feedback on the current processes, and evaluate where the gaps are. Providing a quick win for developers by standardizing a long-term blocker will gain you a lot of momentum and buy-in from them.
For example, you could provide a standardized way for developers to run a security scan for their Docker images in their existing CI/CD pipelines without changing their way of working much at all. Offering this process as a self-service product where it is maintained, versioned, has clear documentation, and is always available could be your first step.
Afterward, start identifying enthusiasts in different development teams who can help create comparable products to make developers’ lives easier. Whenever you gain traction and your small team of enthusiasts is growing, look toward forming this central platform team that will start maturing your existing products and creating new ones.
By now, you might have noticed that I have used the word product multiple times. If you are wondering why, it is because you can increase the success rate of your IDP journey by treating the IDP as a product.
Make the developers your main stakeholders and use their feedback to shape your roadmap. If you follow this mindset, you will go a long way toward success.
Measure your successes (and your failures)
As with any high-quality product, having clear metrics is imperative to identify whether it is working, if you need to pivot, or if it is a complete disaster.
The metrics that make sense will depend on the product you publish to your developers, but common ones are deployment frequency and onboarding time (be it new or existing teams onboarding the platform), to name just a few. Another way to measure success is by surveying your customers and extracting the results to determine the success.
The assumption is that you have exposed one product and are starting small. The metrics relayed back decide where to go next and how to start scaling the IDP.
Cultivate your community
We now have a successful product that is being used and is enhancing the developer experience. You might be asking yourself: “What should I do next?” For more information on what developer experience means, please refer to this informative blog by my colleague Rene van Osnabrugge.
Next is raising the visibility of the IDP and its value so that you can refine the future vision. Publish the surveys and metrics you collected before to the organization and highlight these success stories. Recruit evangelists who help promote the IDP and continue gathering developers’ requirements to determine what your next product should be.
When you establish this process of a short feedback loop, understand what your main stakeholders require, and then cater to their needs, you are well on your way to increasing adoption and engagement and making this journey enjoyable and effective.
Be mindful of maintenance, however. You must stay on top of regular patches and updates for the components of your existing products. Being transparent regarding these updates and communicating the changes beforehand, with clear documentation on what is changing, provides confidence to the developers to keep using the products and prevents losing the momentum behind your transformation journey.
Expand your vision
The transformation train is roaring throughout the organization, the IDP is scaling well, and you are on top of your roadmap. Your primary stakeholders are happy, and the significant productivity and developer experience changes are visible throughout. Congrats on this outstanding achievement! But it does not stop here.
As with all products, your IDP will go through a lifecycle, so keep collecting metrics that give you clear insights into your IDP and plan accordingly. Having one eye on industry trends ensures that your IDP stays relevant and effective.
Building a community around your IDP where your stakeholders share their success stories, the challenges they face, and assist each other is a great sign that you have succeeded.
Conclusion
The journey described here is a massive undertaking, and being armed with all the necessary knowledge and tools enhances your chances of success. I highly recommend reading more about this topic, both success stories and failed attempts. Finally, here is some personal advice from me: Read Team Topologies by Matthew Skelton and Manuel Pais, which has a wealth of information on how to form your teams, and follow platformengineering.org, which also provides excellent content around Platform Engineering.
As part of Xebia's Holistic Horizons Series
What is an Internal Developer Platform?
Internal Developer Platforms (IDPs) have been the ongoing trend for the past couple of years, and for good reasons. It is a transformative trend in software development, streamlining and optimising the development process.
For readers trying to grasp the concept of IDPs, imagine a scenario where a developer needs a specific set of tools and resources. Instead of navigating a complex infrastructure, they turn to an IDP — a simplified interface that offers exactly what they need with ease and efficiency. The IDP provides an abstracted layer on top of the complex infrastructure simplifying the context for the developers.
Starting this initiative is usually a huge undertaking, with the number of tools and SaaS platforms available. This creates a sense of not knowing where to start, and rightfully so. IDPs are specific to every team and every organisation, and there is no one blueprint that can be applied to everyone. The aim of this blog is to highlight one tool that can be a starting point of this initiative.
Enter Crossplane
A rising star in the IDP realm, Crossplane aims to unify resource management under a “universal control plane” rendering it a powerful tool in the diverse and fragmented world of cloud services (definitely, a breath of fresh air). It is a bold project, with a lot of nuances and is highly extensible.
Where Crossplane shines
The strength of Crossplane lies in its features. Having a “universal control plane” opens the route to standardisation across cloud providers. For example, an organisation might need to ensure that all cloud resources adhere to its own compliance standards. Crossplane allows you to define these policies centrally, ensuring consistent compliance across all cloud services. You no longer need to ensure your compliance standards are applied on each cloud provider you use; Crossplane takes care of that.
However, it’s important to acknowledge the learning curve associated with Crossplane. While its capabilities are vast, mastering its concepts can be challenging. The ROI, though, is significant. The ability to provision resources across multiple cloud providers while leveraging native Kubernetes features like namespaces for resource separation and a reconciliation loop to maintain desired states is a game changer. This capability is a stark contrast to tools like Terraform, which, while effective in defining intended state, fall short in areas like drift detection and reconciliation.
In a direct comparison between Crossplane and Terraform, setup and learning curve prove to be more difficult than that of Terraform. Crossplane lives inside Kubernetes, which by turn, is bound to introduce difficulties. As for the learning curve, it is much steeper than that of Terraform.
As for state management, Crossplane’s continuous reconciliation is one of the biggest advantages. The presence of real-time detection drift supersedes what Terraform provides.
A point of similarity between Crossplane and Terraform is the availability of providers for all major cloud providers. Both tools boast a huge number of providers that can be leveraged for resource creation.
Team Collaboration
Team collaboration is at the core of making the most out of Crossplane. It’s not just about adding another tool to the tech stack; it’s about enhancing the synergy between developers and platform teams. By listening to the developers’ needs, platform teams can create tailored Custom Resources, which are then consumed by developers.
These developers, your main stakeholders, will give you an idea of the abstraction level they are looking for. This will, in turn, guide you, as a platform team, on how to create these Custom Resources, keeping in mind the organisations’ governance and compliance, monitoring and logging, and CI/CD which creates a standardised ecosystem.
Developer Experience
Before delving into this section, a disclaimer I would like to point to is that any IDP, process improvement, or enhancement must have the developer as the main point of focus to enhance developer experience. A new shiny tool that promises everything does not enhance developer experience.
With that being said, Crossplane, done well, plays a pivotal role in shaping the developer experience in organisations. By abstracting complex cloud resources, the platform team allows developers to focus more on creating value and less on infrastructure creation or administration. This abstraction makes the developer’s life easier, simplifying access and management of resources without requiring extensive knowledge of underlying cloud platforms.
A case study published by Upbound1, the company behind Crossplane, claims that a client leveraging Crossplane, has allowed the client to save an estimated 11,000 hours of technical work. A typical dev or test server setup could take a couple of days, from opening a request, to getting access, and deploying your application. Cutting that down to mere minutes can greatly enhance the developer experience.
An important consideration here is the knowledge of Kubernetes within your organisation. You do not need to run your services on Kubernetes, but having some knowledge allows you to make the most out of Crossplane.
Where Crossplane falls short
While Crossplane shines in many aspects, as we have discovered above, it’s important to consider its potential drawbacks to provide a full picture. The most obvious challenge is the steep learning curve. Crossplane contains a wide array of concepts and nuances that require solid knowledge to fully leverage its capabilities. This complexity can be daunting for teams and organisations alike, as the initial investment in learning and adapting to Crossplane can be substantial, potentially slowing down initial adoption and integration.
Another challenge is the potential for increased operational complexity. As Crossplane excels in resource management across multiple cloud, it can introduce complexities in tracking and managing these resources. This complexity requires visibility and monitoring to ensure that things run smoothly.
Lastly, and as mentioned in the previous section, organisations not fully invested in Kubernetes might find Crossplane a bridge too far to cross. The dependency on Kubernetes, whilst considered an advantage, can introduce limitations or issues if the Kubernetes ecosystem faces any limitations or issues.
Conclusion
In conclusion, Crossplane represents a significant leap forward in the world of IDPs. Its ability to simplify, unify, and optimise resource management across diverse environments is greatly welcome. I believe that we are looking at the next big thing that should be approached with great caution but with a greater excitement.
Resources
[1] — https://resources.upbound.io/case-studies/case-study-grupo-boticario