What is DevOps?
DevOps is a way of delivering software that includes people, process, right technology, and automation working together to reduce time to market features with quality.
The traditional models of delivering software (e.g. ITIL, PRINCE2) are very old (>20 years). The traditional model also involves multiple handovers and silos in duties. The software space has changed a lot now (the concept of Cloud, i.e. infrastructure as code did not exist back then, today many open source tools exist to take care of monitoring, implement CICD, agile process for quicker software delivery, etc.).
DevOops encourages teams to think in “from concept to cash” model. This is a core principle in lead product development. The concept of from concept to cash model, makes teams to own their product/service from ideation to retirement and not just write some code and deploy into production.
DevOps and Digital Transformation
In the world of Digital transformations, every industry is disrupted today. AWS disrupted data centers, Amazon disrupted retail business, Uber disrupted taxi business, Airbnb disrupted hotel industry, data disrupted traditional telco business, etc., to name a few. The disruption is possible with the availability of high speed Internet, new media like smart phones, the way software is delivered such as play store and app store. Customers demand more from their service providers and are able to share their opinion about a product or a company through social media (Facebook, twitter, customer reviews, etc.) and it impacts the business. Organizations should be equipped to provide what their customers want and quickly.
Enterprises who still follow the traditional model of releasing new features once in six months or every year will lose out to competition even though their ideas may be very sound and great. DevOps is the core to success of Digital transformations. DevOps concepts such as Agile, CICD, Lean, enables new ideas to be built, put in hands of customers and feedback received quickly. DevOps has proven to have less bugs due to better testing through automation, improve reliability of the software, team morale and efficiency also improved. This enables confidence to release more frequently. Organizations who do not take advantage of these modern process, technology and tools and change the way they delivery service to their customers will be over taken by their competition.
In order to embrace the DevOps journey at scale, enterprises need to change their team structures, the entire software delivery flow, all the way from requirements gathering to architecture, design, development, testing, operations, security, and support. Implementing these types of changes at enterprise level is not easy.
There are many factors that are barrier to big transformations like DevOps. Some of the common ones are
- Fear of change – Success of DevOps lies with willingness of different teams in organization to change their traditional model of working. Some people are afraid of the change and do not want to give up their status quo. They are afraid of uncertainties that comes along with the changes and/or not willing to learn new things. People are also afraid of failures. The management have to provide support to teams to take risk and do not penalize them for failure and allow them to experiment new things for the betterment of the company.
- Skills – Another factor that hinders in adopting DevOps is skills. DevOps encourages as much as automation possible in all possible steps. This means employees have to have right skills or learn new skills to build automation. The lack of talented people in the industry makes is hard for enterprises to change to DevOps fast. Organization should invest in their employees and allow them to acquire new skills so that they can give back.
- Inflexible Command and Control – some organizations have a rigid process which prevents teams from doing things fast. Examples: Projects and funding must go through different approval process, which may be time consuming. A Software has to be reviewed and approved by operations before put in production, which may take long time for turn around. Development team cannot deploy software into production, etc.
- Misunderstanding of DevOps – Some people think DevOps means eliminating all the process and functions, teams do whatever they like to do. This is not what DevOps aims to do (refer to DevOps definition in the top). To practice good DevOps at enterprise level, some process must be strictly followed to keep things running smoothly at all times. At the same time, the waste and wait times between different stages of software delivery should be eliminated and automation employed as much as possible so that errors can be eliminated and speed can be achieved.
- Missing Sense of Urgency – Some organizations may have vision and strategy to move towards Digitization and DevOps but they may lack the sense of urgency. In today’s digital world investors expect the company to make profit every quarter and the customers keep demanding more for less. In order to meet the demands of the both the ends, organizations have to act now, continuously innovate and question their status quo every time. Some enterprises engage outside consultants for long period (in greater of 6 months to 1 year) to define a strategy for the company’s Digital and DevOps journey. While outside consultants may bring new knowledge and outside view, it has to be done for short duration and acted upon quickly.
To add to the complexity, enterprises need to make these necessary changes while continuing to meet their customer demands. Being said that, it doesn’t mean it is impossible to implement devops in big enterprises. There are big enterprises who have adopted DevOps practice at enterprise level and successful. ING, Verizon, NetFlix, Amazon, GE, Walmart, Target, are reaping the benefits of DevOps. Many articles are available in the Internet that talks about success stories, what the organizations have done to adopt DevOps. Here are some references:
Ideas for Enterprise Level DevOps
Big enterprises who want to follow DevOps ways of doing work have to address the barriers listed above. In the below sections, we will look into some ideas on how to practice DevOps at scale to get the maximum benefit.
In order to achieve efficiency with DevOps, organizations should eliminate the silos in roles and teams like Dev, test, ops, security, product, etc. Lesser the number of teams of different function and lesser the hierarchy it is better. Time will be saved between hand offs and work can shared among the members easily. A team (a group of people who own a service/product/application from concept to retirement) should be given some defined SLAs and held accountable to meet those SLAs. Some of the examples of the SLAs could be, less than one percent error / roll back or roll forward changes for new deployments or changes. Able to respond and fix production bugs in couple of hours.
Common vision and roadmap for Senior Leadership and teams
As we have said earlier, Senior Leadership sponsor and support is a must for getting bigger benefits from DevOps. A common vision and strategy should be defined and all the executives and their teams needs to sign up for that. Example of a common vision could be “No more Enterprise releases”, “There is only one layer between business and the engineering team”, “No more manual testing”, etc. The how part of achieving the common goal can be left to the teams to use their creativity and new approaches to make it happen.
Get outside help
Organizations should look for outside help to augment the internal skills. Neither outsourcing the entire DevOps work nor fully depending on the existing in-house talent will give the big bang for the company. In order to move fast and take advantage of external knowledge, companies should find a right partner to work with. This will allow either to delegate the day to day work to the partner and the internal resources spend time in adopting the new DevOps journey related work or make the partner do all the foundation work DevOps and teams can pick up from there.
Invest in people
Enterprises who want to do Digital transformation and practice DevOps should invest in their people. Not all members will understand DevOps and it will take time before majority of the organization practices DevOps. Getting skilled people is costlier and time consuming. Investing in the existing team will provide good ROI with increased team moral, better software, etc. Companies should provide necessary training (some could be mandatory), opportunity to change roles, and time for people experiment new ideas (e.g. Google provides times for their engineers to work on anything outside their regular project work).
Enforcement vs Autonomy for teams
Strictly enforcing one tool or language for whole of organization will limit the innovation and make it difficult to find large pool of good talent. At the same time, allowing each team to use their own tool or their own process will make it difficult for big enterprise to keep track with many teams. Allowing teams to use their own choice of solution will make it difficult to rotate people between teams. The right solution is to find a balance between control versus innovation and freedom. Enterprises should find few tools (a small set, e.g. Java, golang, node for back-end programming, angular, react, for front-end, etc) and allow teams to pick what they need from the finite list. A simple and clean process need to be defined for teams to bring additional tools or languages for their work so that improvements can be made in the current solutions.
The legacy model of software life cycle management such as ITIL, Waterfall, PRINCE2 are focused on risk averse and had many controls built into it. This will allow different groups accountable for specific function. With the DevOps model, all the hierarchies and layers are demolished and team is given more autonomy. At the enterprise level, not all teams will show case higher level of responsibility and maturity. Customer expectations cannot be compromised at any time due to change in way of doing things. Organizational level controls need to be established using technology and process to help teams to provide superior product and customer service with DevOps way.
Enterprises can use in-house or COTS solutions to enforce their controls. For example, Netflix built their own solution (chaos monkey) to randomly delete servers to make sure their applications are resilient to failure of underlying cloud servers. https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116
Texas A&M university uses Machine Learning and Artificial Intelligence to identify and triage security incidents. They were able to bring security incident resolution from few hours to 10 – 20 mins. ISCACA has 10 DevOps controls recommendation for companies who wish to embrace DevOps practice. http://www.isaca.org/About-ISACA/Press-room/News-Releases/2015/Pages/ISACA-Report-10-Important-Dev-Ops-Controls.aspx
Some companies follow dual model IT. Some legacy COTS products follow old model SDLC and modern software (e.g. inhouse built services, web applications, mobile apps, etc) follow DevOps approach. It is strongly recommended that Enterprises follow only single model. In order to provide great products to customers, different systems (Legacy COT, SaaS, inhouse solutions) have to work together. Following dual speed IT will create disconnect in architecture of systems, different team structures, and rate of software delivery, etc., which will cause more confusion than enabling all the teams to work together to deliver software faster. Dual speed IT will confuse business and create complexity for them to figure out which part of IT to work with. It will also create road blocks for collaboration between different parts of organization, and lowers employee morale too.
Right Tools and Technology
Enterprises can gain most out of DevOps by using right tools and technology. Though teams have autonomy to try new tools and techniques to foster innovation, at the enterprise level certain tools need to be used to gain most. Some of the areas where organizations have to choose proper tools would be, source code control systems, requirements gathering, central logging, central monitoring and dashboards, and team collaboration (communication and knowledge sharing).
Single source control system will allow different teams to search and reuse others code without duplication. It will also allow security teams to scan the repos for any security violation and notify teams on a timely basis.
Using a good collaboration software will enable teams to share knowledge, easily reach others
A central dashboard can enable everybody in the organization to see what is going on with their systems and software. This will bring awareness to the entire organization.
Whatever the tools and technologies are chosen, it should be easy for the teams to adopt. Tools can be chosen based on the inputs from different teams and based on consensus or majority.
At the same time solutions should be there to catch teams who violate their SLAs. E.g. for all the issues raised for a given team what is the violation in SLA to resolve them and based on threshold the teams’ manager or one up should be notified of missing SLAs.
Adopting DevOps practice at an enterprise level is a big task. But it is not an impossible task. When enterprises adopt certain fundamentals like changing their team structures, automating any repeatable work as much as possible, supporting the teams to take risk and failure fast, simplifying SDLC process, etc., DevOps practice can be implemented at scale in any big enterprise. Enterprise also need to define and build some controls, use right technology and tools to keep things flowing smooth and do not have any customer impact.
High performing software engineering teams
A HPSE team will always exceed expectations, have less conflict, high team morale, higher quality of products, fast delivery and cheap cost. In order for a team to be performing at higher performance, three factors need to work in tandem: People, process and technology. This assessment helps teams to evaluate where they stand in these areas and come up with actionable plan to perform at a higher maturity.