Our Practical Approach to CMMI Dev Certification

Intended Audience

• Anyone trying to improve their engineering practices to reduce project risk and increase customer satisfaction.
• Companies interested in or are in the process of getting their CMMI Dev certification.
• Development team members who want to understand how to be compliant with CMMI Dev or help simplify their procedures.

What is CMMI

The Capability Maturity Model Integration (CMMI) is a process and behavioral model that helps organizations streamline process improvement and encourage productive, efficient behaviors that decrease risks in software, product, and service development. CMMI was developed by the Software Engineering Institute at Carnegie Mellon University, as a process improvement tool for projects, divisions, or organizations. The U.S. Government helped develop the CMMI and often prefers or requires it for its software development contracts.

Why Am I Writing This?

Getting our CMMI Dev L3 certification has been part of our strategic plan to quantify our professional practices, increase quality, and reduce risk, which in turn makes us more competitive for larger federal and commercial customers. The process of becoming certified and adopting CMMI has not been straightforward and ended up costing us more than it could have. We changed consultants midstream and also changed who led the effort internally, which translates to more time and money. Our first consultants were extremely knowledgeable and capable, but they were not the right fit for us. Having finally received our certification, we found it surprising that our practices based on agile methodologies provided the foundation for nearly all CMMI Dev Level 3 requirements. Note that in order to be CMMI compliant, you have to have procedures in place and train your staff on procedures as opposed to only having practices in place. Having said that, if you follow good practices, you are likely much closer to becoming CMMI compliant than it may seem. For example, we found out that our practice could be turned into procedures, and that the artifacts we collected based on our practices, nicely mapped to the artifacts needed by the procedures. During our Final Findings call, the folks evaluating and certifying us were very happy with our efforts and also very appreciative of us presenting the artifacts for our procedures so concisely. Their affirmatively positive findings, and the fact that we were much closer to becoming CMMI compliant than we originally thought, is the reason for writing this article. We want you to know what we wish we had known, so you can go through it more effectively than we did. The root cause of the problem is that our internal CMMI team; consisting of engineering leads, management, and executive staff; really struggled to get our heads around what is really needed and how to become compliant.

Why is CMMI Confusing?

In order to become CMMI Certified, one needs to 1) understand the model, 2) develop company procedures, 3) train the staff, and 4) prepare for the audit.

In order to understand the model, you need to read the model and/or take courses on the topic. Reading through the 800-page CMMI model is daunting. The practice areas come across as vague, convoluted, and even non-committal in some places. To make matters worse, despite Model 2’s numerous references to Agile methodologies, there are places that seem to contradict it, making it feel as though you are trying to fit a square peg in a round hole. We’ve caught ourselves going to the model to put together compliant steps, but came back empty-handed. In some cases, we ended up even more unsure about our existing practices. This was discouraging because we believe that if capable senior engineers, with years of experience executing projects successfully using agile methodologies, struggled to understand what needs to be done, something is wrong. At the very least, full adoption of the procedures will be a major uphill battle. Furthermore, we have to make sure our procedures do not result in inefficient use of funds — both ours and our clients’. To develop company procedures, we tried several approaches:

• We started directly from the model to develop compliant procedures. We ended up covering all the requirements laid out in the model, but end up with procedures that we felt were too complicated for normal day-to-day use by our development staff.
• We asked our original consultants for baseline procedures that we could tailor to our needs, but they weren’t in favor of that. Their explanations of the practice areas and what we needed for the procedures were at the same level of abstraction as the model which left us spinning our wheels. Despite their immense level of knowledge, we could see that we’ll have a painful process ahead of us with a lot of time spent flying blind. For example, we would work on a procedure for a week or two only to get feedback from them asking why we did X, or Y was overdone or not necessary.
• We evaluated other companies’ processes as a starting point, but they were often too different from what we could see our team truly adopting in the fast-paced, Agile environment that characterizes the majority of our projects.
• We asked our new consultant to help us with a baseline set of procedures based on how we do things to give us somewhere to start from, but they still needed plenty of work.

That’s where we realized that we should start from the baseline procedures that our new consultant developed for use based on his understanding of what we do, feel free to completely change them to match our practices, then ask for feedback to see if we are missing anything. To our surprise, our consultant kept saying that they are fantastic and compliant. In fact, we only ended up making improvements in a few places with one small item coming back by the auditors. Our biggest lesson learned is that CMMI manages to pull off the “We can’t tell you what to do, but we can easily tell you if it is compliant or not when we see it”. This is because there are hundreds of ways to do things and be perfectly compliant, and understandably, the model makes a conscious effort to not tell you how to do anything.

Preparing for the audit can be challenging as well because you have to apply CMMI to all your in-scope projects. Knowing the difference between project records, artifacts, the interview, and the presentation can be confusing as well.

CMMI Dev Level 3

Achieving CMMI certification requires that processes are documented and repeatable. For development projects, this results in a consistent structure that serves to reduce risk, increase quality, and aid in overall project success. CMMI has multiple levels from 1 through 5. Level 3 is arguably known as a good balance of structure without excessive burden. For example, in the Peer Review practice area, level 2 only requires performing peer reviews while level 3 requires analysis of peer reviews for lessons learned. In our opinion, gleaning these “lessons learned” is well worth the effort, as it feeds back into a culture of continuous improvement. The Causal Analysis & Resolution practice area requires “Analysis and improvement proposals” for level 3, while levels 4 and 5 require “Statistical and other quantitative techniques”. While this additional level of analysis is necessary for some projects and certain industries, rolling them out across all our in-scope projects requires additional cycles which are not always the best use of resources.

CMMI Dev is broken down into nineteen Practice Areas, within two distinct categories. Nine practice areas are scoped at the project level, four practice areas are scoped at the organization level, while six practice areas apply to both levels and are randomly assigned to the project or organization level for your audit. For Example, the Technical Solution practice area exclusively applies to the Project level; Governance exclusively applies to the Organizational level; while Process Quality Assurance and Managing Performance and Measurement apply to both the Project and Organization levels.

Dedicated Project Level:

• Planning (PLAN)
• Monitoring and Control (MC)
• Requirement Development and Management (RDM)
• Estimating (EST)
• Technical Solution (TS)
• Product Integration (PI)
• Configuration Management (CM)
• Verification and Validation (VV)
• Peer Review (PR)

Dedicated Organization Level:

• Process Asset Development (PAD)
• Process Management (PCM)
• Governance (GOV)
• Implementation Infrastructure (II)

Both Project and Organization Level:

• Risk and Opportunity Management (RSK)
• Decision Analysis and Resolution (DAR)
• Causal Analysis and Resolution (CAR)
• Process Quality Assurance (PQA)
• Managing Performance and Measurement (MPM)
• Organizational Training (OT)

Here we’ll explain our approach to developing complaint procedures that fit our organization, with details on three of the Project level practice areas that we feel paint a clear picture of how the rest of the procedures can be developed. The procedures we’ll cover here are Technical Solution (TS), Monitoring and Control (MC), and Planning (PLAN).

Our Approach

Going into this we determined that we would consider the certification effort successful only if all our engineers make our processes part of their daily habits and activities. In order to get there, the procedures needed to be concise and intuitive so that they are easily understood and consistently applied. We also hoped to save our engineers from needing to refer back to the model and face the same confusions that we faced. We wanted the steps in each procedure to be so intuitive that simply reading the title of each step maximizes the ability of engineers to recall and internalize the steps. Additionally, as former lecturers, we believe in learning through examples, especially for more complicated topics, so our process documentation is rich with examples.

Technical Solution (TS) Practice Area

Let’s look at the requirements of the Technical Solution practice area directly from the CMMI model, which is a relatively straightforward practice area:

At a first glance, it may look like the requirements lay out the steps for TS procedures, but at a closer inspection, you’ll find that 2.1 replaces 1.1 and 3.1, 3.2, 3.3, 3.4, 3.5, and 3.6 replace 2.1 and 2.2. This is seen in various practice areas where a higher level sharpens and refines some requirements in the previous level to replace them in order to raise the bar. In contrast, 2.3 added a new requirement to “Provide guidance on the use of solution”, but it is not included in the level 3 requirement. In order to be compliant for level 3, all the requirements in previous levels still need to be met. As a result, we developed our procedure to be as follows:

Technical Solution Procedure

1. Define Criteria for Design Decisions
2. Design Alternative Solutions
3. Perform Build, Buy, Reuse Analysis
4. Select Solution
5. Develop and Manage Information For Implementation
6. Design Interfaces and Connections
7. Provide Guidance On The Use of Solution

As you can see in this example, our procedure is designed to be compliant with the model. We’ve converted the requirements into steps that are concise yet descriptive so that the development team can avoid having to refer to additional documentation. Our actual Technical Solution Procedure is a document that states the CMMI intent, value, focus, and the requirements table straight out of the model, then lists these 7 steps, and explains each step in a short paragraph for those who can use the explanation.

Monitoring and Control (MC) Practice Area

As another example, let’s take the Monitoring and Control practice area, which is critical for adapting Agile practices to work within the framework of CMMI:

This practice area is a bit more difficult to translate into steps, especially while still capturing the intent of an Agile approach. However, we don’t have to base the steps on the requirements and can simply define the procedure as:

Monitoring and Control Procedure

1. Schedule
2. Prepare
3. Perform

This simple procedure in the context of Agile means that you schedule the Agile ceremonies such as Sprint Planning, Sprint Review, Sprint Retrospective, Daily Standup, Product Owner Acceptance, and other ceremonies that are important to your organization. The Prepare step is about gathering information and potentially performing demo dry-runs. And the Perform step is about doing the actual meetings. If you are already doing Agile, you likely already meet most if not all the requirements in the table. If you are not already doing Agile, or even Waterfall, in a holistic manner, the requirements table lets you make sure you are not missing anything.

Planning (PLAN) Practice Area

The Planning (PLAN) practice area produces a living document that takes the contract’s Statement of Work as input and turns it into successful deliverables. The following are the practice area’s requirements:

This practice area is not as easy to convert into a procedure compared to the Technical Solution practice area. If you already follow a typical Project Management Plan based on the PMI PMP, you’re most likely already compliant. We decided to define our procedure as follows, based on our experience and guidance from our professional CMMI consultant. In order to create the PLAN for a project, we define the following nine steps:

Planning Procedure

1. Work Breakdown Structure (or Product Backlog)
2. Deliverables Per WBS
3. Resource Staffing Plan
4. Stakeholder Communications Management Plan
5. Risk Management Plan
6. Issues and Pending Decisions Plan
7. Change Management Plan
8. Records Plan
9. Technical Approach (Optional, for customers)
   1. Agile Methodologies
   2. PE Practice Area Overviews

We also want to explain how we plan to successfully deliver to our clients to get their buy-in as opposed to just using CMMI as a badge to make us more competitive. There are some customers who don’t have experience with Agile methodologies or even with Software Engineering; they might assume the process is similar to building a house based on a pre-existing template. This is why we added a Technical Approach to our plan. While this is not a CMMI requirement, we find it valuable for explaining our Agile approach and for providing very brief overviews of the more relevant practice areas.

Artifacts for CMMI Evaluation

During the course of performance on any project, important records and even decisions are typically accumulated in a variety of different systems. These include Jira, Confluence, GitHub, Google Drive / MS Office (for documents, spreadsheets, diagrams), Figma, Photoshop, Slack, MS Project, Redmine, Trello, etc. When it comes time for you to demonstrate compliance, you have to put together Artifacts that assert that you are doing the work. Artifacts are often sample screenshots that collectively show that you meet the needs of each practice area. That is, you don’t show all your Jira tickets and provide access to your client’s Intellectual Property or classified information. Instead, you take a screenshot of a few tickets that are either clear of IP and classified information, or the specifics are redacted in the screenshot. As you can imagine, in order to show compliance, you can end up with tens or hundreds of screenshots to paint a proper picture for your consultants and auditors. The common approach is to put the screenshots in a folder so that your consultants and auditors can sift through them and check for compliance with the requirements.

Project Affirmation

The purpose of the audit is to document and demonstrate that your organization and projects are compliant with corresponding practice areas. The affirmation by the auditors is usually done by interviewing designated leads in your organization. We had a lead for project-level practice areas and another lead for our organization-level practice areas. Some auditors allow you to put together a presentation for each practice area and ask you questions as you present. Other auditors prefer to start with the interview, but allow you to or may appreciate a presentation handy covering the practice areas and the artifacts.

We started with collecting artifacts in a folder for each practice area only to realize that it was too burdensome to go through them all in order to get a clear picture of how the requirements are met. Note that you may end up with about 250 artifacts. We wanted to find a way to make it easy for ourselves to verify that we have met all the practice area requirements, make it easy for our auditors to perform their affirmation, and most importantly, for development staff to have clear examples of a CMMI compliant project artifacts that are easy to learn from. We decided to use wiki pages instead of folders and artifact files. Each wiki page has a table of contents that shows the procedure steps for the reader to easily see the numbered steps, as well as give the readers the ability to jump to a specific step to see the corresponding artifacts for that step. This means that our development teams do not have to click through numerous artifacts to understand CMMI compliance and can instead scroll through a single page with embedded steps and see all of the artifacts (screenshots from different platforms and text from meeting notes) in the proper order associated with the right step of the procedure. We also provide a short overview paragraph in each practice area’s wiki page as such:

These pages were essentially our project presentation as well. Our consultant, auditors, and the 2 reviewers in training absolutely loved this approach and were thankful for making it so easy for them to check for compliance.

Guidance to Our Development Teams

We tell our developers that if they are following CMMI properly on a project, they should be able to generate a wiki page for each practice area showing the Artifacts which are examples of how they are meeting each of the 14 Prominent Edge CMMI procedures.

Wrapping It Up

We’ve explained our approach to CMMI and some of our lessons learned with the hope that it helps you get your certification more efficiently than we did and be more effective. If you already have good practices, you already have a solid foundation for your CMMI processes. You should rely on your practices that fit your culture to develop your processes with the help of a practical CMMI consultant. Keep in mind that the true value of CMMI is only realized when you adopt your processes across the organization. We hope that you have found it useful and we’d love to hear your thoughts.