RFP Patterns and Techniques for Successful Agile Contracting

REV-03.18.2016.0

RFP Patterns and Techniques for

Successful Agile Contracting

Authored by members of the NDIA System Engineering Agile Working Group:

Mary Ann Lapham Keith Korzec

Larri Ann Rosser Greg Howard

Steven Martin Michael Ryan

Thomas E. Friend John H. Norton III

Peter Capell

November 2016

SPECIAL REPORT

CMU/SEI-2016-SR-025

Software Solutions Division

[Distribution Statement A] This material has been approved for public release and unlimited distribution.

Please see Copyright notice for non-U.S. Government use and distribution

http://www.sei.cmu.edu

CMU/SEI-2016-SR-025

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

This material is based upon work funded and supported by the Department of Defense under Contract

No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineer-

ing Institute, a federally funded research and development center.

Any opinions, findings and conclusions or recommendations expressed in this material are those of the

author(s) and do not necessarily reflect the views of the United States Department of Defense.

References herein to any specific commercial product, process, or service by trade name, trade mark,

manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation,

or favoring by Carnegie Mellon University or its Software Engineering Institute.

This report was prepared for the

SEI Administrative Agent

AFLCMC/PZM

20 Schilling Circle, Bldg 1305, 3rd floor

Hanscom AFB, MA 01731-2125

NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING

INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON

UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED,

AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR

PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE

OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY

WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK,

OR COPYRIGHT INFRINGEMENT.

[Distribution Statement A] This material has been approved for public release and unlimited distribu-

tion. Please see Copyright notice for non-US Government use and distribution.

Internal use:* Permission to reproduce this material and to prepare derivative works from this material

for internal use is granted, provided the copyright and “No Warranty” statements are included with all

reproductions and derivative works.

External use:* This material may be reproduced in its entirety, without modification, and freely distrib-

uted in written or electronic form without requesting formal permission. Permission is required for any

other external and/or commercial use. Requests for permission should be directed to the Software En-

gineering Institute at [email protected].

* These restrictions do not apply to U.S. government entities.

DM-0004062

CMU/SEI-2016-SR-025 i

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Table of Contents

Acknowledgments v

Executive Summary vi

Abstract viii

Introduction 1

1.1

Purpose 1

1.1.1

Rationale 1

1.1.2

Topics Not Included 3

Background 4

2.1

Agile Definition 4

2.2

Lean Thinking and Kanban 5

2.3

INCOSE Definition of Agility 5

Acquisition Context 6

RFP Changes 7

4.1

Section C 7

4.1.1

The Statement of Objectives (SOO) 8

4.1.2

Statement of Work (SOW) 9

4.1.3

Reviews in an Agile Environment 9

4.1.4

Key Technical Reviews 12

4.2

Section L 16

4.2.1

Subfactor 1 – Agile Development Process 16

4.2.2

Subfactor 2 – Systems Engineering Practices 17

4.2.3

Subfactor 3 – System Test and Delivery 17

4.3

Section M 17

4.4

CDRLs 18

4.5

Cost Proposal and Schedule Instructions 20

4.6

Negotiation 22

Changes in Program Execution 24

5.1

Start-Up (Kickoff) 24

5.2

Measures/Metrics 24

5.3

WBS Discussion 26

5.4

Change Management 27

5.5

Risk Assessment 27

5.6

Development Methodologies 27

5.6.1

The Waterfall Model 28

5.6.2

The Agile Model 29

5.7

Mixing Waterfall and Agile Methodologies 30

5.7.1

The Changing Landscape of Software Acquisitions—Employing Hybrid Agile

Models 31

5.7.2

DoDI 5000.02 Facilitates Hybrid Agile Software and Waterfall Hardware

Development 31

5.8

Bidder’s Library 33

Conclusion 34

CMU/SEI-2016-SR-025 ii

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Appendix A - Agile Fundamentals 35

Appendix B - Marbach Updated Principles to the Agile Manifesto 37

Appendix C - Acronyms 38

Appendix D - Glossary 41

References 43

CMU/SEI-2016-SR-025 iii

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

List of Figures

Figure 1:

Requirements Moving En Masse Through the Process [Palmquist 2013] 10

Figure 2:

Agile Building Blocks [Palmquist 2013] 10

Figure 3:

Agile Development Pattern (Example 1) 11

Figure 4:

Another View of Reviews in Agile Environment 12

Figure 5:

CDRL Delivery 19

Figure 6:

SW Development MIL-STD 881C Appendix K WBS Breakout (Traditional Waterfall)

[PARCA 2016] 26

Figure 7:

Possible Agile SW Development MIL-STD-881C WBS Breakout (Agile Capability-Based)

[PARCA 2016] 26

Figure 8:

Engineering V Model 28

Figure 9:

DoD 5000.02 Hybrid Model 32

Figure 10:

The Agile Software Development Lifecycle 36

CMU/SEI-2016-SR-025 iv

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

List of Tables

Table 1:

SOO versus SOW 7

Table 2:

Example Definition for Adjectival Ratings 18

CMU/SEI-2016-SR-025 v

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Acknowledgments

To Dan Ward for answering our call for how F.I.R.E. can support or shape technical reviews.

To the author team for sticking with the project and enduring the many meetings to produce this

product:



Mary Ann Lapham, SEI



Keith Korzec, SEI



Larri Ann Rosser, Raytheon Intelligence Information and Services



Greg Howard, The MITRE Corporation



Steven Martin, Space and Missile Systems Center



Michael Ryan, BTAS



Thomas E. Friend, Agile On Target



John H. Norton III, Raytheon Integrated Defense Systems



Peter Capell, SEI

To our independent reviewers for their insight:



Dr. Elizabeth J. Wilson, Raytheon Integrated Defense Systems (Retired)



Gari Palmer, Raytheon Integrated Defense Systems



William E. Novak, SEI

And always to Gerald Miller, our unfailingly patient editor.

CMU/SEI-2016-SR-025 vi

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Executive Summary

Agile methods have proven effective in rapidly and responsively delivering functionality in com-

mercial environments in which products are developed based on internally identified needs and

offered for sale to multiple uncommitted customers. Agile approaches are becoming more and

more attractive to the Department of Defense (DoD) and other federal agencies in light of recent

budget pressures and a widespread need to bring capabilities to users more quickly than in the

past.

To set the stage for this report, the authors turn to the Performance Assessments and Root Cause

Analyses (PARCA) Agile and Earned Value (EV) Desk Guide, which describes the situation:

Constantly evolving threats have presented a demand for an acquisition process that is able

to respond quickly to emerging requirements and rapidly changing environments. To ad-

dress this, the DoD 5000.01 has encouraged the following characteristics in acquisitions:

1. Flexibility: tailoring program strategies and oversight

2. Responsiveness: rapid integration of advanced technologies

3. Innovation: adapt practices that reduce cost and cycle time

4. Discipline: use of program baseline parameters as control objectives

5. Effective Management: decentralization to the extent practicable

These characteristics have led to an increased focus on flexible development approaches

that include Agile philosophies and integrated program management tools such as Earned

Value Management [PARCA 2016].

The TechFAR Handbook further observes

In the Government, digital services projects too often fail to meet user expectations or con-

tain unused or unusable features. Several factors contribute to these outcomes, including the

use of outdated development practices and, in some cases, overly narrow interpretations of

what is allowed by acquisition regulations… The TechFAR consists of a handbook, which

discusses relevant FAR authorities and includes practice tips, sample language, and a com-

pilation of FAR provisions that are relevant to Agile software development [OUSD 2014].

Agile methods by design comprise technical and programmatic practices employed to reduce

wasted effort, decrease bureaucratic overburden, and enhance the productivity and engagement of

developer teams. This document is intended to support the writers of RFPs as they incorporate

Agile concepts into programs early in the lifecycle, by providing examples of language that will

lay the groundwork for contracts on which programs rely.

Specific topics addressed include



RFP Section C

CMU/SEI-2016-SR-025 vii

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see



RFP Section L



RFP Section M



RFP contract data requirements lists (CDRLs)



RFP Cost Proposal and Schedule Instructions



Negotiation



Start-Up



Changes in Program Execution



Hybrid Approaches



Bidder’s Library

CMU/SEI-2016-SR-025 viii

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Abstract

Increasing budget constraints and emphasis on fielding capability faster have led the U.S. Depart-

ment of Defense (DoD) and other federal entities to pursue the benefits of Agile software devel-

opment—reduced cycle times, flexibility to adapt to changing conditions and user needs— that

software development practitioners have achieved in the commercial market.

This report is written by the National Defense Industrial Association’s System Engineering Agile

Working Group to provide information on request-for-proposal (RFP) patterns and techniques for

successful Agile contracting that can and have been used for contracts seeking to employ Agile

methods. This report is intended to support the writers of RFPs in bringing Agile concepts into

programs at the earliest possible time, providing examples of the kinds of language that will affect

the foundations of contracts on which programs rely.

CMU/SEI-2016-SR-025 1

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

1 Introduction

This report is authored by the National Defense Industrial Association’s (NDIA) System Engi-

neering Agile Working Group in response to multiple inquiries about what language should be

used in a request for proposal (RFP) to enable the use of an Agile approach. The report addresses

the following topics:

Section 1 provides the introduction, purpose, and topics not included in this discussion.

Section 2 provides the background including an Agile definition, Lean thinking and Kanban,

and the International Council on Systems Engineering (INCOSE) definition of

Agility.

Section 3 discusses the overall acquisition context for employing Agile methods.

Section 4 discusses overall RFP changes including Section C (statement of objectives [SOO]

or statement of work [SOW], reviews in an Agile environment, key technical

reviews); Section L; Section M; contract data requirements lists (CDRLs); cost

proposal and schedule instructions; negotiation; and start-up.

Section 5 discusses potential changes in program execution that will be encountered when

employing Agile concepts including measures and metrics, work breakdown

schedules (WBS), change management, risk assessment, and hybrid approaches.

1.1 Purpose

This report provides guidance on the language of government-contract RFPs to assist RFP authors

in supporting agility in government systems acquisitions. “Agile methods” comprise those tech-

nical and programmatic practices that are employed to conscientiously be responsive to change,

improve quality and fit for purpose, reduce wasted effort, decrease bureaucratic overburden on

programs, and enhance the productivity and engagement of developer teams. This report is in-

tended to support the writers of RFPs in incorporating Agile concepts into programs early in the

lifecycle through various elements of the RFP process.

1.1.1 Rationale

The government is becoming increasingly aware that the traditional methods of RFP and contract

development that are typically employed are not well suited to provide a quickly fielded capabil-

ity based on an iterative or Agile development approach. Routinely, an RFP is formulated around

a known solution with a majority of the requirements already identified. As in the past, the devel-

opers and acquirers follow the processes laid out in directed “best practices” such as the DoD

5000 series. Also like so many programs before, the results are often less than expected, cost

much more, and are fielded years later than needed [GAO 2008].

Thus, a modified approach for the acquisition process is needed. Instead of contracting for an

“end-to-end” system, segments of the government recently began following the lead of the private

sector by acquiring capabilities that build on one another to eventually create the complete system

CMU/SEI-2016-SR-025 2

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

incrementally. The challenges with this approach are in defining the scope (SOO and SOW re-

quirements), assessing the offer (evaluation criteria), evaluating progress (milestones and earned

value [EV]), and confirming completion (sell-off criteria).

Confusion often occurs when the program office tries to interpret iterative activities relative to the

traditional DoD acquisition lifecycle framework. There seems to be a common misunderstanding

that the development methodology must mirror the acquisition lifecycle [Lapham 2014]. Prob-

lems occur when trying to overlay “traditional” acquisition milestone events directly atop devel-

opment methodologies that use iterative work units, without understanding the relationship among

the work units and the milestone events. This can be even more complex if Agile software meth-

odologies are being used to develop a system that includes significant specialized hardware (e.g.,

acquisition of a satellite, ship, plane, or the like.). Many of these are issues that can be addressed

by carefully writing an RFP that enables the government to take full advantage of the benefits that

an Agile acquisition offers [Lapham 2014].

Agile or iterative system developments are based on collaboration between the contractor and cus-

tomer. The acquirers are a part of the team rather than just a compliance organization. Without di-

rect involvement of the system’s intended users or knowledgeable surrogates (program manage-

ment office [PMO] members or appointed contractor resources), early validation of the direction

of the implementation is more difficult than necessary to achieve. This early validation is essential

to ensure that iterative development proceeds in a stable manner, based on layer after layer of

evolving, useful functionality. Some programs may refer to these evolving layers as “threads,”

“releases,” “builds,” “iterations,” “spirals,” “slices,” and other similar terms [Lapham 2014].

There isn’t only one approach to specifying contractual terms to incentivize proposal of pro-

ductive Agile approaches. Because there isn’t a single definition of “Agile,” saying “we

want a contractor to use Agile methods” isn’t useful. Being clear about the relationship ex-

pected without inappropriately constraining the “how” things get done is the balance to

seek.

We have seen every DoD contract type used at one time or another for programs using Ag-

ile. Conditions under which one contract type or another works well are still anecdotal.

Firm Fixed Price seems to be the most difficult to use to build the collaborative relationships

that are the hallmark of effective Agile programs [Foreman 2014].

Like the system under development, the RFP must address technical evaluations that are incre-

mental. Similarly, plans, documentation, and other CDRLs are also developed incrementally over

the entire life of the program. While different from the traditional acquisition world, this method

has the advantage of providing documentation of the system “as built” rather than “as designed.”

This approach allows the government to take advantage of technical advances, changing opera-

tional environments, and/or requirements, as detailed design decisions are typically made closer to

their implementation versus all up front. Furthermore, using as-built documentation removes any

additional burden to the maintenance and logistics community that may come from using as-de-

signed documentation.

Since the system is intended to evolve over the duration of the contract, the RFP must include the

definition of success, especially if it used for incentive fees or progress payments to the developer.

CMU/SEI-2016-SR-025 3

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

As a principle, Agile welcomes changing requirements, even late in development. Thus, a clear

plan on how to deal with the inevitable changes should be included in responses to an RFP to

avoid the need to renegotiate the contract when issues arise. Likewise, it is advisable to address

and document the government process to de-scope lower priority features and re-prioritize re-

maining features in response to new or changed requirements.

Finally, the RFP must include mechanisms to give acquirers real-time insight into the capabilities

to be delivered, metrics, plans, specifications, schedules, and the like.

1.1.2 Topics Not Included

The authors have deliberately avoided including contractual language that could be “cut and

pasted” into an RFP and convey a preference for one Agile methodology over any other.

In addition, discussion of specific program work plans or tasks, Rapid Acquisition, and hardware-

only buys are not included.

CMU/SEI-2016-SR-025 4

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

2 Background

Traditional and Agile development both have a place in modern systems acquisition. Some gen-

eral characteristics of programs more suited to each development approach are provided below.

Keep in mind that these are generalities and there may be instances where a particular approach

can be used successfully on a program that does not strictly conform to the characteristics listed

for that approach. (For example, an Agile development approach could provide significant bene-

fits to a program with stable requirements.) More detail on Agile development concepts is pro-

vided in Appendix A.

Traditional Approach



programs with stable requirements and mission environment, with known solutions to the re-

quirements



programs with a homogeneous set of stakeholders who communicate well via documents



programs for which the technology base is evolving slowly (technology is not expected to be

refreshed/replaced within the timeframe of the initial development)

Agile Approach



programs with evolving requirements and environment



programs where solutions are sufficiently unknown that significant experimentation is likely

to be needed



programs for which the technology base is evolving rapidly



programs with stakeholders who can engage with developers in ongoing, close collaboration

[Nidiffer 2014]

So what exactly do we mean when we say “Agile”? For purposes of this report, the authors are

not limiting the discussion to just software, although that’s where the Agile Manifesto originated.

Many of these Agile practices started with Agile and Lean manufacturing [Dove 1993]. INCOSE

has also developed a definition of what it means by agility. One last piece that needs to be defined

is the use of Lean in this Agile methodology and implementation. All three of these ideas are de-

fined in the following paragraphs.

These ideas are explored throughout the paper and can be applied across the system lifecycle. In

addition, many of the concepts discussed in this report can also be employed where the govern-

ment understands the end state but the contractor wishes to use an Agile development methodol-

ogy.

2.1 Agile Definition

“Agile is an iterative and incremental (evolutionary) approach to ... development which is per-

formed in a highly collaborative manner by self-organizing teams within an effective governance

framework with ‘just enough’ ceremony that produces high quality software in a cost effective

and timely manner which meets the changing needs of its stakeholders” [Ambler 2013].

CMU/SEI-2016-SR-025 5

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

The concept was formalized in 2001 with the publication of the Manifesto for Agile Software De-

velopment as shown in Appendix A.

2.2 Lean Thinking and Kanban

Lean thinking is a way of thinking about any activity and seeing the waste inadvertently generated

by the way the process is organized by focusing on the concepts of

1. value

2. value streams

3. flow

4. pull

5. perfection

Kanban is a method for managing knowledge work with an emphasis on just-in-time delivery

while not overloading the team members. This approach presents all participants with a full view

of the process from task definition to delivery to a customer. Team members pull work from a

queue.

Lean thinking and Kanban techniques are often incorporated into programs using iterative and in-

cremental approaches.

2.3 INCOSE Definition of Agility

Today we operate in unpredictable, uncertain, risky, variable, and evolving (UURVE) environ-

ments. Responding to this type of environment requires agility. This agility can be achieved by

implementing Agile practices (e.g., Agile contracting, which buys services and/or capabilities in-

stead of an overly specified product). With this environment in mind, INCOSE defines agility in

the following manner:

This working group views agility as a sustainable system and process capability under

UURVE conditions, enabled and constrained fundamentally by system architecture. This ar-

chitecture delivers agile capability as reconfiguration, augmentation, and evolution of sys-

tem and process functionality, during operation throughout the lifecycle. The architecture

enables the system or process to respond to new and immediate situational requirements ef-

fectively. Effectiveness of response is measured in response time, response cost, response

quality, and response scope sufficient to sustain functional intent [INCOSE 2016].

Achieving agility for systems in today’s environment requires the use of philosophies and meth-

ods such as those seen in Agile, Lean, and Kanban.

See https://en.wikipedia.org/wiki/Lean_thinking

See https://en.wikipedia.org/wiki/Kanban_(development)

CMU/SEI-2016-SR-025 6

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

3 Acquisition Context

Agile methods have proven effective in rapidly and responsively delivering functionality in com-

mercial environments where products are developed based on internally identified needs and of-

fered for sale to multiple uncommitted customers. While the federal acquisition model is different,

recent budget pressures and environmental changes make the benefits of such methods attractive

to the DoD and other federal entities. The Performance Assessments and Root Cause Analyses

(PARCA) Agile and Earned Value (EV) Desk Guide describes the situation:

Per the DoD Defense Acquisition System Directive 5000.01, an acquisition system is “a di-

rected, funded effort that provides a new, improved, or continuing materiel, weapon or infor-

mation system, or service capability in response to an approved need… [t]he primary objec-

tive…is to acquire quality products that satisfy user needs with measurable improvements to

mission capability and operational support, in a timely manner, and at a fair and reasonable

price.” Constantly evolving threats have presented a demand for an acquisition process that

is able to respond quickly to emerging requirements and rapidly changing environments. To

address this, the DoD has encouraged the following characteristics in acquisitions:

1. Flexibility: tailoring program strategies and oversight

2. Responsiveness: rapid integration of advanced technologies

3. Innovation: adapt practices that reduce cost and cycle time

4. Discipline: use of program baseline parameters as control objectives

5. Effective Management: decentralization to the extent practicable [PARCA 2016]

These characteristics have led to an increased focus on flexible development approaches

that include Agile philosophies and integrated program management tools such as Earned

Value Management [PARCA 2016].

Inevitably, the attempt to merge the practices of the federal acquisition community and the com-

mercial Agile community has resulted in questions and challenges, beginning with the acquisition

process. The TechFAR Handbook observes

In the Government, digital services projects too often fail to meet user expectations or con-

tain unused or unusable features. Several factors contribute to these outcomes, including the

use of outdated development practices and, in some cases, overly narrow interpretations of

what is allowed by acquisition regulations… The TechFAR consists of a handbook, which

discusses relevant FAR authorities and includes practice tips, sample language, and a com-

pilation of FAR provisions that are relevant to Agile software development [OUSD 2014].

The rest of this report continues the exploration of applications of Agile methods and processes to

federal acquisitions specifically addressing some of the issues that may arise when creating an

RFP that enables Agile methods.

CMU/SEI-2016-SR-025 7

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

4 RFP Changes

Traditional RFPs are generally based on the government expecting a traditional waterfall-struc-

tured

program approach for creating the system in question. The traditional government procure-

ment process and associated guidance does not necessarily facilitate employing Agile methods as

part of the solution. As discussed in Section 3 of DoDD 5000.01 [DoD 2007], the PARCA Agile

and Earned Value Management guide [PARCA 2016] and the TechFAR Handbook [OUSD 2014]

address this issue by providing some alternative guidance. In addition, DoDI 5000.02 provides

some alternative models to address iterative and incremental development. This section explores

options that would either directly request Agile responses as part of the solution or at the very

least allow for Agile responses even if the government is still expecting a traditional program

structure.

4.1 Section C

Section C provides the government’s requirements and expectations of the contractor's perfor-

mance in the form of a SOO or SOW as part of the RFP. The SOO reflects a performance-based

acquisition (PBA) and is best suited for an Agile acquisition.

If a SOO is provided, the govern-

ment will normally expect the contractor to provide a SOW or a performance work statement

(PWS) as part of its proposal. The government-provided SOW is best suited for a traditional ac-

quisition in which the government has a high degree of confidence in the ability to specify (both

qualitatively and quantitatively) the expected approach and product end state. Table 1 highlights

the differences.

Table 1: SOO versus SOW

SOO Factor SOW

The government understands the

objectives but expects the end

state to evolve.

Government

Understanding

The government has a high level of confidence

in the end state.

Change is expected to be a

significant factor in achieving the

end state.

Change

Change is not anticipated, or if encountered will

not be disruptive.

This approach provides the offerors

trade space and flexibility in

developing their proposals. It

should probably be used unless the

totality of the work effort required is

very well understood by the

government.

Constraint

Constrains offerors to the specific tasks

identified, so it must be unambiguous and

comprehensive. The government needs to

apply specific constraints on the tradeoff space

of lifecycle cost, performance, interoperability,

logistics/training, etc.

The authors assume some familiarity with sequential, waterfall-based software development models. A brief

refresher is provided in Section 5.6.1.

See AcqNotes, Acquisition Process, Performance-Based Acquisition, at

http://acqnotes.com/acqnote/acquisitions/performance-based-acquisitions

CMU/SEI-2016-SR-025 8

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

This approach conforms to PBA

practices.

Performance-Based

Acquisitions (PBA)

4.1.1 The Statement of Objectives (SOO)

The SOO is used in PBA “which structures the acquisition around the purpose and outcome de-

sired as opposed to the process by which the work is to be performed.”

With performance-based

acquisition, the government no longer develops a prescriptive SOW directing how the contractor

will achieve project milestones. Instead, the government develops a SOO or PWS that describes

the overall outputs and objectives but does not specify how to achieve those outputs.

The SOO is “incorporated into the solicitation that states the overall performance objectives. It is

used in solicitations when the Government intends to provide the maximum flexibility to each of-

feror to propose an innovative approach.”

It focuses on the government’s objectives and identi-

fies the broad, basic, top-level objectives of an acquisition/procurement. The SOO is developed

after performing a risk assessment that highlights the high and moderate risks for business, pro-

grammatic, and technical areas against the requirement document. It is best applied when the gov-

ernment can describe the objectives but is not ready to specify the end state and expects change as

a factor in achieving the end state. It provides offerors the flexibility to develop cost-effective so-

lutions and propose innovative alternatives to meet the objectives.

A SOO supplements the performance-based requirements document. In a DoD acquisition, a SOO

aligns with an initial capabilities document (ICD); programmatic direction from the acquisition

decision memorandum (ADM), acquisition strategy, and acquisition plan; technical requirements

from system specifications; and the draft WBS and dictionary.

The SOO provides enough information and detail for the offeror to structure a sound program, de-

signed to be executable and satisfy government objectives. The offeror uses the SOO to develop

the contractor statement of work (CSOW), the contract(or) work breakdown structure (CWBS),

the integrated master schedule (IMS), and other documents supporting and defining the proposed

effort.

The contractor will sell off the government objectives by meeting the predefined and negotiated

definition of done

when using Agile methods. Once the definition of done is met, the government

objectives can be said to have been met.

Derived from AcqNotes, Proposal Development, Statement of Objectives, at

http://acqnotes.com/acqnote/tasks/statement-of-objectives

See AcqNotes, Acquisition Process, Performance-Based Acquisition, at

http://acqnotes.com/acqnote/acquisitions/performance-based-acquisitions

See http://www.acq.osd.mil/dpap/ccap/cc/jcchb/HTML/Topical/sow.html

Done is also referred to as “Done” or “Done Done.” This term is used to describe a product increment that is

considered releasable—it means that all design, coding, testing, and documentation have been completed and

the increment is fully integrated into the system.

CMU/SEI-2016-SR-025 9

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

4.1.2 Statement of Work (SOW)

The SOW describes the government’s needs for the work to be done in developing or producing

the goods or services. The SOW is “a portion of a contract which establishes and defines all non-

specification requirements for contractor's efforts either directly or with the use of specific cited

documents.”

It also facilitates the preparation of a proposal and aids the government in conduct

of the source selection and contract administration after award.

The preparation of an effective SOW requires both an understanding of the goods or services that

are needed to satisfy a particular requirement and an ability to define what is required in specific,

performance-based, quantitative terms. This requires the government to have a high level of confi-

dence that it understands the end state and doesn’t expect change to be disruptive. The SOW

should reference qualitative and quantitative design and performance requirements contained in

specifications such as those developed according to MIL-STD-961.

4.1.3 Reviews in an Agile Environment

In a traditional waterfall model, technical reviews are used as control gates to move from one se-

quential phase to the next (e.g., concept > requirements > design > develop > test and integration

> deployment). Figure 1 shows the traditional progression of requirements through the waterfall

model [Palmquist 2013]. A traditional program tends to be risk averse and uses reviews for risk

mitigation.

In an Agile program, the focus is on completing each work unit from analysis to deployment in-

stead of completing each phase in sequential order. For example, as shown in Figure 2, the Agile

world uses the same building blocks but looks at them differently than the traditional world

[Palmquist 2013]. An Agile program tends to use technical reviews as opportunities for infor-

mation sharing, face-to-face coordination, and confidence building.

For this reason, Agile programs may treat some reviews (most frequently design and test readi-

ness) incrementally as part of the design-develop-integrate-test (DDIT) cadence, which allows for

a faster start (from contract start to first incremental design review) and more opportunities to ad-

dress changes in technology and the mission environment. In general, the larger part of the pro-

gram lifecycle that is Agile, the greater the benefit, but Agile methods can provide value even

when applied only to software development. The decision about which reviews are conducted se-

quentially and which incrementally should be based on program needs and challenges. Keep in

mind there is no one correct way to implement Agile concepts within a program. Having provided

this caution, we are providing in the following paragraphs two examples of how a program may

implement Agile concepts within the milestones typically seen in a government program.

Derived from AcqNotes Proposal Development, Statement of Work (http://acqnotes.com/acqnote/tasks/state-

ment-of-work)

See http://www.acq.osd.mil/dpap/ccap/cc/jcchb/HTML/Topical/sow.html

CMU/SEI-2016-SR-025 10

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Figure 1: Requirements Moving En Masse Through the Process [Palmquist 2013]

Figure 2: Agile Building Blocks [Palmquist 2013]

A generic Agile program development pattern is shown in Figure 3. In this example, the pattern is

based on a structure where the period of performance contains a series of time-boxed releases (1-

n). Each release (as shown in the expanded portion) begins with release planning, followed by a

series of DDIT iterations (e.g., sprints 1-n) and concludes with a system demo followed by a de-

ployment decision. Notice that a design review is shown at release planning and a deployment de-

cision is shown at the end of the release. These reviews can be used as tailored alternatives to sat-

isfy the preliminary design review (PDR) and critical design review (CDR).

Requirements

Document

Requirement #1

Requirement #2

Requirement #3

Requirement #4

Requirement #5

Anal

Design Build

Test Deploy

Agile Process

Analyze

Design

Build

Test

Deploy

High-Priority

Requirement

Next High Priority

Requirement

Next High Priority

Requirement

Next High Priority

Requirement

Next High Priority

Requirement

CMU/SEI-2016-SR-025 11

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Figure 3: Agile Development Pattern (Example 1)

Note that the graphic implies deployment at the end of a release. A more sophisticated Agile

model separates deployment from the release time box. In this model, software is still developed

on cadence (the release time box) but released on demand. Rather than push new software to the

system, new software is always available for the system to pull at any time.

Our second example, Figure 4, depicts the contrast between typical milestones and reviews in a

traditional waterfall environment versus incremental reviews in an Agile model. In this model, the

traditional System Requirements Review (SRR), PDR, and CDR events are replaced by incremen-

tal design reviews and, if needed, system-level reviews. The system-level reviews are smaller in

scope than the corresponding events in the traditional waterfall programs, focusing on the system

architecture, hardware integration, system-level risks, and other system-level concerns.

Graphic provided by the MITRE Corporation (https://www.mitre.org/publications/technical-papers/defense-agile-

acquisition-guide-tailoring-dod-it-acquisition-program).

CMU/SEI-2016-SR-025 12

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Figure 4: Another View of Reviews in Agile Environment

4.1.4 Key Technical Reviews

Technical reviews should reflect the Agile principles, the Agile principles as defined in the Mar-

bach paper presented at INCOSE 2015 [Marbach 2015], guidance from F.I.R.E. by Dan Ward

[Ward 2014], or some combination thereof, depending on the environment and needs of the indi-

vidual program.

Two Agile principles—“Business personnel, customers or their advocates, and implementers must

work together daily” and “Continuous attention to technical excellence and good design enhances

agility”—are the most applicable in this area. Marbach also lists nearly identical counterparts:

“Business personnel, customers or their advocates, and implementer must work together daily

throughout the project” and “Continuous attention to technical excellence and good design en-

hances agility” [Marbach 2015]. (See Appendix B for a comparison of the Agile principles and

the Marbach principles.) To summarize F.I.R.E.: reviews must add value to the program—not just

be held because status quo demands them; keep them to a minimum while keeping them small,

quick and agile to minimize potential changes in the technical and operational environments. In

addition, apply the heuristic of “Minimize the distance between decision and action” [Ward

2014].

Key technical reviews are listed below.

Reproduced and modified with permission from the Raytheon Company.

CMU/SEI-2016-SR-025 13

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

System Requirements Review (SRR)

When Conducted. At the beginning of the period of performance and any time at which there is a

major change in the capabilities baseline.

Purpose. The System Requirements Review (SRR) is a multi-disciplined technical review to en-

sure that the developer is ready to proceed with the initial system design.

The review will in-

clude the contractor’s understanding of the program’s objectives, capabilities and non-functional

requirements baseline, constraints such as architecture, time critical deliveries, and milestones.

Input. Capabilities baseline, program vision, and roadmap.

Output. Initial program backlog.

Preliminary Design Review (PDR)

When Conducted. Post SRR/system functional review (SFR) (in traditional/waterfall); scheduled

when appropriate to verify architectural consistency between the hardware and software, technical

performance measures (TPMs), and the like (in an Agile development program, only if needed).

Purpose. In a traditional waterfall program, PDR demonstrates that the preliminary design meets

all system requirements with acceptable risk and within the cost and schedule constraints. It estab-

lishes the basis for proceeding with detailed design.

In Agile, the large scale and formality of a

PDR is replaced with more frequent, less formal interactions between the customer and the con-

tractor. Examples include release planning and release demos. As such, a traditional PDR is not

recommended on purely Agile programs. In some hybrid programs, a PDR may be desirable. For

example, in a program using Agile software development and a waterfall approach to hardware

development, a system-level PDR might be used to assess overall program progress. In that case,

the focus of the PDR event should be verifying architectural consistency between the hardware

and software, TPMs, and hardware components. See Figure 4 for an example of how PDR and

CDR can be adapted to an Agile environment.

Input. Artifacts demonstrating that the preliminary design meets all system requirements with ac-

ceptable risk within cost and schedule constraints.

Output. Decision to proceed with detailed development (in traditional/waterfall); insight into cur-

rent program status regarding

architectural consistency between the hardware and software,

TPM maturity, etc. (in an Agile development program).

See https://acc.dau.mil/CommunityBrowser.aspx?id=638317

See https://en.wikipedia.org/wiki/Design_review_%28U.S._government%29

See https://en.wikipedia.org/wiki/Design_review_%28U.S._government%29#cite_note-11

CMU/SEI-2016-SR-025 14

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Critical Design Review (CDR)

When Conducted. Post PDR (in traditional/waterfall); scheduled when appropriate to verify archi-

tectural consistency between the hardware and software, TPMs, etc. (in an Agile development

program, only if needed).

Purpose. In a traditional waterfall program, CDR demonstrates that the maturity of the design is

appropriate to support proceeding with full-scale fabrication, assembly, integration, and test. CDR

determines that the technical effort is on track to complete the flight and ground system develop-

ment and mission operations, meeting mission performance requirements within the identified

cost and schedule constraints.

As is the case with PDR, a traditional CDR is not recommended

on purely Agile programs for the same reason; however, again, on some hybrid programs, a CDR

may be desirable and would be focused on the same type of topics as an “Agile PDR,” but the ex-

pected level of maturity (of TPMs for example) would be greater.

Input. Artifacts demonstrating that the maturity of the design is appropriate to support proceeding

with full-scale fabrication, assembly, integration, and test, within cost and schedule constraints.

Output. Decision to proceed with fabrication, assembly, integration, and test (in traditional/water-

fall); insight into current program status regarding

architectural consistency between the hard-

ware and software, TPM maturity, etc. (in an Agile development program)

Release Planning and Review

When Conducted. At the beginning of the release (nominally a week in duration).

Purpose. Demonstrates confidence (sufficient understanding of the requirements and design ap-

proach) for moving into the DDIT activities shown in Figure 3 that produce the release code base-

line. The release planning combines elements of a traditional PDR and CDR, with enough confi-

dence in maturity to move into the next increment; the remaining level of maturity is achieved

during the DDIT phase. Release planning meeting and design reviews are iterative—the first re-

lease is typically different from subsequent ones in that there is a greater focus (sometimes an ex-

clusive focus) on architectural concepts and enabling elements. Typically, the first release also

does not involve adjudicating high-priority defects against features, although it may require effort

to refactor inherited system artifacts from legacy programs. The design items presented at release

planning pertain to the work (capabilities and architecture) allocated to that release. When an item

(e.g., architecture) from a previous release changes in the current release, the changes would be

presented as an update (i.e., “updates as needed”). The program backlog drives release planning

and determines the priorities of the system component requirements allocated to the release. Pro-

gram and business stakeholders establish release priorities and objectives. Development teams de-

rive stories from the requirements and commit to their team objectives, which are rolled up to pro-

gram objectives.

See https://en.wikipedia.org/wiki/Design_review_%28U.S._government%29

See https://en.wikipedia.org/wiki/Design_review_%28U.S._government%29#cite_note-11

CMU/SEI-2016-SR-025 15

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Input. Release backlog (allocated from the program backlog).

Output. Finalized release backlog and initial team backlogs. Confidence-based decision to move

into DDIT iterations jointly made between government and contractor.

Test Readiness Review (TRR)

When Conducted. Prior to system demo.

Purpose. The TRR is conducted to determine if the system under review is ready to proceed into

formal testing by deciding whether the test procedures are complete and verify their compliance

with test plans and descriptions.

In an Agile program, the preference is for the test organization

to be continuously involved beginning at release planning, where the test strategy is discussed, the

backlog is prioritized to align with test events, and the like. The TRR may be needed on a pro-

gram where the test organization has not been closely involved with the iteration demos or if sig-

nificant hardware is involved and/or the system is being tested at a government site. The TRR will

review the results of the tests conducted during the DDIT iterations and identify workarounds,

shortfalls, constraints and level of confidence for conducting a system demo.

Input. Iteration test results, system demo test plan.

Output. Decision to conduct system demo.

4.1.4.1 Release Demo and Deployment Review

When Conducted. At the conclusion of the DDIT iterations.

Purpose. The release demo and deployment review is a technical review of the release require-

ments and the as-built system design. It determines, based on the results of the release demo, if

the release objectives and requirements have been met and the system performs as expected. If

necessary, this may include a deployment review to determine the user’s and the system’s readi-

ness to accept the new baseline.

Input. Results of the system demo.

Output. Decision on the adequacy of the as-built baseline and readiness for deployment.

4.1.4.2 Other Regulatory Review Information

IEEE Standard 15288.2-2014, IEEE Standard for Technical Reviews and Audits on Defense Pro-

grams, lists the following reviews: alternative systems review (ASR), system requirements review

(SRR), SFR, preliminary design review (PDR), CDR, test readiness review (TRR) (contained

within the program’s test and evaluation master plan), functional configuration audit (FCA), sys-

tem verification review (SVR), production readiness review (PRR), and physical configuration

See http://acqnotes.com/acqnote/acquisitions/test-readiness-review

CMU/SEI-2016-SR-025 16

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

audit (PCA) [IEEE 2016]. Note that IEEE Standard 15288-2 has not been updated for Agile meth-

ods, and lists traditional plan-driven reviews such as PDR and CDR, which are tailored into the

Agile SDLC.

Other technical reviews (e.g., site acceptance review, operational readiness review, transition

readiness review, etc.) may be employed according to the needs of the program. However, if addi-

tional reviews are employed, they should be tailored to be as streamlined as possible. Remember

the Agile principle that “simplicity—the art of maximizing the amount of work not done”—is es-

sential, along with the clarification recommended by Marbach that

“

a truly Agile development

project does not force artificial reporting and process requirements on the Implementation

Team” [Marbach 2015].

4.2 Section L

In an RFP, Section L contains the instructions for providing the proposal to the government.

These instructions can be long and complicated. But the key elements of the proposal are an-

swered in the cost and technical factors. Technical factors will be discussed in this section but cost

considerations will be deferred to Section 4.5. An RFP can have one or more technical factors de-

pending on the scope and complexity of the system to be developed. For this discussion, only one

technical factor is used for system development, but the factor has three subfactors. During the

evaluation of the proposals, the rating for each subfactor is rolled up into an overall rating for the

technical factor. The method in which the subfactors are evaluated is explained in Section M of

the RFP. The technical subfactors are Agile development process, systems engineering practices,

and system test and delivery.

1. The first subfactor is essential in describing how the entire system lifecycle will be managed

and implemented using Agile concepts and methods.

2. The second subfactor requests the detailed description of the Agile systems engineering and

other discipline-specific practices to be used during the development. It would also include

the schedule of work and key events that will enable the integration of the total solution.

These engineering processes should reflect and be compatible with Agile concepts discussed

in the other subfactors since the intent is to enable a holistic Agile development approach.

3. The final subfactor is used to evaluate how the completed system will be delivered to the

user, which would include several steps such as acceptance, certification, installation, train-

ing, and transition from a previous system. Again, the engineering processes in this subfactor

should reflect and be compatible with Agile concepts discussed in the other subfactors.

In all cases, how the subfactors work together (since this is a request for an Agile proposal)

should be included in the evaluation criteria. The following sections contain example wording of

the subfactors to be used in the RFP.

4.2.1 Subfactor 1 – Agile Development Process

Describe the Agile development lifecycle process to design and build “system name” based on the

Agile development methodology. When describing the process, explain how the process imple-

ments the Agile principles listed in Appendix B from Principles for Agile Development [Marbach

2015], which are adapted for DoD and federal systems from the principles listed on the Agile

CMU/SEI-2016-SR-025 17

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Manifesto website (http://www.agilemanifesto.org). The description should include specific de-

tails on the management of requirements decomposition and the product backlog (prioritization of

work with the government), the orchestration of the demonstrations (collaboration with the users

on product evaluations during development), and the definition of “done” (meeting the govern-

ment’s product delivery requirements defined in the CDRLs). Include shorter but more frequent

integrated client/developer meetings, user involvement, and other items specific to the Agile ap-

proach.

4.2.2 Subfactor 2 – Systems Engineering Practices

Describe systems engineering and any other specific engineering processes required to develop

the architecture for “system name” not included in Subfactor 1. In addition, a system capabilities

and features roadmap should be provided that maps capabilities and features to the development

iterations and system build deliveries. This subfactor should also include further details about

meeting the entrance, exit, and review criteria for the technical reviews as stated in IEEE 15288

Volumes 1 and 2 and IEEE 12207. (More details on the Technical Reviews are in Section 4.1.4.)

4.2.3 Subfactor 3 – System Test and Delivery

Describe the testing planned prior to delivering the software to the government. While some of

this information should be included in the Agile development process subfactor, specific details

for automated test, functional, integration, vulnerability, and defect tracking should be added here.

Also, describe how support will be provided during government development test and evaluation

(DT&E) and operational testing and evaluation (OT&E). This should include test environment,

test data/connections to external systems/stub to simulate outside systems, test scripts, support,

training, and response to issues (e.g., deficiency reports [DRs]).

Describe the process for delivering the software and its associated CDRLs to the government for

fielding the system including documentation, distribution, tracking, transition of ops, training, cer-

tifications, build process, archiving, and recovery.

4.3 Section M

Section M of the RFP explains how the decision for the award will be made. As with traditional

proposals, the evaluation of Agile proposals is based on the following factors:



technical



past performance



cost/price



cost/price risk

The relative importance of the above factors must be addressed in accordance with FAR section

15.304(e). Within the technical factor, the subfactors are the key discriminators to support a

meaningful evaluation of competing proposals. Since these subfactors were all objective, they

would receive an adjectival rating such as outstanding, satisfactory, marginal, and unsatisfactory.

If a subfactor is found to be unsatisfactory, it must be rated unsatisfactory [Navy 1998]. Table 2

provides an example of definitions for the adjective ratings.

CMU/SEI-2016-SR-025 18

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Table 2: Example Definition for Adjectival Ratings

Adjectival Rating Definition

Outstanding Proposal significantly exceeds requirements in a way that benefits the gov-

ernment or meets requirements and contains at least one exceptional en-

hancing feature that benefits the government. Any weakness is minor.

Satisfactory Proposal meets requirements. Any weaknesses are acceptable to the gov-

ernment.

Marginal Proposal contains weaknesses or minor deficiencies that could have some

impact if accepted.

Unsatisfactory Proposal does not comply substantially with requirements.

Based on the example subfactors used in the previous section, the following paragraphs are exam-

ples of how the subfactors could be evaluated:

Subfactor 1: Agile development process subfactor will primarily be evaluated by how the pro-

posed system development process will adhere to each of the principles of Agile development and

the government’s Agile development approach. Additionally the proposal will be evaluated on

how the contractor will partner with the customer to manage the product backlog with the pro-

gram office; provide software demonstrations to the program office and the user; and if the arti-

facts to be provided at the end of each sprint meet the program office’s definition of done.

Subfactor 2: Systems engineering practices subfactor will be evaluated by how the proposed sys-

tems engineering process will be performed as part of a holistic Agile development approach

while meeting the government expectations for technical reviews. Also, this evaluation will in-

clude the review of the capabilities and features roadmap and its mapping to the iteration sched-

ule, the planned releases for integration and test, and planned schedule of activities to complete

the delivery of the system.

Subfactor 3: System test and delivery subfactor will be evaluated by activities, test events, and de-

livery of artifacts required to perform government developmental and operational testing and

fielding of the system to the user. In addition, details about the test environment and supporting

test structure such as test scripts, test data, and simulated connections to external systems will be

evaluated for meeting government test and security requirements. Also, this subfactor will evalu-

ate the use of automated tools used for testing throughout the development process and automated

tools for installing and sustaining the software.

4.4 CDRLs

The number and type of CDRLs will depend on the program environment. The same approach of

mindful tailoring as cited in DoDI 5000.02 should be applied to the CDRLs the program needs.

Due to the anticipated close and continuous coordination between the government and the con-

tractor, the number of formal deliveries of the CDRLs may be reduced (e.g., real-time information

sharing between the contractor and the government via an integrated data environment). However,

CMU/SEI-2016-SR-025 19

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

all items that are needed for the operation of the system, including the software, user manuals, and

technical data for sustainment activities, must be on the CDRL.

For development and design documentation such as the system requirements specification and

software design description, additional care needs to be taken to ensure the due dates reflect the

incremental development and delivery of the products. The design documentation will reflect the

“as-built” product not the “as-designed” product. Keep in mind that the contractors will most

likely employ rapid documentation techniques such as auto-generated documents and wikis. Ac-

cess to any joint integrated data environment should be provided to the government so that it has

visibility into the technical information to meet the delivery requirement. The use of these con-

structs should be explained in their overall Agile development management plan. Formal deliver-

ies should be aligned with the program cadence.

As shown in Figure 5, deliveries on an Agile program occur more frequently at smaller events

than in a traditional waterfall program. As such, much of the documentation will be delivered and

reviewed while it is in-process with the maturity of the CDRL increasing over time. This can re-

duce risk earlier than in a traditional waterfall approach as the government can see early versions

of the system and its documentation, then watch as risks are burned down during each subsequent

iteration.

Figure 5: CDRL Delivery

CMU/SEI-2016-SR-025 20

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

4.5 Cost Proposal and Schedule Instructions

Cost

In a traditional waterfall model, the scope is fixed with cost with schedule and/or quality being

variable. In a “customary” Agile program, cost, schedule, and quality are fixed with the scope (ca-

pabilities and features) being prioritized. Agile is flexible, however, and allows for several ap-

proaches to be used in the RFP and the cost proposal in particular. Specifically, the RFP must de-

fine the deliverable scope and, additionally, what is fixed and what is variable. The structure and

approach of the contract should be based on the driving needs of the particular government entity

responsible for the contract.

Defining Deliverable Scope—What’s Fixed, What’s Variable?

The first option, referred to as “customary” above, is to give a contractor a fixed schedule and

budget while enabling the content to be variable. This is achieved through prioritization of the

backlog to ensure that if there are capabilities or features that are not completed within the allo-

cated time and cost, they are of lower priority. In this case the government is essentially buying

“development capacity” for some specified period or multiple periods. This approach is similar to

a “time and materials” contractual approach. A variant of this is discussed in the TechFAR Hand-

book, which recommends an initial period (on the order of six months) using the time and materi-

als approach to establish a velocity for the Agile team and then moving the contract to firm-fixed-

price (FFP) periods [OUSD 2014]. This approach lends itself to software-intensive systems and

may not be appropriate for systems where hardware development is involved.

Another approach, sometimes used by the National Geospatial-Intelligence Agency, is to use a

multi-year indefinite delivery, indefinite quantity (IDIQ) contract with task awards set up on a

short fixed schedule (on the order of six months). A SOO describing the desired functionality (in

terms of a prioritized backlog) is provided to the contractor. The contractor then proposes a PWS

to address the prioritized backlog for the upcoming task order period. It is somewhat expected that

not all low-priority backlog items will be completed before the end of the task order. Near the end

of the existing task order, the next task order is developed with a revised and reprioritized back-

log. This model enables the National Geospatial Agency (NGA) to be very responsive to new

mission needs.

In highly complex systems or systems requiring hardware development, such as is often the case

with defense systems, a hybrid model may be more appropriate. In this case, the high-level capa-

bilities are fixed by the government, the features and their associated priority (prioritized backlog)

are developed jointly between the government and the contractor, and the stories (the “how”) are

totally within the contractor’s control. Changes to capabilities or features usually require a con-

tract modification (if total scope is going to increase) or trading (reducing) other scope so as not to

impact the overall cost and schedule of the program. In this case, a FFP-type development con-

tract is not normally appropriate. Cost plus fixed-fee (CPFF) or a similar contract vehicle would

be applicable.

CMU/SEI-2016-SR-025 21

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Parametric Bidding: Waterfall Actuals Don’t Necessarily Apply

Historical actuals and other metrics are key to the cost estimate. The government must understand

the metrics/measures supporting the contractor’s estimate to accurately assess a contractor’s cost

bid. Waterfall actuals (and to a large extent waterfall metrics) don’t necessarily carry over to an

Agile development program. Actuals reflect the particular skill mix within specific disciplines and

efficiencies that often correlate to different development stages in the waterfall model. In an Agile

program, the focus is on optimizing the skill mix and defining metrics that provide overall value

to the customer. As such, the staffing mix and staffing curves will likely look quite different when

comparing Agile and waterfall programs.

When looking at historical actuals from previous Agile programs, the higher the percentage of the

program that was performed using the Agile methodology, the more accurately they can be as-

sessed and the less likely they will be broken out by discipline. If the breakdown by discipline can

be determined, it will likely be inconsistent with similar programs executed using the waterfall

model.

Using Story Points for Bidding

First the authors must warn that using story points for bidding is not a good practice unless there

is a standard and normalized process to determine story point metrics. This is because story points

are typically a sizing measure unique to each team and are not used to compare productivity

across teams but rather for the team to estimate work and determine if they are improving.

However, if monetized story points are the expected work unit in the RFP, the historical actuals

must be examined from the perspective of what the contractor is using as a basis for the expected

productivity. The most reliable estimates will come from existing Agile teams with historical data

on their velocity for the equivalent type of development. The next most reliable estimate would

come from a prior Agile program. If no existing or prior Agile programs are cited, the government

could use the approach cited previously, which is to start up the program using a time and materi-

als approach to “set” the expected velocity for the Agile teams per the TechFAR. In these cases,

remember to account for slower velocities during the start-up of the program, as the teams require

some initial time to ramp up.

Schedule

Unlike a traditional schedule, which is often organized in a serial fashion following the waterfall

process (i.e., design then code then test) by discipline, an Agile integrated master schedule should

be focused on the development of system capabilities and features. Known dependencies between

features should be clearly identified. Completion criteria for each feature should be clearly under-

stood and the associated progress towards working capabilities (which is the overall goal) should

also be easily discernible. An Agile schedule will employ a rolling baseline where details are de-

veloped two events forward. This just-in-time planning approach defers commitment-level deci-

sions until they are ready to execute, preserving trade space and agility to respond to change. The

temptation to include stories in the IMS should be avoided. Capturing that low level of detail

overly constrains the contractor and likely ensures that the IMS will be in a constant state of

change. Stories should be managed in the Agile toolset at the working-team level. As such, these

CMU/SEI-2016-SR-025 22

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

tools will provide the quantifiable backup data (sometimes known as inchstones) used to update

the IMS.

4.6 Negotiation

While the Agile Manifesto [Agile Alliance 2001] values “customer collaboration over contract

negotiation,”

this does not suggest that contract negotiation is not required, but rather that the con-

tract as negotiated should enable Agile behavior, including making allowance for refinement and

prioritization of requirements by the government during program execution, including the possi-

bility that some low-priority requirements will not be completed [OUSD 2014].

To apply Agile management processes effectively, the government and contractor must discuss

and clearly define a few key elements of the contract.

The scope of the project must be defined clearly, so that it’s clear when an engineering change

proposal (ECP), request for change (RFC) or new task order (for IDIQ type contracts) is required,

but not in such detail that swapping features or changing focus within the scope of the project is

impeded. This requires a change in mindset by both parties. First, there must be greater focus on

defining the boundaries of the project scope, so that it’s obvious when a proposed change is out of

scope. Second, the parties must recognize that not every possible feature, function, or idea within

the scope of the project will be done, and develop a plan to ensuring that government needs are

met without the contractor being required to do more work than was bid and planned.

The TechFAR Handbook recommends developing a high-level product or system vision to de-

scribe the anticipated scope of the overall acquisition along with tactical documents like a SOO

and PWS for each task order, increment, or delivery to share short-term expectations and define

sign-off criteria. Since these tactical documents cover short time periods (8 to 26 weeks are com-

mon), the focus is on streamlined formats that deliver high-value information. In a typical exam-

ple, the government uses the SOO to specify the features (part of a capability) to be developed in

this period, the non-functional requirements to be met, and a (minimal) set of CDRLs. The con-

tractor responds with a PWS that focuses on the extent of the expected implementation and how

the customer expectations will be verified.

When negotiating a contract for Agile development, focus on defining the following:



identification and management of change



in-scope tradeoffs versus ECPs/RFCs



in-scope trade considerations



What level do you trade? Features, capabilities, or both?



Who can approve trades?



How are trades documented?



What are the sell-off expectations and requirements for traded items?



setting the stage for the first program increment



lessons learned and pitfalls to avoid

CMU/SEI-2016-SR-025 23

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

See Agile Government Leadership’s Agile Government Handbook for additional references and

recommendations [Agile 2016].

CMU/SEI-2016-SR-025 24

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

5 Changes in Program Execution

While this section is not specific to RFP development, it provides additional information that the

government needs to understand as it frames its RFP.

5.1 Start-Up (Kickoff)

The start-up for an Agile program can be quite different from that of a traditional program. The

individual government program offices may also want to include an explanation of this activity

within Section L under the Agile development management plan.

Because Agile practices focus on small increments of work executed in tight time boxes and/or

employ a Kanban pull process, start-up on programs using such methods tends to be austere, in-

tense and focused on getting the Agile machine up and running. A clear set of expectations and

plan for start-up is essential, as the output of start-up provides initial input for Agile (usually

scrum) operations, and any start-up tasks that are not completed will detract from the capacity of

the teams. The following topics should be included in the Agile development management plan:



reviews



development and infrastructure runway (What has to be in place before the scrum teams can

work?)



initial backlog grooming (population of the backlog and subsequent prioritization)



cadence (battle rhythm on which the program operates)



subcontractors and suppliers and their interaction with the Agile cadence



managing the technical baseline

5.2 Measures/Metrics

Agile methods are seen by some as an effective means to shorten delivery cycles and manage

costs for the development and maintenance of major software-reliant systems in the DoD. If these

benefits are to be realized, the personnel who oversee major acquisitions must be conversant in

the measures/metrics used to monitor these programs.

There is generally a reliance on a time-box approach in place of the traditional phase-gate ap-

proach to managing progress. This means that the development organization is working to max-

imize valuable work performed within strictly defined time boundaries. The schedule remains

fixed, and the work to be performed is the variable. If Kanban methods are used in conjunction

with Agile methods, then the program needs to be aware of a synchronization activity that must

occur on release boundaries at a minimum. This is in contrast to many traditional approaches

where the period of performance may be subject to negotiation, while attempting to fix the scope

of the work product(s) to be delivered.

CMU/SEI-2016-SR-025 25

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

A focus on delivering usable capabilities replaces the scheduled delivery of interim work products

and the ceremonies that typically focus on milestones. This means that the customer (or customer

representative) should expect to see the working product frequently. The planned interactions and

demonstrations permit a disciplined process for changing the requirements and shaping the final

form of the delivered product. Ideally, acceptance testing happens iteratively throughout develop-

ment, rather than at the end.

Quality and customer satisfaction is an area where Agile methods provide greater opportunity for

insight than traditional development approaches tend to allow. Delivery and progress monitoring

is the area in which perhaps the greatest difference is seen in Agile development, compared to tra-

ditional approaches. The frequent delivery of working (potentially shippable) products renders a

more transparent view of progress than is typically apparent through examination of intermediate

work products. Demonstrations of system capabilities allow early opportunities to refine the final

product, and ensure that the development team is moving toward the desired technical perfor-

mance—as opposed to just asking whether they will complete on schedule and within budget

[Hayes 2013].

In many instances, earned value management (EVM) metrics are required. These are typically

based on a plan that is completed up front and rarely changes [Agile Alliance 2001]. If learning

occurs that changes the plan, both revision of the plan and the EVM milestones require an arduous

change process. Agile implementations, by definition, allow and even embrace change. An Agile

response to EVM requirements could provide considerable amounts of data taken from a typical

Agile implementation, but that data would need to be interpreted and translated to compliance

metrics. The RFP should request information on how that translation would be accomplished and,

if missing, the government should request that information. In addition, the government personnel

need to have an understanding of the Agile metric data and what it does or doesn’t mean [Hayes

2013].

Agile and Earned Value Management: A Program Manager’s Desk Guide states “Agile, as a

product development methodology, can exist within the disciplines required for EVMS compli-

ance.” There are currently no DoD standards for Agile methodology metrics. Metrics such as ve-

locity and burn-down/burn-up must be agreed on by the contractor and the government PMO un-

less they are only used internally by the contractor teams. Agile metrics should provide status that

supports and aligns with similar EVM metrics [PARCA 2016].

Be prepared to mine and effectively use the metrics data that naturally occur in typical Agile

teams. Keep in mind that the metrics from one Agile team cannot be compared to those of another

team as the type and nature of both the work performed and skills of the teams are typically quite

different [Hayes 2013].

The increased number of iterations that result in both a product to evaluate and the exercise of de-

fect detection and testing activities actually makes some data collection for measurement easier in

Agile settings—but the typical traditional measures used for reporting progress need adaptation to

provide productive views of progress in Agile projects [Foreman 2014].

CMU/SEI-2016-SR-025 26

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

5.3 WBS Discussion

The WBS is one of the key artifacts in an RFP as it is one of the primary vehicles for communica-

tion between the government and the contractor. An appropriately constructed WBS is critical for

successfully structuring and tracking the progress of the program. For an Agile program, provid-

ing a standard waterfall WBS can be the cause of significant unnecessary work on the part of both

the government and the contractor. Although MIL-STD-881C is required to facilitate cost and

earned value reporting, it does also allow for a structure that is both compliant and geared towards

an Agile program [PARCA 2016]. In lieu of the “standard” waterfall-based workflow hierarchy

most commonly associated with 881C, a capabilities- and features-based structure should be used.

See Agile and Earned Value Management: A Program Manager’s Desk Guide for a more thor-

ough discussion including a generic example.

Figure 6 and Figure 7 provide two different views of WBS, one from the waterfall perspective and

one from an Agile perspective. Notice that the constructs of the WBS reflect the constructs of the

method as was depicted in Figure 1 and Figure 2.

Figure 6: Software (SW) Development MIL-STD 881C Appendix K WBS Breakout (Traditional Water-

fall) [PARCA 2016]

Figure 7: Possible Agile SW Development MIL-STD-881C WBS Breakout (Agile Capability-Based)

[PARCA 2016]

CMU/SEI-2016-SR-025 27

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

5.4 Change Management

Effective change management should be as simple as adding to the product backlog. It seldom is.

Change management becomes an issue when the developer contends the change is out of scope of

his or her original estimate and negotiated price. One possible solution is a written understanding

between the developers and the acquirers that every change introduced may displace an existing

backlog item thus keeping the initial planned effort consistent. This understanding must take into

account what level of change is being considered. By level of change, the authors are referring to

level of abstraction of the requirements—capability/feature level versus story level. The highest

level of requirements abstraction that establishes the program’s scope should be under formal

change control and assessed for scope impact. The lower levels of requirements abstraction (e.g.,

stories) should be able to be changed by the product owner without needing a contract modifica-

tion.

It is important to define how changes will be handled in the RFP and how the resulting agreement

will be documented in the contract.

5.5 Risk Assessment

The underlying risk management process will not change substantially under an Agile develop-

ment/acquisition program. However, the iterative nature of an Agile program will address risks on

a more frequent basis as the iterative, incremental work reduces the risk throughout the program.

The projects will benefit from seeing some risks mitigated more quickly. Mitigation activities that

come out of the risk management process are fed into the backlog and prioritized in the same

manner as other requirements, with the stakeholders deciding the level of risk with which they are

comfortable.

The RFP should request that the developer identify the existing risk management processes to en-

sure they meet the project’s needs.

5.6 Development Methodologies

Many organizations still employ waterfall development methods for situations that have well-

known, stable requirements. Hardware development is also more often implemented using water-

fall. Alternatively, teams developing applications that must be highly flexible and quickly updated

to changing environments can find Agile methods more conducive to success. Other programs are

not completely black or white in their software and hardware development methodologies: they

benefit by employing a hybrid approach that takes advantage of the strengths of both Agile and

waterfall development.

Before creating an RFP, the government must decide within its acquisition strategy what develop-

ment method is acceptable for its program. This section compares and highlights the advantages

and disadvantages of these three development models: waterfall, Agile, and the Agile/waterfall

hybrid model. Keep in mind that even within each of these models, there is a great deal of varia-

tion with no black and white answers.

CMU/SEI-2016-SR-025 28

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

5.6.1 The Waterfall Model

The waterfall model is an approach for developing software that breaks a project into sequential

phases. A program moves to the next phase only when the preceding phase completes and is veri-

fied. (See Figure 1 in Section 4.1.3.) Figure 8 presents the waterfall phases and highlights some

potential waterfall advantages and disadvantages.

Figure 8: Engineering V Model

Potential Advantages of the Waterfall Software Development Model



Waterfall development is typically suitable for hardware projects and where software re-

quirements are well understood and stable. Note that hardware Agile development may be

less mature than Agile software development, but the principles still hold true across hard-

ware, software, and systems.



The waterfall model is well known within the government, thoroughly defined, and focuses

on schedule-driven activities by project phase.



It is a very regulated model in that each phase has specific deliverables and checkpoint re-

view processes.



Documentation and developmental artifacts are meticulously reviewed.

See AFLCMC Systems Engineering Technical Review (SETR) Guide Version 1.4.

CMU/SEI-2016-SR-025 29

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Potential Disadvantages of the Waterfall Software Development Model



The waterfall model is not typically suitable for projects where requirements are at risk of

emerging or changing. There are many examples of programs where the requirements were

thought to be well understood but still suffered at the hands of the waterfall model.



There is infrequent communication between the stakeholders, end users, and the develop-

ment teams.



Development projects typically take longer to produce a useable product than similar Agile

or hybrid programs.



Because system capabilities are not normally demonstrable until late in the lifecycle, isolat-

ing and fixing defects is more difficult and costly.

Keep in mind the above advantages and disadvantages are what is typically thought of for the wa-

terfall model. There is no black and white to these two sets of ideas; these

advantages and disad-

vantages must be considered based on the specific program environment.

Note: The founder of the waterfall development methodology, Winston Royce, warned of em-

ploying this methodology for large software systems back in 1970: “I believe in this concept, but

the implementation is risky and invites failure. The testing phase, which occurs at the end of the

development cycle, is the first event for which timing, storage, input/output transfers, etc. are ex-

perienced, as distinguished from analyzed. If these phenomena fail to satisfy the various external

constraints, then invariably a major redesign is required. Either the requirements must be modi-

fied, or a substantial change in the design is required. In effect, the development process has re-

turned to the origin and one can expect up to 100-percent overruns in schedule and/or cost”

[Royce 1970].

5.6.2 The Agile Model

Wikipedia defines the pure Agile model as “a group of software development methods based on

iterative and incremental development, where requirements and solutions evolve through collabo-

ration between self-organizing, cross-functional teams.” The scrum model is presented in Appen-

dix A as it is the most commonly used Agile method.

The following presents generalizations of the potential advantages and disadvantages of the Agile

methods. As with waterfall, there is no black and white answer. The implementation of Agile in a

specific environment is dependent on the issues that are being solved within an UURVE environ-

ment.

Potential Advantages of the Agile Software Development Model



It is typically good for programs with high requirements volatility and frequent changes to

the user interface.

CMU/SEI-2016-SR-025 30

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see



It facilitates high stakeholder, product owner, scrum master, and development team collabo-

ration.



Generally, there is increased software quality as working software is continuously devel-

oped, re-tested, and demonstrated.



Emergent requirements and developmental priority fluctuations are more easily accommo-

dated than in the waterfall process.



Continuous improvement to the product and process is facilitated through refactoring and

sprint retrospectives.



Release times are shortened for deliverable working code; the overall product development

lifecycle can shrink.

Potential Disadvantages of the Agile Software Development Model



Many DoD acquisition program offices do not fully understand or embrace Agile develop-

ment methodologies.



Contracting often locks in scope and budget up front, making pure Agile somewhat difficult

to implement.



Technical reviews and audits often require iterative execution, as opposed to one-time water-

fall events.



It can be difficult to manage partners employing waterfall development methodologies.



It requires training and experience in Agile vocabulary, roles, ceremonies, tools, and metrics

collection and analysis. It requires adjustments to program-level activities.



There is a challenge of learning and implementing the new product owner role (or equivalent

in non-scrum Agile implementations). This is only an initial disadvantage and should be re-

solved once the personnel are trained, gain experience, and use this as the new way of doing

business.

5.7 Mixing Waterfall and Agile Methodologies

There are several factors that may lead to the use of mixed development methodologies on a pro-

gram. They could include the maturity level of Agile as applied to systems engineering and hard-

ware engineering, the Agile maturity level of an organization, or the size/complexity of a pro-

gram. Another possibility is that a company may desire to take advantage of Agile software

development methodologies (to reduce cost) while adhering to a traditional (waterfall) SOW. Re-

gardless, there is not a blanket recommendation for or against the use of such mixed methodolo-

gies. However, it should be noted that mixing methodologies is not without pitfalls and may be

disruptive to both the government's program office and the companies with established processes.

Product owner and scrum master are terms usually associated with the scrum Agile method. These terms may

not be associated with other forms of Agile even though the roles performed by those positions may be repre-

sented.

Some may consider this an advantage as there is early insight into technical data.

Managing Agile and waterfall teams working together requires coordinated planning and synchronization points.

CMU/SEI-2016-SR-025 31

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

As Agile software development methodologies are generally more mature than Agile systems en-

gineering (SE) or Agile hardware processes, one notable model combines traditional waterfall

phases, process reviews, and artifact collection with iterative Agile development and functional

releases. Requirements analysis and design are performed in a waterfall fashion with code devel-

opment of evolving functionality being iteratively demonstrated to the product owner and other

stakeholders.

In additional to potential cost savings, leveraging a combined waterfall and Agile approach may

facilitate incremental buy-in by a program manager cautious of converting directly from waterfall

to Agile acquisition. While potentially reaping some of the efficiencies often seen in Agile soft-

ware development, this model still suffers from the problems inherent with completely defining

the solution (captured in requirements) before coding even begins. This is only one example of

many possible hybrid approaches.

5.7.1 The Changing Landscape of Software Acquisitions—Employing Hybrid

Agile Models

Agile author and software expert Chuck Cobb proposes a hybrid approach consisting of a plan-

driven “envelope” at the macro level, coupled with a micro-level Agile approach for use case

(UC) development and test from the product backlog. This envelope makes it less difficult to pro-

vide cost and schedule estimates for a hybrid Agile acquisition program. His advice is that soft-

ware development using Agile or waterfall is not a binary decision, noting: “Seeing Agile and

plan–driven approaches as mutually exclusive is an example of Binary Thinking. There are lots of

ways to blend Agile and plan-driven approaches together in the right proportions to fit a particular

software and hardware situation; it just requires more skill to do that” [Cobb 2013].

5.7.2 DoDI 5000.02 Facilitates Hybrid Agile Software and Waterfall Hardware

Development

Acquisition instruction DoDI 5000.02 proposes six acquisition program models as a starting point

for program-specific planning [DoD 2015]:



Model 1: Hardware Intensive Programs



Model 2: Defense Unique Software Intensive Programs



Model 3: Incrementally Fielded Software Intensive Programs



Model 4: Accelerated Acquisition Programs



Model 5: Hybrid Program A (Hardware Dominant Programs)



Model 6: Hybrid Program B (Software Dominant Programs)

The hybrid acquisition model shown below from 5000.02 is a model depicting how a major weap-

ons system combines hardware development as the basic structure with a software intensive de-

velopment that is occurring simultaneously with the hardware development program. In a hard-

ware intensive development, the design, fabrication, and testing of physical prototypes may

determine overall schedule, decision points, and milestones, but iterative software development

and releases often dictate the pace of program execution and must be tightly integrated and coor-

dinated with hardware development decision points.

CMU/SEI-2016-SR-025 32

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Figure 9: DoD 5000.02 Hybrid Model

[DoD 2015]

Potential Advantages of the Hybrid Model



It is good for integrated hardware/software product development teams: hardware (waterfall)

and software (Agile or Agile hybrid).



It is good for software-intensive product development with IEEE 15288.2

checkpoints such

as PDR and CDR, making it easier to meet audits, standards, and regulatory requirements

than with a purely Agile development.



The model facilitates presentation layer changes to components implementing the user inter-

face and managing user interaction.



The model facilitates waterfall requirements/design with iterative development, test, and

demonstration (Agile).

However, other hybrid models may include SE in the Agile meth-

odology side.



It facilitates frequent presentation layer changes such as thin (mobile) and rich (stand-alone)

client applications.



A hybrid approach facilitates increased emphasis on documentation as a deliverable.



A hybrid approach facilitates increased emphasis on automated test cases and test scripts as

iterative deliverables.

IEEE-Std-15288.2 is the 2014 IEEE Standard for Technical Reviews and Audits on Defense Programs

See https://www.scrumalliance.org/community/articles/2010/october/negotiating-scrum-through-a-waterfall

CMU/SEI-2016-SR-025 33

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see



Integration test commences earlier in a hybrid Agile/waterfall software development lifecy-

cle (SDLC), and Agile tool integration is gradual.



It’s easier to gain buy-in from program offices with an internal culture of and propensity for

waterfall development.



Hybrid models merge strategic planning and cost management with iterative product owner

value, balancing the need for control with adaptability to changing requirements.

Disadvantages of the Hybrid Model



It requires extensive knowledge of waterfall and Agile models to implement.



Project planning and resultant scheduling are more difficult—such as differing EV structures

between the Agile/waterfall portions, completion of system capabilities in software (Agile)

versus incremental progress in waterfall, and others.

5.8 Bidder’s Library

The contents of the bidder’s library should not vary from the types of data typically included on a

traditional program RFP.

CMU/SEI-2016-SR-025 34

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

6 Conclusion

The authors have provided information that can be employed to write an RFP that allows the use

of Agile methods. Many of the items in the RFP will remain unchanged. However, the implemen-

tation of certain practices, such as metrics and cadence, will change.

This report is not meant to be prescriptive but rather to be used as a resource to help answer some

of the questions RFP writers will encounter when they endeavor to write an RFP allowing the im-

plementation of Agile methods. Undoubtedly, once a larger majority of government program of-

fice personnel have experience in this arena, this report will require a much needed update.

CMU/SEI-2016-SR-025 35

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Appendix A - Agile Fundamentals

Manifesto for Agile Software Development

We are uncovering better ways of developing software by doing it and helping others do it.

Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

The guiding principles behind the Agile Manifesto are as follows:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable

software.

Welcome changing requirements, even late in development. Agile processes harness change for

the customer's competitive advantage.

Deliver working software frequently, from a couple of weeks to a couple of months, with a prefer-

ence to the shorter timescale.

Business people and developers must work together daily throughout the project.

Build projects around motivated individuals. Give them the environment and support they need,

and trust them to get the job done.

The most efficient and effective method of conveying information to and within a development

team is face-to-face conversation.

Working software is the primary measure of progress.

Agile processes promote sustainable development. The sponsors, developers, and users should be

able to maintain a constant pace indefinitely.

Continuous attention to technical excellence and good design enhances agility.

Simplicity—the art of maximizing the amount of work not done—is essential.

The best architectures, requirements, and designs emerge from self-organizing teams.

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its

behavior accordingly [Agile Alliance 2001].

CMU/SEI-2016-SR-025 36

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Scrum is the most commonly used Agile method. Its lifecycle is depicted in Figure 10.

Figure 10: The Agile Software Development Lifecycle (reproduced with permission from Mountain Goat

Software)

CMU/SEI-2016-SR-025 37

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Appendix B - Marbach Updated Principles to the Agile

Manifesto

Original Agile Manifesto Principles [Agile Alliance 2001] Marbach Updated Principles [Marbach 2015]

Our highest priority is to satisfy the customer through early

and continuous delivery of valuable software.

First satisfy the customer through early and continuous de-

livery of valuable capabilities.

Welcome changing requirements, even late in develop-

ment. Agile processes harness change for the customer's

competitive advantage.

Plan for evolving requirements, and retain as much flexibil-

ity as is valuable throughout development, especially when

change leads to a competitive advantage.

Deliver working software frequently, from a couple of

weeks to a couple of months, with a preference to the

shorter timescale.

Deliver working capabilities frequently, from a couple of

weeks to a couple of months, with a preference to the

shorter timescale.

Business people and developers must work together daily

throughout the project.

Business personnel, customers, or their advocates, and

implementers must work together daily throughout the pro-

ject.

Build projects around motivated individuals. Give them the

environment and support they need, and trust them to get

the job done.

No change.

primary measure of progress agile

processes promote sustainable de-

velopment the sponsors

The most efficient and effective method of conveying infor-

mation to and within a delivery team is personal conversa-

tion.

Working software is the primary measure of progress. Working capabilities are the primary measure of progress.

Agile processes promote sustainable development. The

sponsors, developers, and users should be able to main-

tain a constant pace indefinitely.

Agile processes promote sustainable delivery. The spon-

sors, developers, and users should be engaged and able

to maintain a constant pace to the project completion.

Continuous attention to technical excellence and good de-

sign enhances agility.

No change.

Simplicity—the art of maximizing the amount of work not

done—is essential.

Simplicity (“the art of maximizing the amount of work not

done”) is essential, especially within the implementation

team. A truly Agile development project does not force arti-

ficial reporting and process requirements on the implemen-

tation team.

The best architectures, requirements, and designs emerge

from self-organizing teams.

The best architectures, requirements, and designs emerge

from self-organizing teams, based on a minimal set of

guiding principles.

At regular intervals, the team reflects on how to become

more effective, then tunes and adjusts its behavior accord-

ingly.

No change.

CMU/SEI-2016-SR-025 38

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Appendix C - Acronyms

ADM

Acquisition Decision Memorandum

ASR

Alternative Systems Review

CDR

Critical Design Review

CDRL

Contract Data Requirements List

CPFF

Cost Plus Fixed-Fee

CSOW

Contractor Statement of Work

CWBS

Contract(or) Work Breakdown Structure

DDIT

Design-Develop-Integrate-Test

DoD

Department of Defense

DoDD

Department Of Defense Directive

DoDI

Department Of Defense Instruction

Deficiency Report or Discrepancy Report

DT&E

Developmental Test & Evaluation

ECP

Engineering Change Proposal

Earned Value

EVM

Earned Value Management

FAR

Federal Acquisition Regulations

FFP

Firm-Fixed-Price

F.I.R.E.

Fast, Inexpensive, Restrained, And Elegant

GAO

Government Accounting Office

ICD

Interface Control Document

CMU/SEI-2016-SR-025 39

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

IDIQ

Indefinite Delivery, Indefinite Quantity

IMS

Integrated Master Schedule

INCOSE

International Council on Systems Engineering

MIL STD

Military Standard

NDIA

National Defense Industrial Association

NGA

National Geospatial Agency

OT&E

Operational Test and Evaluation

OUSD

Office of the Under Secretary of Defense

PARCA

Performance Assessments and Root Cause Analyses (PARCA)

PBA

Performance-Based Acquisition

PDR

Preliminary Design Review

PMO

Program Management Office

PWS

Performance Work Statement

RFC

Request for Change

RFP

Request for Proposal

SDLC

Software Development Lifecycle

Systems Engineering

SEI

Software Engineering Institute

SFR

System Functional Review

SOO

Statement of Objectives

SOW

Statement of Work

SRR

System Requirements Review

SVR

System Verification Review

Software

CMU/SEI-2016-SR-025 40

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

TPM

Technical Performance Measurement

TRR

Test Readiness Review

Use Case

UURVE

Unpredictability, Uncertainty, Risk, Variability, and Evolution

WBS

Work Breakdown Structure

CMU/SEI-2016-SR-025 41

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Appendix D - Glossary

Glossary terms as defined in a variety of industry sources [CC Pace 2016, Scrum Alliance 2016,

Scrum Guides 2016, SolutionsIQ 2016].

Agile

The name coined for the wider set of ideas that scrum falls within; the Agile

values and principles are captured in the Agile Manifesto.

Backlog Item

See Product Backlog Item.

Burndown

See Sprint Burndown, Product Burndown.

Capability

A customer-required ability of the system that provides value to the customer.

Daily Scrum

A short daily team meeting to share progress, report impediments and make

commitments.

Done

Also referred to as “Done” or “Done Done” this term is used to describe a

product increment that is considered releasable; it means that all design, cod-

ing, testing, and documentation have been completed and the increment is

fully integrated into the system. Done also refers to holistic completion of

smaller components of an Agile system including user stories, features, and

capabilities that may not be releasable.

Empiricism

Empiricism is the principle of “inspect and adapt,” which allows teams or indi-

viduals to try something out and learn from the experience by conscious re-

flection and change.

Epic

A very large user story that is eventually broken down into smaller stories; ep-

ics are often used as placeholders for new ideas that have not been thought

out fully. (This definition is used by the SAFe scaling framework.) See also,

Capability.

Estimation

The process of agreeing on a size measurement for the stories in a product

backlog.

Feature

A part of a capability that can be completed within a program increment; it

also has business value, is estimable and testable.

Iteration

See Sprint.

Non-Functional Requirements

(NFR)

Constraints on functional items in the backlog. NFRs include standards, relia-

bility, maintainability, availability, and performance. NFRs are qualities of the

delivered items and cannot be delivered by themselves.

Planning

See Sprint Planning

Process

Simply the way someone works. Everyone has a process. It can be pre-de-

fined, empiric, or merely chaotic.

Product Backlog

A prioritized list of stories that are waiting to be worked on.

Product Backlog Item

Any item that is on the backlog list, which will include user stories, epics, and

possibly technical stories to deal with technical debt, etc.

Product Owner

The person who holds the vision for the product and is responsible for main-

taining, prioritizing, and updating the product backlog.

Release

The term used to describe a concrete time box or cadence used to complete

features. Release duration can vary, but is typically three to six months. Many

practitioners use the release cadence as their rolling wave planning period.

CMU/SEI-2016-SR-025 42

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

Retrospective

A session where the team and scrum master reflect on the process and make

commitments to improve.

Roadmap

Capabilities and their associated features mapped to program increments and

software builds/deliveries; the dependencies between the features are also

captured. A roadmap is a living artifact, which is less accurate the farther in

the future it projects.

Scrum Master

A servant leader to the team, responsible for removing impediments and mak-

ing sure the process runs smoothly so the team can be as productive as pos-

sible.

Self Organization

The principle that those closest to the work best know how to do the work, so

set clear goals and boundaries and let them make all tactical and implementa-

tion decisions.

Sprint

A term frequently used interchangeably with iteration to describe a concrete

time box or cadence used to complete stories. Sprint duration can vary, but is

typically two to four weeks.

Sprint Burndown

A visible chart that indicates on a daily basis the amount of work remaining in

the sprint.

Sprint Planning

A meeting between the team and the product owner to plan the sprint and ar-

rive at an agreement on the commitment.

Stakeholder

Anyone external to the team with an interest in the product being developed.

Story

A backlog item usually using the template form: as a [user], I want [function],

so that [business value]; also known as a user story. Stories contribute to the

completion of a feature and can be completed within a single sprint.

Team

The development team, responsible for committing to work and delivering and

driving the product forward from a tactical perspective.

Team Member

Any member of the team, including developers, testers, designers, writers,

graphic artists, and database admins.

Time Boxing

Setting a duration for every activity and having it last exactly that (i.e., neither

meetings nor sprint are ever lengthened—ever).

Velocity

The rate at which a team completes work, usually measured in story points.

Vision Statement

A high-level description of a product that includes who it is for, why it is neces-

sary, and what differentiates it from similar products.

Work in Progress (WIP)

Limiting the work in progress is a principle used in Kanban to encourage qual-

ity and completion of work, among other things.

CMU/SEI-2016-SR-025 43

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

References

URLs are valid as of the publication date of this document.

[Agile 2016]

Agile Government Leadership. Agile Government Handbook. Agile Government Leadership.

2016. http://www.agilegovleaders.org/wp-content/uploads/2016/02/AgileGovernmentHandbook-

1.pdf

[Agile Alliance 2001]

History: The Agile Manifesto. Agile Alliance. 2001.

http://agilemanifesto.org/history.html

[Ambler 2013]

Ambler, Scott. Agile Testing and Quality Strategies: Discipline Over Rhetoric. 2013.

http://www.ambysoft.com/essays/agileTesting.html

[CCPace 2016]

CCPace. Glossary of Scrum Terms. 2016. http://www.ccpace.com/asset_files/ScrumGlossary-

TermsNEW.pdf

[Cobb 2013]

Cobb, Charles G. Managing Agile Development–Making Agile Work for Your Business. Outskirts

Press. 2013.

[DoD 2007]

U.S. Department of Defense. DoDD 5000.01 The Defense Acquisition System. Nov. 20, 2007.

http://www.dtic.mil/whs/directives/corres/pdf/500001p.pdf

[DoD 2015]

U.S. Department of Defense. DoDI 5000.02 Operation of the Defense Acquisition System. Jan. 7,

2015. https://acc.dau.mil/CommunityBrowser.aspx?id=716926

[Dove 1993]

Dove, R. Lean and Agile: Synergy, Contrast, and Emerging Structure. Defense Manufacturing

Conference '93. Nov. 29-Dec. 2, 1993.

[Foreman 2014]

Foreman, John. Report from the Trenches: Agile Adoption in the Federal Government. Associa-

tion for Enterprise Information Agile in Government Conference. June 2014.

http://www.afei.org/PE/4A02/Pages/default.aspx

[GAO 2008]

U.S. Government Accountability Office. Defense Acquisitions: Assessments of Selected Weapon

Programs (GAO-08-467SP). 2008. http://www.gao.gov/new.items/d08467sp.pdf

CMU/SEI-2016-SR-025 44

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

[Hayes 2013]

Hayes, William; Miller, Suzanne; Lapham, Mary Ann; Wrubel, Eileen; & Chick, Timothy. Agile

Metrics: Progress Monitoring of Agile Contractors. CMU/SEI-2013-TN-029. Software Engineer-

ing Institute, Carnegie Mellon University. 2014.

http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=77747

[IEEE 2016]

IEEE. IEEE Standard for Technical Reviews and Audits on Defense Programs (IEEE Standard

15288.2-2014). 2016. https://standards.ieee.org/findstds/standard/15288.2-2014.html

[INCOSE 2016]

Agile Systems & S.E.: Mission Objectives. INCOSE. 2016. http://www.incose.org/Chap-

tersGroups/WorkingGroups/Transformational/agile-systems-se

[Lapham 2014]

Lapham, Mary Ann; Bandor, Michael; & Wrubel, Eileen. Agile Methods and Request for Change

(RFC): Observations from DoD Acquisition Programs. CMU/SEI-2013-TN-031. Software Engi-

neering Institute, Carnegie Mellon University. 2014.

http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=77732

[Marbach 2015]

Marbach, Phyllis; Rosser. Larri; Gundars, Osvalds; & Lempia, David. Principles for Agile Devel-

opment. Volume 25. Issue 1. Version of Record online: 28. INCOSE International Symposium.

October 2015.

http://onlinelibrary.wiley.com/doi/10.1002/j.2334-5837.2015.00079.x/pdf

[Navy 1998]

U.S. Navy. Acquisition Packages: Proposal Evaluation Plan Template. Acquisition Guide. 1998.

http://www.navair.navy.mil/nawctsd/Resources/Library/Acqguide/acqpack.htm

[Nidiffer 2014]

Nidiffer, Kenneth; Miller, Suzanne; & Carney, David. Potential Use of Agile Methods in Selected

DoD Acquisitions: Requirements Development and Management. CMU/SEI-2013-TN-006. Soft-

ware Engineering Institute, Carnegie Mellon University. 2014.

http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=89158

[OUSD 2014]

Office of the Undersecretary of Defense. TechFAR Handbook for Procuring Digital Services Us-

ing Agile Processes. 2014. https://playbook.cio.gov/techfar/

[Palmquist 2013]

Palmquist, Steven; Lapham, Mary Ann; Garcia-Miller, Suzanne; Chick, Timothy; & Ozkaya,

Ipek. Parallel Worlds: Agile and Waterfall Differences and Similarities. CMU/SEI-2013-TN-021.

Software Engineering Institute, Carnegie Mellon University. 2013.

http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=62901

CMU/SEI-2016-SR-025 45

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

[PARCA 2016]

Office of the Undersecretary of Defense, Performance Assessments and Root Cause Analyses.

Agile and Earned Value Management: A Program Manager’s Desk Guide. 2016.

http://www.acq.osd.mil/evm/docs/PARCA%20Agile%20and%20EVM%20PM%20Desk%20Gui

de.pdf

[Royce 1970]

Royce, Winston. Managing the Development of Large Software Systems, IEEE WESCON Pro-

ceedings. TRW Corp. August 1970.

[Scrum Alliance 2016]

Scrum Alliance. Scrum Alliance. 2016. https://www.scrumalliance.org/

[Scrum Guides 2016]

The Scrum Guide. Scrum Guides. 2016. http://www.scrumguides.org/scrum-guide.html

[SolutionsIQ 2016]

Agile Glossary. SolutionsIQ. 2016. http://www.solutionsiq.com/agile-glossary/

[Ward 2014]

Ward, Dan. F.I.R.E., How Fast, Inexpensive, Restrained, and Elegant Methods Ignite Innovation.

HarperCollins Books. 2014.

CMU/SEI-2016-SR-025 46

SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see

REPORT DOCUMENTATION PAGE

Form Approved

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, search-

ing existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regard-

ing this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters

Services, Directorate for information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of

Management and Budget, Paperwork Reduction Project (0704-0188), Washington, DC 20503.

AGENCY USE ONLY

(Leave Blank)

REPORT DATE

November 2016

REPORT TYPE AND DATES

COVERED

Final

TITLE AND SUBTITLE

RFP Patterns and Techniques for Successful Agile Contracting

FUNDING NUMBERS

FA8721-05-C-0003

AUTHOR

(

)

Mary Ann Lapham, Keith Korzec, Larri Ann Rosser, Greg Howard, Steven Martin, Michael Ryan, Thomas E. Friend, John H. Norton III,

Peter Capell

PERFORMING ORGANIZATION NAME

(

)

AND ADDRESS

(

)

Software Engineering Institute

Carnegie Mellon University

Pittsburgh, PA 15213

PERFORMING ORGANIZATION

REPORT NUMBER

CMU/SEI-2016-SR-025

SPONSORING

MONITORING AGENCY NAME

(

)

AND ADDRESS

(

)

AFLCMC/PZE/Hanscom

Enterprise Acquisition Division

20 Schilling Circle

Building 1305

Hanscom AFB, MA 01731-2116

10.

SPONSORING

MONITORING

AGENCY REPORT NUMBER

n/a

11.

SUPPLEMENTARY NOTES

DISTRIBUTION

AVAILABILITY STATEMENT

Unclassified/Unlimited, DTIC, NTIS

B DISTRIBUTION CODE

13.

ABSTRACT

(

MAXIMUM

200

WORDS

)

Increasing budget constraints and emphasis on fielding capability faster have led the U.S. Department of Defense (DoD) and other federal

entities to pursue the benefits of Agile software development – reduced cycle times, flexibility to adapt to changing conditions and user needs

- that software development practitioners have achieved in the commercial market.

This report is written by the National Defense Industrial Association’s System Engineering Agile Working Group to provide information on

request-for-proposal (RFP) patterns and techniques for successful Agile contracting that can and have been used for contracts seeking to

employ Agile methods. This report is intended to support the writers of RFPs in bringing Agile concepts into programs at the earliest possible

time, providing examples of the kinds of language that will affect the foundations of contracts on which programs rely.

14.

SUBJECT TERMS

Agile, government, RFP, DoD

15.

NUMBER OF PAGES

16.

PRICE CODE

17.

SECURITY CLASSIFICATION OF

REPORT

Unclassified

18.

SECURITY CLASSIFICATION

OF THIS PAGE

Unclassified

19.

SECURITY CLASSIFICATION

OF ABSTRACT

Unclassified

20.

LIMITATION OF

ABSTRACT

NSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89) Prescribed by ANSI Std. Z39-18

298-102