USE OF AN OPEN SYSTEMS APPROACH FOR WEAPON SYSTEMS Report No. D-2000-149 June 14, 2000 Office of the Inspector General Department of Defense Additional Copies To obtain additional copies of this audit report, contact the Secondary Reports Distribution Unit of the Audit Followup and Technical Support Directorate at (703) 604-8937 (DSN 664-8937) or fax (703) 604-8932 or visit the Inspector General, DoD, Home Page at: www.dodig.osd.mil. Suggestions for Future Audits To suggest ideas for or to request future audits, contact the Audit Followup and Technical Support Directorate at (703) 604-8940 (DSN 664-8940) or fax (703) 604-8932. Ideas and requests can also be mailed to: OAIG-AUD (ATTN: AFTS Audit Suggestions) Inspector General, Department of Defense 400 Army Navy Drive (Room 801) Arlington, VA 22202-2884 Defense Hotline To report fraud, waste, or abuse, contact the Defense Hotline by calling (800) 424-9098; by sending an electronic message to Hotline@dodig.osd.mil; or by writing to the Defense Hotline, The Pentagon, Washington, D.C. 20301-1900. The identity of each writer and caller is fully protected. Office of the Inspector General, DoD Report No. D-2000-149 June 14, 2000 (Project No. D1999-D000AE-0101.000) (Formally Project No. 9AE-0091.00) Use of an Open Systems Approach for Weapon Systems Executive Summary Introduction. This report discusses the extent that acquisition program managers considered and used an open systems approach in the design and development of major defense weapon systems. The Under Secretary of Defense for Acquisition, Technology, and Logistics mandated the use of an open systems approach in the acquisition process to reduce the cost of ownership of weapons systems while increasing interoperability and useful life. The Under Secretary chartered an open systems Joint Task Force (the Joint Task Force) in November 1994 to facilitate DoD use and implementation of an open systems approach in weapon systems acquisition. Objectives. The primary audit objective was to evaluate the extent that program managers considered and used an open systems approach in weapons systems development. We also reviewed the effectiveness of management controls applicable to the audit objective. Results. The Joint Task Force has worked diligently to implement the open system approach for DoD weapons systems. However, the Joint Task Force needed increased assistance from the Defense and Component acquisition executives, as well as program managers, to implement the use of an open systems approach in the systems acquisition process. Of the 17 major Defense acquisition programs that gained approval to begin program definition and risk reduction or to enter engineering and manufacturing development between March 1996 and July 1999, 14 programs proceeded into the next acquisition phase without program mangers clearly defining open system design objectives or strategy for achieving the objectives. Specifically, users and program managers did not include language concerning the required use of an open systems design in acquisition planning documents. The following list of seven acquisition planning documents shows the number of programs that did not include language concerning the required use of open systems out of the number of programs that prepared the cited document. • operational requirements document (6 of 17 programs), • single acquisition management plan (2 of 12 programs), • acquisition plan (3 of 5 programs), • system engineering management plan (2 of 6 programs), • request for proposal (9 of 17 programs), • contract statement of work (8 of 15 programs), and • test and evaluation master plan (11 of 17 programs) ii As a result, DoD acquisition managers did not have assurance at program milestone reviews that program managers required and stressed the importance of implementing open system design objectives in acquisition strategies to weapon systems contractors (finding A). Detailed documentation reviews of 4 of the 17 major Defense acquisition program offices showed that program managers for 3 of the 4 programs did not document a means for determining the extent of design openness of systems, subsystems, and components. Also, DoD guidance on open systems did not require program managers to assess the impact of a given level of design openness on the long-term viability and affordability of systems. Without a means to measure the progress and the impact of implementing an open systems approach, acquisition decision makers can not readily gauge how well program managers are achieving the advantages of using an open systems design approach or assessing the susceptibility of a weapon systems design to obsolescence or costly upgrades to counter foreign military threats (finding B). See Appendix A for details on the management control program as it relates to controls over program managers considering and using an open systems design approach in key acquisition planning documentation. Recommendations in this report, if implemented, will improve the process for defining and documenting open systems objectives in key acquisition planning documentation and correct the material control weakness identified in the report. Summary of Recommendations. We recommend that the Under Secretary of Defense for Acquisition, Technology, and Logistics enforce the requirement that program managers consider and use open systems during the acquisition milestone review process and that program manager progress in inserting open systems design requirements in key acquisition planning documents is measured. We also recommend that program managers be required to include open systems objectives in test and evaluation master plans and to assess the impact of projected system design openness to provide acquisition milestone decision makers assurance at acquisition milestone reviews that program managers had stressed the importance of implementing open systems objectives into acquisition strategies. Additionally, we recommend that the Joint Task Force provide program managers with general templates for inserting open systems design language in the key acquisition planning documents and provide guidance to help program managers document the means for determining the extent of system design openness. Management Comments. The Director, Operational Test and Evaluation, agreed to support the Under Secretary of Defense for Acquisition, Technology, and Logistics in revising acquisition policy to require program managers to include open systems objectives in test and evaluation master plans. The Army also agreed with the recommendations in the report. Audit Response. The comments from the Director, Operational Test and Evaluation, were responsive. The Under Secretary of Defense for Acquisition, Technology, and Logistics did not comment on the draft of this report. A discussion of management comments is in the Findings section of the report, and the complete text is in the Management Comments section. We request that the Under Secretary of Defense provide comments to the recommendations in this final report by July 14, 2000. Table of Contents Executive Summary i Introduction Background 1 Objectives 3 Findings A. Addressing Use of Open Systems Objectives in the Weapon System Acquisition Process 4 B. Determining the Extent of System Design Openness 16 Appendixes A. Audit Process Scope 22 Methodology 23 Management Control Program 23 Summary of Prior Coverage 24 B. Definitions of Open Systems Terms 25 C. Public Law and Government Policy 28 D. Open Systems Joint Task Force Education and Outreach Initiatives 29 E. Inclusion of Open Systems Objectives in Acquisition Planning Documents 31 F. Report Distribution 34 Management Comments Director, Operational Test and Evaluation Comments 37 Assistant Secretary of the Army (Acquisition, Logistics, and Technology) Comments 38 Army Program Executive Office for Ground Combat and Support Systems Comments 39 1 Background Open Systems Approach. This report discusses the extent that acquisition program managers considered and used an open systems approach in weapon systems development. An open systems approach to weapon system development is an acquisition reform initiative requiring program managers to implement open specifications for interfaces between systems, subsystems, and components. Industry standards boards (national and international) develop or adopt open specifications to accommodate insertion of new technologies into systems. An open system design for a weapon system is characterized by: • well defined, widely used, preferably nonproprietary interfaces and protocols; • use of interface standards developed or adapted by industry-recognized standards bodies. • definition of all aspects of system interfaces to facilitate new or additional capabilities for a wide range of applications; and • explicit provision for system expansion or upgrade through the incorporation of additional or higher performance elements with minimal negative impact on the existing system. DoD use of an open systems approach will reduce the cost of ownership of weapons systems, delay system obsolescence, and allow fielding superior warfighting capability more quickly. An open systems approach reduces weapon system cost through facilitating program manager use of widely accepted standard products from multiple suppliers in DoD weapon systems. If program managers define weapon system architecture by specifications and standards used in the private sector, DoD can leverage the benefits of the commercial market place, and take advantage of the competitive pressures that motivate commercial companies to improve products and reduce prices. Program managers can then have access to alternative sources for key subsystems and components to construct DoD weapon systems. The open systems approach could reduce the DoD investment early in the weapon system life cycle because some of the required systems or components may be available or under development without direct DoD investment. Also, program managers can competitively select production sources from multiple competitors. Additionally, an open systems approach delays system obsolescence by allowing program managers to incrementally insert technological improvements into existing or developing systems rather than having to make large-scale system redesigns or to develop new systems. Further, an open systems approach enables program managers to deliver weapons systems to warfighters more quickly as a result of reduced developmental effort. Other benefits of program managers using an open systems approach include: 2 • greater system interoperability for more effective joint and allied war fighting, • closer cooperation between commercial and military electronics industries, and • better international competitiveness of the U.S. electronics industry. Overall, DoD use of the open system approach should help in pursuing a focused modernization effort to maintain a qualitative superiority in warfighting capabilities, in meeting the combat forces needs, in controlling cost growth increases, and in helping DoD to meet its goals and performance measures. Appendix A provides details on DoD goals and performance measures in response to the Government Performance and Results Act that are pertinent to this report. Appendix B provides a listing of terms and definitions germane to understanding program manager implementation of an open systems approach in designing weapon systems architecture. Public Law and Government Policy. Public law and Government acquisition policy mandate that program managers consider and use an open systems approach in the weapon system acquisition process. Program managers must consider and use open systems both in developing new weapon systems and in modifying existing, or legacy, systems. Appendix C provides details on relevant public law and Government acquisition policy concerning the use of an open systems approach. Open Systems Joint Task Force. The Under Secretary of Defense for Acquisition and Technology (now Under Secretary of Defense for Acquisition, Technology, and Logistics) chartered an open systems Joint Task Force (the Joint Task Force) in November 1994 to facilitate DoD use and implementation of an open systems approach in weapon systems acquisition. Specifically, the Under Secretary chartered the Joint Task Force to sponsor and accelerate the adoption of open systems in weapon systems and subsystems electronics to reduce life-cycle cost and facilitate effective weapon system intra- and interoperability. In executing the Under Secretaries charter, the Joint Task Force has promoted program managers’ implementation of open systems policy, identified opportunities for implementing an open systems approach, developed training and education programs, and coordinated the identification and selection of open systems specifications and standards. Appendix D provides an overview of the completed and ongoing initiatives of the Joint Task Force. Although the original charter for the Joint Task Force expired in November 1998, the Under Secretary provided funding to support the continuation of the Joint Task Force’s efforts for an additional 3 years in June 1998. 3 Objectives The primary audit objective was to evaluate the extent that program managers considered and used an open systems approach in weapons systems development. We also evaluated the management controls related to the audit objective. Appendix A discusses the audit scope and methodology, as well as management controls and prior audit coverage. 4 A. Addressing Use of Open Systems Objectives in the Weapon System Acquisition Process The DoD acquisition community has not fully applied the use of open systems objectives in the acquisition planning and review process. Of the 17 major Defense acquisition programs that gained approval to begin program definition and risk reduction or to enter engineering and manufacturing development between March 1996 and July 1999, 14 programs proceeded into the next acquisition phase without program managers clearly defining open system design objectives for the system and the strategy for achieving the objectives in key acquisition planning documents. The DoD and Component acquisition executives allowed this condition to occur because they did not enforce the requirement that program managers use an open systems design approach in key acquisition documents as part of the acquisition milestone review process. Without acquisition executive enforcement, program offices did not aggressively seek guidance and training in using an open systems approach for their programs. With respect to training, the Joint Task Force had emphasized providing open system guidance and training to a number of individual acquisition programs but had only provided general guidance to the broader acquisition community through the Defense Acquisition Deskbook, seminars, and publications. As a result, DoD acquisition decision makers did not have assurance at program milestone reviews that program managers required and stressed the importance of implementing open system design objectives in their acquisition strategies to weapon system contractors. Open Systems Policy The Under Secretary of Defense for Acquisition and Technology (the Under Secretary) first mandated program manager use of an open systems approach in his November 29, 1994, memorandum, “Acquisition of Weapons Systems Electronics Using Open Systems Specifications and Standards.” The memorandum required DoD Components to use open systems specifications (electrical, mechanical, thermal) for acquisition of weapon systems electronics to the greatest extent practical in an effort to reduce life-cycle cost and facilitate effective weapon system intra- and inter-operability. In DoD Regulation 5000.2-R, “Mandatory Procedures for Major Defense Acquisition Programs (MDAPS) and Major Automated Information Systems (MAIS) Acquisition Programs,” March 15, 1996, the Under Secretary expanded the requirement for use of an open systems approach in developing systems to all system elements (mechanical, electrical, and software). The Under Secretary provided additional mandatory policy to program office use of open systems design in Change 3, March 23, 1998, and Change 4, May 11, 1999, to DoD Regulation 5000.2-R. Change 3 added open systems to the essential elements that program managers must address in their acquisition strategies. Change 4 required program 5 managers to establish open systems objectives; to document their approach to specifying the level(s) of openness of systems, subsystems, and or components to be acquired; and to document the means for determining the extent of openness of systems, subsystems, and components. Recognizing the need for high-level management attention to program use of open systems, the Under Secretary of Defense for Acquisition, Technology, and Logistics issued the memorandum, “Open Systems Acquisition of Weapon Systems,” July 10, 1996, directing that DoD and Component acquisition executives ensure that program managers include open systems plans in the documentation provided to support acquisition milestone decisions. The memorandum further directed that the overarching integrated product teams supporting the acquisition executives include open systems planning as a specific item in their oversight and review discussions on acquisition programs. Appendix C provides more detailed information on DoD open systems policy as well as information on public law concerning the use of open systems. Defining Open Systems Objectives We reviewed acquisition planning documents for 17 major Defense acquisition programs that gained approval to begin program definition and risk reduction (Milestone I), or to enter engineering and manufacturing development (Milestone II), between March 1996 and July 1999. DoD and Component acquisition executives allowed 14 of the 17 major Defense acquisition programs to proceed to the next acquisition phase without requiring program managers to clearly define the open system objectives for the systems or the strategy for achieving the objectives. Specifically, users and program managers did not include language concerning the required use of an open systems design in acquisition planning documents. The following list of seven acquisition planning documents shows the number of programs that did not include language concerning the required use of open systems out of the number of programs that prepared the cited document: • operational requirements document (6 of 17 programs), • single acquisition management plan (2 of 12 programs), • acquisition plan (3 of 5 programs), • system engineering management plan (2 of 6 programs), • request for proposal (9 of 17 programs), • contract statement of work (8 of 15 programs), and • test and evaluation master plan (11 of 17 programs). Appendix E provides details on the inclusion of open system requirements in acquisition planning documents for the 17 acquisition programs reviewed and 6 indicates which programs were new weapon system acquisitions and which programs were upgrades to existing, or legacy, weapon systems. The Joint Task Force emphasized that the degree that program managers could implement an open systems design for legacy systems may be limited because of modifications to an existing weapon system design that may not be open. Including Open Systems Objectives in Acquisition Planning Acquisition planning documents serve as a roadmap to program managers and contractors for program execution from program initiation through postproduction support. Therefore, the DoD acquisition community, in coordination with the Joint Chiefs of Staff and supporting organizations involved in the weapons systems requirements generation process, must include open systems requirements in acquisition planning documents to maximize DoD effectiveness in implementing an open systems approach for developing and acquiring weapon systems. The following discusses the general purpose of the seven acquisition planning documents that the Joint Task Force agreed should include requirements for an open systems design approach and provides the results of our review and examples of how program managers documented the required use of an open systems approach. • Operational Requirements Document. The operational requirements document is a product of the requirements generation system. The operational requirements document is a formatted statement containing operational performance parameters for the proposed concept or system. Chairman of the Joint Chiefs of Staff Instruction 3170.01A, “Requirements Generation System,” August 10, 1999, requires that the operational requirements document contain the performance and related operational parameters as well as program affordability estimates a weapon system acquisition program must meet. The Instruction promotes program use of the open systems approach by having requirements and acquisition planners to include a description of the benefits of evolutionary acquisition in the operational requirements document to match projected threats. While users for 6 of the 17 programs did not include open systems related language in their operational requirements documents, the Operational Requirements Document for the National Missile Defense Program, March 10, 1997, clearly defined requirements for the program manager to use an open systems approach. Specifically, the operational requirements document required the program manager to comply with an established open systems architecture and to establish and enforce a coherent set of common open technical standards across missions and systems. Additionally, the operational requirements document stated that the program manager was to maximize the use of standard parts and components already in the military supply system. Further, the operational requirements document stated that the program manager must ensure standardization, interoperability, and commonality within the system to ensure interoperability with other systems and to reduce the cost of system ownership. 7 • Single Acquisition Management Plan. In his memorandum, “Reengineering the Acquisition Oversight and Review Process,” April 28, 1995, the Under Secretary allowed program mangers to consolidate milestone documentation in a single acquisition management plan. The single acquisition management plan provides a concise, integrated description of the overall acquisition and program management strategy and provides a vehicle for obtaining the statutory and regulatory approvals required for a program manger to implement the acquisition program. Although the Under Secretary did not mandate a specified format for the single acquisition management plan, the plan normally includes a program summary, a mission description, the acquisition strategy, the engineering and technical approach, and the program support strategy. Program managers who use a single acquisition management plan in place of separate acquisition documents for the acquisition plan and the engineering management plan, should include specific open systems language in the acquisition strategy and systems engineering sections of the plan to document that the contractor will be required to use an open systems acquisition and management approach. Program managers for 2 of the 12 programs did not include open systems related language in their single acquisition management plans. However, the Air Force Space Based Infrared System Single Acquisition Management Plan, October 1, 1996, strongly emphasized the Air Force’s commitment to open systems in the engineering and manufacturing design of the system. In the single acquisition management plan, the program manager stated that he would maintain a systems engineering process over the entire program to include use of an open systems concept. Further, the program manager stated in the plan that he would add incentives to an open system approach by encouraging the contractor to implement an architecture that defined internal system interfaces by using open standards that industry had adapted. • Acquisition Plan. The Defense Federal Acquisition Regulation Supplement, Part 207, “Acquisition Planning,” October 1, 1999, requires program managers to prepare written acquisition plans for weapon system acquisitions when contract costs are expected to total $5 million or more. In the acquisition plan, the Supplement requires program managers to include information on acquisition background, objectives, and a plan of action for the development effort that addresses product descriptions, logistics considerations, budget and funding, and other program considerations. The program manager provides the acquisition plan to the contract administration organization to facilitate resource allocation and planning for evaluating and managing contractor risk. Program manager inclusion of open systems objectives in the acquisition plan is essential for ensuring that assigned contract administration organizations evaluate contractor implementation of an open systems approach in the design of the weapon system. Program managers for 3 of the 5 programs did not include open systems related language in their acquisition plans. The program manager for the 8 United States Marine Corps H-1 Upgrade Program issued an acquisition plan on September 27, 1996, that clearly documented the program manager’s plans for using an open systems approach. In the acquisition plan, the program manager specified that the contractor would use an open systems approach that entailed contractor use of commercial interface standards to ensure form, fit, and function interchangeability, both internal and external, to the components that comprise the cockpit upgrade. Further, the program manager stated that the contractor through the system design effort must allow for future modifications and system integration flexibility to reduce the overall lifecycle cost of the helicopter cockpit with particular emphasis on reducing system operations and support costs. • Systems Engineering Management Plan. The Defense Acquisition Deskbook Critical Process Assessment Tool, “Systems Engineering,” August 14, 1998, defines the systems engineering management plan as the program plan for conducting systems engineering. The Deskbook states that the program office may need the systems engineering management plan to understand the contractors system engineering process well enough to maintain insight into its progress. The program manager’s inclusion of open systems objectives in this document is important because the contractor needs to focus on design flexibility and open interface selection to adapt to technology evolution and enable system upgrades over time. Program managers for 2 of the 6 programs did not include open systems related language in their system engineering management plans. The Navy Theater Wide Theater Ballistic Missile Program Systems Engineering Management Plan, March 22, 1999, did include open systems related language. In the plan, the program manager stated that the program office engineering staff would ensure that the contractor used an open systems architecture to the maximum extent possible to simplify the contractor making modifications to the subsystems and to enhance interoperability with other theater missile defense and theater missile defense support systems. • Request for Proposal. The Federal Acquisition Regulation, Subpart 15.203, “Requests for Proposal,” October 1, 1999, requires contracting officers for negotiated acquisitions to use requests for proposals to communicate Government requirements to perspective contractors and to solicit contractor proposals. In three sections of the request for proposal, contracting officers can give contractors information needed to develop contract proposals that will effectively implement an open systems approach. The three sections are Section L, “Instructions, Conditions, and Notices to Offerors or Quoters;” Section M, “Evaluation Factors for Award;” and the statement of objectives. Through these proposal sections, the contracting officer can advise prospective contract offerors that they will be required to develop a system using an open systems approach and that their proposal must address an open systems concept if they want to be considered as a responsive offeror to the request for proposal. Program managers for 9 of the 17 programs did not include open systems related language in the request for proposal. An example of a request for 9 proposal that included open systems related language is the engineering and manufacturing development contract for the Air Force Global Broadcast Satellite Program. The request for proposal gave offerors clear notice on instructions and evaluation factors pertaining to using an open systems approach in the design and development of the satellite.  Section L of the request for proposal requires that offerors identify contractor efforts to ensure that use of open standards and open architectures will be a driving factor in the contractor’s system design approach. Section L also requires offerors to describe system segment in terms of functions and interfaces and show how open systems architectures and standards apply to the architecture. Further, Section L requires offerors to describe plans for evolving the proposed system architecture to meet system interim, threshold, and objective performance capabilities.  Section M of the request for proposal identifies the factors that the contracting officer will consider in awarding the contract. Section M states that the contracting officer will consider the offeror’s use of an open systems design in the technical evaluation factor assessment used to make the decision to award the engineering and manufacturing development contract.  Offerors use the statement of objectives to develop their proposed contract statement of work that supports and defines their proposed efforts. The statement of objectives states that the contractor should maximize use of commercial equipment and software, open systems, and use of open nonproprietary network and communications protocols and standards. Additionally, the statement of objectives states that the contractor should design the system architecture to accommodate evolutionary hardware and software changes to achieve future system requirements. • Contract Statement of Work. The Federal Acquisition Regulation, Subparts 15.406-1, “Uniform Contract Format,” and 15.406-2, “Part 1 - The Schedule,” October 1, 1995, require Agency solicitations for contracts to include a statement of work, or other description that defines the Government’s requirements. Program manager inclusion of open systems objectives in this document is necessary to ensure that the contractor uses an open systems design approach in the system’s design. Program managers can also use provisions in the contract statement of work, along with the contract data requirements list, to require the contractor to provide the program manager with information to identify the extent of system design openness. Program managers for 8 of the 15 programs did not include open systems related language in their contract statements of work. A positive example is the Army’s statement of work in the contract for the Guided Multiple Launch Rocket System that defined the required level of system openness as subsystem interfaces. The statement of work also required 10 that the contractor describe subsystem interfaces in interface control documents and to list and justify all proprietary [closed] interfaces, including proprietary extensions to open standards. Further, the statement of work required that the contractor identify, at systems design reviews, all configuration items which, as a result of the proposed open systems architecture design, are amenable to efficient technology and or supplier insertion at any stage during system engineering and manufacturing development and system production. • Test and Evaluation Master Plan. DoD Regulation 5000.2-R requires that program managers prepare a test and evaluation master plan to provide a framework within which to generate detailed test and evaluation plans and identifies necessary developmental and operational test and evaluation activities for the weapon system. The program manager’s inclusion of open systems objectives in this document is needed to ensure that developmental and operational test organizations verify contractor insertion of an open systems design through planned testing of system components. Program managers for 11 of the 17 programs did not include open systems related language in their test and evaluation master plans, the Test and Evaluation Master Plan for the Army CH-47 Improved Cargo Helicopter, September 29, 1998, did include open systems related language. The plan stated that the contractor would select or develop all hardware and software for the helicopter cockpit within the concepts of an open systems design. Further, the plan established compatibility of system electronic architecture with the Joint Technical Architecture – Army as a critical technical parameter for the helicopter cockpit program. Accordingly, the plan specified that the contractor must successfully demonstrate compatibility with the Joint Technical Architecture – Army during production qualification tests. The Joint Technical Architecture – Army includes commercial standards to provide program managers with building codes for the development of all Army programs. Use of Open Systems Objectives in the Acquisition Planning and Review Process Program managers did not routinely include open systems design objectives in acquisition planning and review because the Defense and Component acquisition executives did not enforce the requirement that program managers use an open systems design approach in key acquisition documents as part of the acquisition milestone review process. Without acquisition executive enforcement, program offices did not always aggressively seek guidance and training in using an open systems design approach for their programs. With respect to training, the Joint Task Force had emphasized providing detailed guidance and training to designated pilot programs and to those program managers that sought advice, 11 while relying on the efforts of the acquisition executives and advisory guidance in the Defense Acquisition Deskbook, conferences, and a web site to provide guidance and training to the larger acquisition community. Acquisition Milestone Review Process. The Defense and Component acquisition executives had not implemented effective controls to enforce the requirement that program managers use an open systems design approach in key acquisition documents as part of the acquisition milestone review process. Because the DoD and Component overarching integrated product teams supporting the acquisition executives did not routinely include program manager implementation of open systems in their reviews, program managers did not always aggressively seek guidance and training in using an open systems approach. To determine if overarching integrated product teams were addressing the use of open systems during the oversight and review process, we reviewed team reports for 12 of the 17 major Defense acquisition programs included in our audit scope. In 11 of the 12 team reports, the teams did not mention a discussion with program managers concerning the use of an open systems design approach. To remedy the above condition, the Under Secretary needs to establish performance goals and metrics to measure program manager progress in inserting open systems design requirements in key acquisition documents in support of acquisition milestone reviews. The overarching integrated product teams are in the best position to measure program manager performance against established performance goals and metrics as part of the milestone review process. An open systems performance goals and metrics would also ensure that the overarching integrated product teams supporting the milestone decision authorities include open systems planning as a specific item in their oversight and review discussions on acquisition programs. The Joint Task Force indicated that it understands the need for performance goals and metrics relating to program manager compliance with open systems requirements. As to whether program managers addressed open systems in key acquisition planning documents, the Joint Task Force stated that any open systems performance metric would be binary in nature (yes or no). In addition to addressing open systems in the key seven acquisition planning documents previously discussed, the Joint Task Force stated that program manager completion of market research on the availability and affordability of commercial interface standards and system architecture studies would provide evidence that a program manager had used an open system approach. Guidance and Training for Pilot Programs and Programs Seeking Advice. The Joint Task Force has provided extensive guidance and training on implementing an open systems design approach to the Navy AV-8B Aircraft Open Systems Core Avionics and the Air Force’s F-15E Multipurpose Display Processor program offices. The Principle Deputy Under Secretary of Defense for Acquisition, Technology, and Logistics designated the two acquisition programs as pilot programs for implementing an open systems approach on February 15, 1996. Accordingly, the Joint Task Force worked directly with the pilot program staffs in: 12 • developing open systems strategies, • applying open systems concepts to their programs, and • providing funding to assist the programs offices in implementing an open systems approach. The Joint Task Force also provided advice and assistance on an open systems design approach to program office for another 26 acquisition programs through collaboration with program office integrated product teams. Guidance and Training for the General Acquisition Community. The Joint Task Force and DoD Components provided advisory guidance to the acquisition community on the use of an open systems design approach through the Defense Acquisition Deskbook. As of January 31, 2000, the Defense Acquisition Deskbook listed 304 documents referencing open systems. As discussed in Appendix D, the Joint Task Force has also developed seminars, tutorials, and a web site that program managers can access to obtain information on implementing an open systems design approach. While available guidance contains constructive information on implementing an open systems approach, the Joint Task Force acknowledged that it could provide additional clarifying guidance. Specifically, the Joint Task Force agreed that it could enhance the understanding and effectiveness of program managers in implementing an open systems strategy by providing guidance templates illustrating how program managers can address open systems design requirements in key acquisition planning documents and by providing open systems guidance tailored to each acquisition phase. Guidance Templates. The Office of the Director, Joint Staff, in coordination with the Joint Task Force, established a requirement for using an open systems approach in universal guidance templates used by the requirements community to prepare the operational requirements document. The Joint Task Force also developed templates of open systems information for four other key acquisition documents. The Joint Task Force provided the templates to program managers requesting assistance. Operational Requirements Document Template. The Joint Task Force provided the Director, Joint Staff, with suggested revisions on open systems information included in the template for the operational requirements document in Chairman of the Joint Chiefs of Staff Instruction 3170.01, “Requirements Generation System,” June 13, 1997. On August 10, 1999, the Director, Joint Staff, issued the revised Instruction, designated 3170.01A, which included changes in response to the Joint Task Force suggested revisions. In the revised template for the operational requirements document, the Director, Joint Staff, included the following points relevant to an open systems approach: • described the benefits of evolutionary acquisition for proposed system (if appropriate) in the general description of operational capability section, 13 • included interoperability as an example of a system performance parameter in the capabilities required section and as a support objective in the program support section, • specified that the system must comply with applicable information technology standards contained in the DoD Joint Technical Architecture, and • specified that the system cost figure should be stated in terms of a threshold and objective to provide flexibility to allow for program evolution in the program affordability section. The revised language in Instruction 3170.01A should increase the emphasis weapon systems users and Service Chiefs of Staff give to open systems when processing the operational requirements document. Also, it will help the acquisition community effectively flow an open systems requirements into the acquisition planning documents prepared based on the operational requirements document. Templates for Other Acquisition Planning Documents. On request, the Joint Task Force provided program office staffs with template examples showing open systems language that can be used in the single acquisition management plan, the systems engineering management plan, the request for proposal, and the contract statement of work. However, the Joint Task Force had not provided the broader acquisition community with the four document template examples or prepared template examples showing open systems language that can be used in preparing the acquisition plan and the test and evaluation master plan. The availability of templates of open systems language that program managers can include in the six key acquisition planning documents would greatly assist program offices in establishing appropriate open systems language in the acquisition planning documents. Also, the Joint Task Force’s development and inclusion of the document templates in the Defense Acquisition Deskbook would be a timely response to a General Accounting Office report citing the need for DoD to improve acquisition training. The General Accounting Office criticized DoD training related to acquisition reform initiatives in GAO/NSIAD-99-206, “Best Practices – DoD Training Can Do More to Help Weapon System Programs Implement Best Practices,” August 1999. The report stated that program officials reported that DoD training did not prepare them for implementing revised practices. The General Accounting Office stated that the DoD training typically provided only an awareness of the practices, not the knowledge needed for actual implementation. The template examples for the six key acquisition planning documents would provide program offices with a practical supplement to conceptual training on implementing an open systems design approach. 14 Assurance That Open Systems Objectives Were Met As a result, DoD acquisition decision makers did not have assurance at program milestone reviews that program managers required and stressed the importance of open systems design objectives to weapon system contractors. Unless Defense and Component acquisition executives fully emphasize the importance of maximizing program manger and contractor use of an open systems design approach from the inception of acquisition programs, DoD will not fully realize the long-term benefits of using an open systems design approach to develop and acquire weapon system. Recommendations, Management Comments, and Audit Response A.1. We recommend that the Under Secretary of Defense for Acquisition, Technology, and Logistics: a. Enforce the requirement for overarching integrated product teams within OSD and DoD components to assess open systems planning as a specific item for inclusion in the acquisition oversight and review process when preparing for system milestone reviews. b. Establish performance goals and metrics to measure program manager progress in inserting open systems design requirements in key acquisition planning documents. Management Comments. The Under Secretary of Defense for Acquisition, Technology, and Logistics did not comment on the recommendation. We request that the Under Secretary of Defense provide comments to the final report. A.2. We recommend that the Under Secretary of Defense for Acquisition, Technology, and Logistics and the Director, Operational Test and Evaluation revise DoD Regulation 5000.2-R, “Mandatory Procedures for Major Defense Acquisition Programs (MDAPs) and Major Automated Information Systems (MAIS) Acquisition Programs,” May 11, 1999, to require program managers to include open systems objectives in test and evaluation master plans to emphasize to developmental testers the need to verify the contractor’s use of an open system design approach. Under Secretary of Defense for Acquisition, Technology, and Logistics Comments. The Under Secretary of Defense for Acquisition, Technology, and Logistics did not comment on the recommendation. We request that the Under Secretary of Defense provide comments to the final report. 15 Director, Operational Test and Evaluation, Comments. The Director, Operational Test and Evaluation, concurred, stating that he would support the Under Secretary of Defense for Acquisition, Technology, and Logistics in making the recommended revision to DoD Regulation 5000.2-R. Audit Response. The Director, Operational Test and Evaluation, comments were responsive to the recommendation. We request that the Under Secretary coordinate with the Director, Operational Test and Evaluation, and provide comments to the final report. A.3. We recommend that the Director, Open Systems Joint Task Force, include in the Defense Acquisition Deskbook suggested general template language relating to program manager implementation of an open systems acquisition strategy in the: a. single acquisition management plan, b. acquisition plan, c. systems engineering management plan, d. request for proposal, e. contract statement of work, and f. test and evaluation master plan. Director, Open Systems Joint Task Force Comments. The Director, Open Systems Joint Task Force, did not comment on the recommendation. We request that the Director provide comments to the final report. Assistant Secretary of the Army (Acquisition, Logistics, and Technology) Comments. Although not required to comment, the Acting Deputy Assistant Secretary for Plans, Programs, and Policy, Office of the Assistant Secretary of the Army (Acquisition, Logistics, and Technology), agreed with the recommendations in the draft report. Further, the Acting Deputy Assistant Secretary stated that the availability of general template language will benefit personnel who have the task of drafting the acquisition documents discussed in the draft report. Army Program Executive Office for Ground Combat and Support Systems Comments. Although not required to comment, the Program Executive Officer stated that he agreed with the draft report. 16 B. Determining the Extent of System Design Openness Detailed review of program documentation at 4 of the 17 major Defense acquisition program offices showed that program managers for 3 of the 4 programs did not document a means for determining the extent of design openness of systems, subsystems, and components. Additionally, guidance on open systems did not require program managers to assess the impact of their planned level of design openness on the long-term viability and affordability of systems. These conditions occurred because the Joint Task Force did not: • provide program managers, in general, with guidance on how to document the means for determining the extent of system design openness; and • establish acquisition policy to recognize that determining the level of openness of system design is most meaningful when combined with program impact assessments. Without a means to measure the progress and the impact of implementing an open systems approach, acquisition decision makers can not readily gauge how well program managers are achieving the advantages of using an open system design approach and assess the susceptibility of a weapon system design to obsolescence or costly upgrade to counter foreign military threats. Policy for Determining the Extent of System Design Openness The Under Secretary of Defense for Acquisition, Technology, and Logistics has recognized the need to determine the extent of openness that program managers achieve in weapon systems design. In Change 3 to DoD Regulation 5000.2-R, March 1998, the Under Secretary established the requirement that program managers document their approach for measuring the level of openness of systems, subsystems, and components to be acquired and devise an open systems strategy to achieve these requirements. In Change 4 to DoD Regulation 5000.2-R, May 1999, the Under Secretary modified the requirement for measuring openness to require that program managers document their means for determining the extent of openness of system, subsystems, and components assuring conformance to open standards and at the specified levels of openness established in their open system objectives. 17 Documenting the Means for Determining the Extent of Openness in Weapons Systems Design Program managers for 3 of the 4 program offices reviewed (the Army Guided Multiple Launch Rocket System, the Navy and Marine Corps UH-1 Helicopter Upgrade, and the Air Force Global Broadcast Service) did not document a means for determining the extent of design openness in systems, subsystems, and components. The program manager for the fourth program office (the Navy Tactical Tomahawk) requested that the prime contractor provide a percentage measurement of the level of design openness for one of the three segments of the program. Guided Multiple Launch Rocket System. Integrated product teams for the Guided Multiple Rocket System were unsure of how to document the means for determining the extent of systems, subsystems, and components design openness. The contract statement of work did require the contractor to provide the program office with information on subsystem interfaces that could provide a basis for determining the extent of an open system design. The contract statement of work established interface control documents as the means for the contractor to define subsystems interfaces and required the contractor to identify subsystem interfaces that were proprietary in nature. However, the contractor had not yet provided the program office with any interface control documents. The integrated product teams stated that the contractor would provide most interface control documents for approval during the time period after the preliminary design review in July 1999 and before the critical design review scheduled for August 2000. H-1 Helicopter Upgrade. The program office for the H-1 Helicopter Upgrade also was unsure of how to document the means for determining systems, subsystems, and components design openness. The program office stated that the concept of open systems was new and that not enough guidance was available for them to fully implement open systems requirements. The program office did implement an open systems acquisition strategy for the avionics systems but not for the propulsion system portion of the upgrade. The program office stated that it would be difficult to determine the extent of openness in the avionics design because the contractor was awarded a streamlined performance- based contract. The contract did not require the contractor to provide visibility over the interface control documents. The program office added that their integrated product teams would have some insight into the avionics system design, but the information would be insufficient to make a periodic determination on the extent of openness of the avionics system design. Accordingly, the program office stated that it would not be able to readily determine the extent of avionics system design openness although it believed that its acquisition strategy would encourage the contractor to use an open systems design approach. The program office cited two factors that would encourage the contractor to use an open systems approach: the contractor, as part of the acquisition strategy, will provide operational support for the first 10 years after the helicopter upgrade; and the contract statement of work required the 18 contractor to provide for an easy replacement of components as part of the system reliability requirement for mean-time-between-failure. While the acquisition strategy for the upgrade program should result in a system with some degree of openness, the program office has no assurance that the contractor will develop systems with the level of openness that the Government desires. Because the contractor may become concerned about continued responsibility for maintaining the system after the 10-year period, the contractor may not have adequate incentive to use open systems interfaces. Consequently, program offices should not rely on operational requirements and contracting strategies alone to achieve an open systems approach and should take contractual action to ensure that the program office can make periodic assessments of the level of openness of the system that contractors include in system designs. Tactical Tomahawk. The program office for the Tactical Tomahawk requested its prime contractor, Raytheon-Hughes, Tucson, Arizona, to provide a percentage determination of the level of design openness for one of the three segments of the program: the Tactical Tomahawk missile, the weapons control system, and the mission planning system. The prime contractor provided the program office with a percentage on the level of design openness for the missile segment through assessing the number of common and commercial interfaces as well as software transportability and modularity. The contractor determined that 240 of the 258 electrical subsystem interfaces (93 percent) in the missile segment were open and that the remaining 18 subsystem interfaces were proprietary or closed. While the prime contractor provided some insight into the level of openness for the missile segment, the contractor did not address software and mechanical interfaces. Since July 1999, the program office has worked with the missile segment contractor to modify that contract with language that will address open systems requirements for electrical as well as software and mechanical interfaces. Global Broadcast System. The program office for the Global Broadcast System was unsure of how to implement the requirement for determining the extent of system subsystem, and component design openness. As program managers encourage contractors to implement open systems design in system interfaces through required program documentation, the program managers and contractors can use this documentation to determine the extent of openness at all levels, including between systems, subsystems, and components. Providing Open Systems Regulation and Guidance for Acquisition Program Managers In addition to program managers not documenting a means for determining the extent of system design openness, DoD guidance did not require program managers to assess the impact of planned system design openness on long-term viability and affordability of systems. These conditions occurred because the Joint Task Force did not: 19 • provide program managers, in general, with guidance on how to document the means for determining the extent of system design openness; or • establish acquisition policy to recognize that determining the level of openness of system design is most meaningful when combined with program impact assessments. Further, while executing systems engineering processes, three of the four program offices visited did not address the extent of system design openness in program design reviews. Documenting the Means for Determining the Extent of Design Openness. The Joint Task Force did not provide program managers, in general, with guidance on how to document the means for determining the extent of system design openness. While the Joint Task Force had formulated draft guidance for program managers to use in determining the extent of system design openness in June 1998, it had not finalized the guidance. The Joint Task Force stated that it had not finalized the guidance because it was still validating the suggested methods for determining the extent of system design openness. In addition, the Joint Task Force stated that it was not convinced that documenting a means for determining the level of system design openness added value unless the program managers also assessed the impact of the planned level of openness on future system long-term viability and affordability. The Joint Task Force acknowledged that it needed to issue guidance in the Defense Acquisition Deskbook for the program managers to use in determining the extent of system design openness. To be helpful, the guidance needs to tell program managers how to structure contract provisions to provide the program office with visibility and influence over system design openness. Program offices for the four programs stated that the prime contractor would be the best source for obtaining a measurement of system design openness but that their development contracts did not require the contractors to provide this information. Also, the program offices stated that performance-based development contracts offered little to no visibility into the contractor’s system design configuration during the development phase of the system acquisition process. Assessing the Impact of Planned Level of System Openness. The Joint Task Force stated that it was not convinced that determining the extent of system design openness added value unless program managers also assessed the impact of a given level of system openness on future system long-term viability and affordability. Some system, subsystem, or component interfaces, particularly those involving rapidly changing electronics, communications, or computer technologies can be far more critical to a system’s continued viability than slower changing system interfaces. For example, program managers for two different systems could each project that 75 percent of their system’s interfaces will be open, but the systems could have greatly different future long-term viability and affordability depending on how fast technology or requirements were expected to change for the remaining 25 percent of the systems’ interfaces 20 that were closed. The Joint Task Force indicated that program managers could provide acquisition decision makers with a much more meaningful program assessment if they provided an assessment of the probable impact of a planned level of systems openness on future system long-term viability and affordability along with a determination of the extent of system design openness. Addressing Open Systems Requirements as Part of the Systems Engineering Process. The Joint Task Force stated that open systems guidance should emphasize implementing open systems design as part of the systems engineering process. Specifically, the Joint Task Force stated that the design reviews such as the preliminary and critical design reviews, which program management and contractor staffs perform periodically during the systems engineering process would provide a forum where the program manager can address whether an open systems approach is being successfully applied. These two design reviews, as described in Military Standard 1521-B, “Technical Reviews and Audits for Systems, Equipment, and Computer Software,” June 4, 1985, provide an appropriate forum from which to review system subsystems and components interfaces. The preliminary design review is a formal technical review of the basic design approach for configuration items. The critical design review follows the preliminary design review and verifies whether detail design solutions have been achieved. Although program manager use of Military Standard 1521-B is no longer mandatory, DoD Regulation 5000.2-R emphasizes the importance of a structured design review process to demonstrate and confirm completion of required design accomplishments. Therefore, in order for program managers to determine the extent of system openness, they need to use whatever design reviews they have required and planned to make the determination. For the four program offices reviewed in detail, only one program manager addressed open systems in a design review. For that system, the Marine Corps H-1 Helicopter, the contractor addressed open systems during its preliminary design review. The program manager indicated that the contractor would also address open systems during the critical design review. If other program managers addressed open systems requirements during their structured design review processes, it could help the program managers emphasize, identify, and achieve a higher degree of design openness in the system development effort. Benefits of Determining the Extent and Impact of System Design Openness Without documented means for program managers and contractors to measure progress and the impact of implementing an open systems approach, acquisition decision makers can not readily gauge how well program managers are achieving the advantages of using an open system design approach. When acquisition decision makers assess a system’s readiness for production, without considering the level of openness of a system, they cannot fully consider: 21 • system life-cycle cost, which includes technology insertion; • whether a system can accommodate economical technology insertion; and • system affordability to keep pace with changing technologies. Given the long development cycle for most major DoD acquisition programs, systems, subsystems and components can become obsolete before systems reach production. DoD efforts to keep pace with technology may become cost prohibitive for some future systems if an open systems approach is not used in the design and could result in a decreased defense capability. Further, program managers having knowledge of obsolescence risk throughout the acquisition of a system, can effectively plan for cost-effective systems upgrades throughout the system’s life. Recommendations B.1. We recommend that the Under Secretary of Defense for Acquisition, Technology, and Logistics revise DoD Regulation 5000.2-R, “Mandatory Procedures for Major Defense Acquisition Programs (MDAPs) and Major Automated Information Systems (MAIS) Acquisition Programs,” to require program managers to assess the impact of projected system design openness, including the readiness and cost impacts of any critical closed proprietary interfaces, as part of the acquisition strategy used to support acquisition milestone reviews. B.2. We recommend that the Director, Open Systems Joint Task Force, update the Defense Acquisition Deskbook to include guidance to help program manager document the means for determining the extent of system openness, to include possible performance measures for gauging progress in implementing an open systems design approach as well as examples of possible contract provisions and strategies that can be used to ensure that the program offices acquire the information needed from contractors to measure the extent of system design openness. Management Comments Required The Under Secretary of Defense for Acquisition, Technology, and Logistics and his Director, Open Systems Joint Task Force did not comment on a draft of this report. We request that the Under Secretary of Defense and the Director, Open Systems Joint Task Force provide comments to the final report. 22 Appendix A. Audit Process Scope We conducted the audit from April 1999 through March 2000 and reviewed documentation dated from November 1994 through January 2000 at the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, DoD Component Headquarters, and obtained documentation from 17 major Defense acquisition program offices. Specifically, we examined operational requirements documents, single acquisition management plans, acquisition plans, requests for proposals, contract statements of work, systems engineering management plans, and test and evaluation master plans. Also, we visited 4 of the 17 major Defense acquisition program offices to determine whether the program offices had documented a means for determining the extent of design openness of systems, subsystems, and components. DoD-wide Corporate Level Government Performance and Results Act (GPRA) Coverage. In response to the GPRA, the Secretary of Defense annually establishes DoD-wide corporate level goals, subordinate performance goals, and performance measures. This report pertains to achievement of the following goal, subordinate performance goal, and performance measures: • FY 2000 DoD Corporate Level Goal 2: Prepare now for an uncertain future by pursuing a focused modernization effort that maintains U.S. qualitative superiority in key warfighting capabilities. Transform the force by exploiting the Revolution in Military Affairs, and reengineer the Department to achieve a 21st century infrastructure. (00-DoD-2) • FY2000 Subordinate Performance Goal 2.4: Meet combat force’s needs smarter and faster, with products and services that work better and cost less, by improving the efficiency of DoD’s acquisition processes. (00-DoD-2.4) • FY 2000 Performance Measure 2.4.1: Major Defense Acquisition Program (MDAP) Cost Growth (In percents). (00-DoD-2.4.1) • FY 2000 Performance Measure 2.4.2: MDAP Cycle Time. (00-DoD-2.4.2.) DoD Functional Area Reform Goals. Most major DoD functional areas have also established performance improvement reform objectives and goals. This report pertains to achievement of the following acquisition functional issue area objective and goal. Objective: Delivering Great Service. Goal: Deliver new major Defense systems to the users in 25 percent less time. (ACQ-1.1) 23 General Accounting Office High-Risk Area. The General Accounting Office has identified several high-risk areas in the Department of Defense. This report provides coverage of the Defense weapons system acquisition high-risk area. Methodology To evaluate program manager consideration and use of open systems in developing and acquiring weapon systems, we evaluated OSD and Military Department policies and procedures. We also examined program manager planning and execution of an open systems approach for their programs as well as the adequacy of DoD acquisition decision-maker reviews of program manager implementation of an open systems approach. We received technical assistance in examining program manager planning and execution of an open systems approach from electronics engineers in the Technical Assessment Division of the Office of the Assistant Inspector General for Audit. Auditing Standards. We conducted this program audit in accordance with auditing standards issued by the Comptroller General of the United States, as implemented by the Inspector General, DoD, and accordingly included such tests of management controls as we deemed necessary. Use of Computer-Processed Data. We did not rely on computer-processed data to perform this audit. Contacts During the Audit. We visited or contacted individuals and organizations within the DoD and at Defense contractors. Further details are available on request. Management Control Program DoD Directive 5010.38, “Management Control (MC) Program,” August 26, 1996, requires DoD managers to implement a comprehensive system of management controls that provides reasonable assurance that programs are operating as intended and to evaluate the adequacy of the controls. Scope of Review of Management Control Program. In accordance with DoD Directive 5000.1, “Defense Acquisition,” March 15, 1996, and DoD Regulation 5000.2-R, acquisition managers are to use program cost, schedule, and performance parameters as control objectives to implement the requirements of DoD Directive 5010.38. Also, the Under Secretary of Defense for Acquisition, Technology, and Logistics issued the memorandum “Open Systems Acquisition of Weapon Systems,” July 10, 1996, directing that acquisition milestone decision authorities ensure that program managers include open systems plans in the documentation provided to support acquisition milestone decisions and that OSD overarching integrated product teams include open systems planning as a specific item in their oversight and review discussions on acquisition programs. Accordingly, we limited our review to management 24 controls directly related to program manager consideration and use of open systems in developing and acquiring weapon systems and to overarching integrated product teams including open systems planning as a specific item in their oversight and review discussions on acquisition programs. Adequacy of Management Controls. We identified a material management control weakness in the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, as defined by DoD Instruction 5010.40, “Management Control (MC) Program Procedures,” August 26, 1996. The Office of the Under Secretary had not implemented effective controls to ensure that acquisition milestone decision authorities within OSD and the DoD Components enforced the Under Secretary’s direction for program managers to include open systems requirements in key acquisition planning documents as part of the acquisition milestone review process (finding A). Recommendations A.1. and A.2., if implemented, will correct the material management control weakness. A copy of the report will be provided to the senior official responsible for management controls in the Office of the Secretary of Defense. Adequacy of Management’s Self Evaluation. The Director, Open Systems Joint Task Force conducted a management control review that examined the adequacy of management controls to manage and oversee the use and expenditure of fiscal, personnel, and physical resources assigned to the Joint Task Force. The material management control weakness we identified occurred within the larger organizations of the DoD and Component acquisition executives, and was, thus, outside the scope of the self-evaluation the Director, Open Systems Joint Task Force performed. Summary of Prior Coverage During the last 5 years, the General Accounting Office issued one report relating to program use of the open systems approach. General Accounting Office, National Security and International Affairs Division, Report 99-101, “Ballistic Missile Defense, More Common Standards and Components Could Result in Cost Savings,” May 21, 1999. 25 Appendix B. Definitions of Open Systems Terms The DoD Open Systems Joint Task Force provided the following definitions that are germane to understanding the implementation of an open systems approach. Architecture. Architecture is the organizational structure of a system or component, the relationships, principles and guidelines governing design and evolution over time. Commercial Item. A commercial item is any item other than real property that is of a type customarily used for nongovernmental purposes and that has been sold to the general public or offered for sale to the general public. Closed Interfaces. Closed interfaces are privately controlled system and subsystem boundary descriptions for interfaces that are not disclosed to the public or are unique to a single supplier. Interface Standard. An interface standard specifies the physical or functional interface characteristics of systems, subsystems, equipment, assemblies, components, items or parts to permit interchangeability, interconnection, interoperability, compatibility, or communications. Interoperability. Interoperability is the ability of two or more systems or components to exchange data and use information. Joint Technical Architecture. The Joint Technical Architecture defines the DoD minimum set of rules governing the arrangement, interaction, and interdependence of the parts or elements, whose purpose is to ensure that systems conform to a specific set of requirements. It identifies system services, interfaces, standards, and the relationships. Level of Openness. The level of openness is the system, subsystem, or component level at which the interfaces conform to open standards. The contractor or supplier may control design, interfaces, repair, and implementation below the level of openness. The level of openness will affect the overall performance, life-cycle costs, long-term supportability, acquisition cycle time, interoperability, intra-operability, ease of technology insertion, and the extent of organic repair of a system. Modular. Modular is the design concept in which interchangeable units are used to create a functional end product. Module. A module is an interchangeable item that contains components. In computer programming, a program unit that is discrete and identifiable with respect to compiling, combining with other modules and loading is called a module. 26 Nondevelopmental Item. A nondevelopmental item is any previously developed item of supply used exclusively for governmental purposes by a Federal agency, a State or local government, or a foreign government with which the United States has a mutual defense cooperation agreement. Open Specifications. Open specifications are public specifications maintained by an open, public consensus process to accommodate new technologies over time and consistent with international standards. Open Standards. Open standards are widely accepted and supported standards set by recognized standards organizations or the commercial market place. Open standards support interoperability, portability, and scalability and are equally available to the general public at no cost or with a moderate license fee. Open System. An open system is a system that implements sufficient open standards for interfaces, services, and supporting formats to enable properly engineered components to be used across a wide range of systems with minimal changes, to interoperate with other components on local and remote systems, and to interact with users in a style that facilitates portability. An open system is characterized by the following: • well defined, widely used, preferably nonproprietary interfaces and protocols; • use of standards which are developed and adopted by recognized standards bodies or the commercial market place; • definition of all aspects of system interfaces to facilitate new or additional systems capabilities for a wide range of applications; and • explicit provision for expansion or upgrading through the incorporation of additional or higher performance elements with minimal impact on the system. Open Systems Approach. An open systems approach is an integrated business and technical strategy to choose commercially supported specifications and standards for selected system interfaces (external, internal, functional, and physical), products, practices, and tools, and build systems based on modular hardware and software design. Program selection of commercial specifications and standards is based on: • those standards that industry standards bodies have adapted or are industry de facto standards (those successful in the market place); • market research that evaluates the short and long-term availability of products; • a disciplined systems engineering process that examines tradeoffs of performance; 27 • supportability and upgrade potential within defined cost constraint; and • allowance for continued access to technological innovation supported by many customers and a broad industrial base. Open Systems Architecture. An open systems architecture is a system architecture produced by an open systems approach and using open systems specifications and standards to an appropriate level. Open Systems Strategy. An open systems strategy focuses on fielding superior warfighting capability more quickly and more affordably by using multiple suppliers and commercially supported practices, products, specifications, and standards, which are selected based on performance, cost, industry acceptance, long-term availability and supportability, and upgrade potential. Portability. Portability is the ease with which a system, component, data, or user can be transferred from one hardware or software environment to another. Proprietary Specifications. Proprietary specifications are exclusively owned by a private individual or corporation under a trademark or patent, the use of which would require a license. Scalability. Scalability is the capability to adapt hardware or software to accommodate changing workloads. Specification. A specification is a document that prescribes, in a complete, precise, verifiable manner, the requirements, design, behavior, or characteristics of a system or system component. Standard. A standard is a document that establishes uniform engineering and technical requirements for processes, procedures, practices, and methods. Standards may also establish requirements for selection, application, and design criteria of material. System Architecture. A system architecture is a description, including graphics, of systems and interconnections providing for or supporting warfighting functions. The system architecture defines the physical connection, location, and identification of the key nodes, circuits, networks, and warfighting platforms and specifies system and component performance parameters. It is constructed to satisfy operational architecture requirements per standards defined in the Joint Technical Architecture. The system architecture shows how multiple systems within a subject area link and interoperate, and may describe the internal construction or operations of particular systems within the architecture. 28 Appendix C. Public Law and Government Policy Public law and implementing Federal and Government acquisition policies mandate that program managers use an open systems approach in the weapon system acquisition process. Public Law Section 12(d) of Public Law 104-113, “National Technology Transfer and Advancement Act of 1995,” March 7, 1996, requires that all Federal agencies and departments use technical standards that are developed or adopted by voluntary consensus standards bodies, using such technical standards as a means to carry out policy objectives or activities. Government Policy Office of Management and Budget. The Office of Management and Budget issued Circular A-119, “Federal Participation in the Development and Use of Voluntary Consensus Standards and in Conformity Assessment Activities,” February 10, 1998. This Circular directs agencies to use voluntary consensus standards in lieu of government-unique standards except where inconsistent with law or otherwise impractical. The policies in the Circular are intended to reduce agency reliance on government-unique standards. Department of Defense. The Under Secretary of Defense for Acquisition and Technology mandated program manager use of an open systems approach in his November 29, 1994 memorandum, “Acquisition of Weapons Systems Electronics Using Open Systems Specifications and Standards.” The memorandum required acquisition program mangers to use open systems specifications (electrical, mechanical, thermal) for acquisition of weapon systems electronics to the greatest extent practical. The memorandum further stated that acquisition program managers should design, develop, and construct systems and subsystems as open systems during the acquisition and modification process to reduce life-cycle cost and facilitate effective weapon system intra and interoperability. On March 15, 1996, the Under Secretary expanded an open systems requirement to include all system elements (mechanical, electrical, and software) in developing weapon systems. 29 Appendix D. Open Systems Joint Task Force Education and Outreach Initiatives Since the Under Secretary of Defense for Acquisition, Technology, and Logistics chartered the Joint Task Force in November 1994, it has made substantial efforts to promote program manager use of an open systems approach in weapon system acquisition. The Joint Task Force’s efforts have emphasized education and outreach to the acquisition community. Education The Joint Task Force developed educational products on the use of open systems and made these products available to the DoD acquisition workforce. The Joint Task Force educational products include a 3-hour computer-based open systems tutorial course available on CD-ROM, an open system implementation guide, a weapon system case study, an open systems engineering tutorial, and various papers, articles, and brochures. To broaden usage of the educational products, the Joint Task Force has made them available on an open systems web site. Additionally, the Joint Task Force provided the Defense Acquisition University input to update acquisition courses to better cover program manager use of an open system approach in developing and acquiring weapon systems. Specifically, the Joint Task Force collaborated with the Defense Acquisition University to modify segments of the following courses to include open systems concepts and principles: • Acquisition Program Management Course Open Systems Program Management Elective • Acquisition Program Management Open Systems Engineering Elective • Communications 201and 301 • Contracting 301 Seminar • Executive Program Management Course • Software Acquisition Management 201and 301 • Systems Acquisition for Contracting Personnel • Systems Acquisition Management 101 and 201 • Systems Engineering 201 and 301 • Test and Evaluation 301 Outreach to the Acquisition Community Open systems Joint Task Force outreach efforts have included performing assessments of program manager implementation of an open systems approach in weapon systems acquisition, participating in acquisition program integrated product teams, providing funding to open system projects within the DoD components, participating in conferences, providing briefings to industry and professional groups, and providing acquisition planning document templates and 30 lessons learned regarding program manager implementation of the open system approach to program managers requesting assistance. Details of these efforts include: • Acquisition Program Assessments. In January 1996, the Joint Task Force examined the DoD acquisition programs to ensure that program managers were effectively implementing the directive on program manager use of an open systems approach in developing and acquiring weapon systems. Since January 1996, the Joint Task Force had performed and reported on assessments of open system implementation for five major Defense acquisition programs. An open systems Joint Task Force had also assessed program implementation of open systems for in five additional programs but did not prepare formal reports of its assessments. • Participation on Integrated Products Teams. The Joint Task Force has participated in more than 25 integrated product teams to enhance program manager use of open systems in weapon systems development. In FY 1999, the Joint Task Force participated on integrated product teams for the C-130 Upgrade, C-17 aircraft, Crusader, F-22, and Surface Combatant for 21st Century. • Funding Open Systems Projects. In February 1996, the Joint Task Force provided funding to the two projects: the Navy’s AV-8B Aircraft Open Systems Core Avionics and the Air Force’s F-15E Multipurpose Display Processor that the Principal Deputy Under Secretary of Defense for Acquisition and Technology designated as pilot programs for implementing an open systems approach within DoD. The Joint Task Force also provided funding to assist the program managers for three additional programs to develop an open systems approach. • Conferences with Industry and Professional Groups. The Joint Task Force has participated in 21 conferences with industry and professional groups. The Joint Task Force participation included presenting papers on open systems architecture and setting up booths to distribute information on the open system initiative. Additionally, the Joint Task Force held numerous briefings with industry and professional societies on current DoD efforts to implement open systems and directed dialog with industry to encourage technical discussions on how program mangers use of open systems effects business opportunities with the DoD. • Templates and Lessons Learned. The Joint Task Force provided acquisition planning document templates to selected program offices to help acquisition staffs with the appropriate inclusion of open systems language in single acquisition management plans and system engineering management plans. Additionally, the Joint Task Force provided the DoD acquisition community with lessons learned on program implementation of an open systems approach. The lessons learned included case studies fully analyzing and documenting examples of open system technical and business strategies that acquisition program managers have used. The Joint Task Force also uses the lessons learned in preparing educational offerings. 33 1Legacy acquisition program. 2New acquisition program. 3Totals include only those programs that were required to prepare each acquisition planning document. 4The weapon system user had not yet completed the operational requirements document but had defined system requirements in the joint initial requirements document. 5DoD acquisition regulations do not require the program office to prepare a system engineering management plan. The program office used a modified integrated program summary to plan systems engineering for the program. 6The program office prepared a system engineering management plan in addition to a single acquisition management plan. 7Not Applicable (N/A) to sole source contracts. Sections L and M of the request for proposal provide instructions and evaluation criteria for other than sole source contracting actions. 8The program office did not provide a statement of objectives but used a statement of work to describe the work it needed the contractor to perform. 9Not Included (N/I) in chart because the contract statement of work contains contractor proprietary and business sensitive data. 10The program office did not yet have an approved test and evaluation master plan but used an interim test and evaluation master plan to plan test and evaluation for the program. 11The program office had not yet completed the initial draft version of the systems engineering management plan. 12N/A. The program manager did not prepare request for proposal because the contractor submitted an unsolicited proposal. Acronyms: ABL Airborne Laser AIM 9-X Short Range Air-to-Air Missile B-1 B-1 Conventional Munitions Upgrade Program (Defense System Upgrade Program) CH-60S Vertical Replacement Program GBS Global Broadcast Service ICH (CH-47F) Improved Cargo Helicopter JASSM Joint Air-to-Surface Standoff Missile JSF Joint Strike Fighter MLRS Multiple Launch Rocket System NATBMD Navy Area Theater Ballistic Missile Defense NMD National Missile Defense NPOESS National Polar-orbiting Operational Environment Satellite System NTW Navy Theater Ballistic Missile Defense SBIRS- High Space Based Infrared System Program High Component SBIRS-Low Space Based Infrared System Program Low Component USMC H-1 United States Marine Corps Helicopter 34 Appendix F. Report Distribution Office of the Secretary of Defense Under Secretary of Defense for Acquisition, Technology, and Logistics Deputy Under Secretary of Defense (Acquisition Reform) Director, Defense Procurement Director, Defense Logistics Studies Information Exchange Under Secretary of Defense (Comptroller) Director, Operational Test and Evaluation Joint Staff Director, Joint Staff Department of the Army Auditor General, Department of the Army Commander, U.S. Army Aviation and Missile Command Department of the Navy Naval Inspector General Auditor General, Department of the Navy Commander, Naval Air Systems Command Department of the Air Force Assistant Secretary of the Air Force (Financial Management and Comptroller) Auditor General, Department of the Air Force Commander, Space and Missile Command Other Defense Organizations Director, Defense Logistics Agency Defense Systems Management College 35 Non-Defense Federal Organizations and Individuals Office of Management and Budget General Accounting Office National Security and International Affairs Division Technical Information Center Congressional Committees and Subcommittees, Chairman and Ranking Minority Member Senate Committee on Appropriations Senate Subcommittee on Defense, Committee on Appropriations Senate Committee on Armed Services Senate Committee on Governmental Affairs House Committee on Appropriations House Subcommittee on Defense, Committee on Appropriations House Committee on Armed Services House Committee on Government Reform House Subcommittee on Government Management, Information, and Technology, Committee on Government Reform House Subcommittee on National Security, Veterans Affairs, and International Relations, Committee on Government Reform Director, Operational Test and Evaluation, Comments 37 Assistant Secretary of the Army (Acquisition, Logistics, and Technology) Comments 38 Army Program Executive Office for Ground Combat and Support Systems Comments 39 40 41 Audit Team Members The Acquisition Management Directorate, Office of the Assistant Inspector General for Auditing, DoD, prepared this report. Thomas F. Gimble Mary Lu Ugone John E. Meling Harold C. James Patrick E. Mchale Rodney D. Britt Donald E. Pierro Addie B. Frundt Renee L. Gaskin Shaun B. Jeffery Jaime A. Bobbio Wei Chang Cynthia B. Stull Bernice M. Lewis