- Describe the categories of simulations
- Describe what a systems integration laboratory (SIL) does
- Explain the need for verification and validation of models and simulations used in Test and Evaluation (T&E)
- Describe live, virtual, and constructive (LVC) simulations and how they can be brought together for a distributed test event
A simulation that involves simulated people operating simulated systems.1
Originally referred to a computer network in which at least some of the processing was done by separate workstations and information was shared by, and often stored at, the individual workstations. The term is used today in a much wider sense, even referring to autonomous processes that run on the same physical computer and interact with each other by message passing. In the context of this lesson, the term is not limited to computers, but also refers to a network where individual systems and capabilities are physically separated across some geographical area and are connected to provide an environment needed to test a product. These systems can be individual components that are linked to simulate a "whole" system, or can be capabilities such as simulated adverse weather or terrain linked to the cockpit of a flight simulator. Thus, using Modeling and Simulation, a distributed environment can be developed from geographically separated systems providing all the components and capabilities needed to conduct a realistic test of a product.
A simulation that combines constructive, live, and/or virtual simulations, typically in a distributed environment. Such simulations typically combine simulators with actual operational equipment, prototypes of future systems, and realistic representations of operational environments.2
A simulation involving real people operating real systems.3
A simulation involving real people operating simulated systems.4
Middleware is the data exchange software used by laboratories and simulations to send and receive data. It provides a common functionality (data distribution, filtering, etc.) to exchange data/information between systems. Middleware provides a means of assuring that test sites with different data formats, structures, sampling rates, and so forth will be able to communicate meaningfully with each other.
Wide-Area Network (WAN)
A communications network designed for large geographic areas.5 A WAN is used to connect various systems such as cell phones, computers, and other devices.
Check Your Understanding
After gaining a basic understanding of the test and evaluation (T&E) process in the previous lesson, the student will now need to apply that knowledge to the application of the various categories of models and simulations to the T&E process. As shown in the "Introduction to Modeling and Simulation" chapter, there are a wide range of models and simulations, and not all will be applicable to T&E.
This lesson will identify the categories of simulations appropriate to Test and Evaluation (T&E). The lesson will then examine some basic requirements that must be met before models and simulations can be used for testing and describe some direct applications, such as hardware-in-the-loop (HWIL) and systems integration laboratories (SIL). The lesson will then address how these applications can be connected to form a distributed environment. A distributed environment is a crucial element in using live, virtual, and constructive (LVC) modeling and simulations to support T&E. In subsequent lessons, we will show how this distributed environment applies directly to making improvements in T&E capabilities.
How Does Modeling and Simulation Apply to Test and Evaluation?
Modeling and Simulation (M&S) is used to develop data as a basis for making managerial or technical decisions. However, it is essential to apply M&S appropriately to achieve an effective and efficient Test and Evaluation (T&E) program. M&S capabilities and limitations are often inadequately understood, and M&S is sometimes planned for with insufficient attention to detail. A M&S capability in T&E involves not just the software tools themselves, but the data that feed them; the computing platforms that execute them; the standards, middleware, and networks that may interconnect them; the encryption capabilities and security constraints that protect them; and, most importantly, the people that plan, develop, integrate, verify, validate, accredit, and use them. Deficiencies in any of these can present a risk for using M&S incorrectly, thus obtaining unreliable and invalid test results.6 This, in turn, can result in poor decision-making on the part of the developer in designing and producing a new product. So, while M&S will be an integral part of many T&E programs, test planners should plan carefully for the use of M&S.
A wide range of models and simulations are used in T&E. The specific method used for a particular application should depend on the requirements of the T&E planning, the capabilities needed, the product maturity level and, in particular, on what aspects of the system or systems being modeled need to be represented.
http://www.youtube.com/watch?v=ghV5uv5AZPU Go to this link to view how computer simulations are used in crash testing of a new car. Think about how much it costs to run a real crash test versus a simulated crash test - they run the real crash to verify the models used in design.
What Is a Model?
As discussed in "Introduction to Modeling and Simulation," a model is simply a physical, mathematical, or otherwise logical representation of a system, entity, phenomenon, or process. A model can be a representation of an actual or conceptual system that can be used to predict how the system might perform under various conditions or in a range of "real-world" environments.
What Is a Simulation?
As discussed in "Introduction to Modeling and Simulation," a simulation is an implementation of a model over time. That is, it is a way to examine how a model behaves over time. More precisely, it is the process of conducting experiments with a model for understanding the behavior of the system modeled under selected conditions. Simulations may include the use of computer inputs, laboratory models, or mock-ups of actual products. Simulations are often programmed for use on a computer; however, in the broadest sense, live military exercises and war games are also simulations.7
What Are the Categories of Simulations?
There are three basic categories of simulations: live, virtual, and constructive. These are often combined and referred to as LVC simulations.8
Live simulation: A simulation involving real people operating real systems. Live simulations may use equipment that is representative of an actual product and can be connected with other systems such as virtual and constructive simulations.9
Virtual simulation: A simulation involving real people operating simulated systems (see Figure below). Virtual simulations sometimes replicate or use actual equipment in a central role by exercising motor control skills, decision skills, or human-generated communications. In many virtual simulations, the operators are immersed in a virtual environment that looks, feels, and behaves like the real thing. Virtual simulations can enable the testing of "dangerous" tasks at no risk to the operator, and hopefully at no risk to the equipment.10
Airmen and civilians from the 552nd Air Control Wing at Tinker Air Force Base, Oklahoma, work in a virtual mission simulator during a training exercise. The programs used in the mission simulators involve real people operating simulated equipment that is configured exactly the same as the actual equipment used on aircraft. (Photo by Staff Sgt. Stacy Fowler, U.S. Air Force)
Constructive simulation: A simulation that involves simulated people operating simulated systems. Real people stimulate, or make inputs, to such simulations, but are not involved in determining the outcomes. Constructive simulations cover the range from a simple single system simulation to complex multi-simulation interactive configurations.11
What Is Verification and Validation?
Product developers must be confident that the models and simulations they use for T&E are credible and will perform as intended. Verification and validation of a model or simulation helps to answer the questions, “Are we using the right one?” and “Is it built correctly?” Credibility within the development community can only be achieved through a robust verification and validation process, followed by an acknowledged willingness by the developer to accept the subject M&S for their T&E requirements. Therefore, the product developer should identify the intended use of M&S and any specific requirements early so that resources can be made available to support development and verification and validation (V&V) of these M&S tools.
Verification: Verification is concerned with functionality, answering the question, “Does it work the way it was designed?” Verification in the context of this lesson is the process of determining that a model or simulation and its associated data accurately represent the developer's conceptual description and specifications. It is often summarized as, “Did we build the model right?” For instance, a model of the Boeing 777-200 landing gear system might include a green light that is supposed to come on in the cockpit to indicate when the gear is fully extended. When that model is tested, if the green light doesn’t come on in the cockpit when the landing gear is fully extended, then the model has failed its verification. If the model fails its verification, then of course it has to be corrected or it will be of no use. Verification should be a continuing process; the analyst should not wait until the entire model or simulation is complete to begin the verification process.12
Validation: Validation is concerned with fidelity, answering the question, "Does it look and act like the real world?" Validation in the context of this lesson is the process of determining the degree to which a model or simulation and its associated data are an accurate representation of the real world from the perspective of the intended uses of the model or simulation. It is often summarized as, “Did we build the right model?” Primary questions for validation to answer include, “Does model match real world?” and “Can the model be substituted for the real system for the purposes of the test?” Using the same example of a model of the Boeing 777-200 landing gear, if the model was designed to have a red light come on when the landing gear was fully extended, the model might pass a verification test as it is operating the way it was designed. However, airplanes generally have a green light come on in the cockpit when the landing gear is fully extended, not a red light. So this model would not pass a validation test, as it would not be representative of the real world. If the model fails its validation, then of course it has to be corrected or it will be of no use. Validation is the precursor to approval to use the model or simulation, and there are many methods for performing validation. If there is an existing system, an ideal way to validate the model is to compare its output to that of the existing system.13, 14
What Is Hardware-in-the-Loop?
Hardware-in-the-Loop (HWIL) is another category of simulation that is known as a hybrid simulation. It usually consists of the actual hardware and software and external stimuli/drivers used to test the system's or sub-system's capability to operate in an environment simulating actual conditions. For the purposes of this lesson, HWIL simulations will be considered a methodology to integrate actual system or sub-system hardware in conjunction with other LVC models, and should not be limited to defining a specific laboratory or system. HWIL simulations place prototype or actual products and working components in the simulation to demonstrate their capability to operate within a selected environment that closely replicates real-world operating conditions. This may be just a single, small sub-system, or it could be very large and complicated, like a cockpit simulator of a Boeing 777-200 commercial airliner. The HWIL provides a system environment for the subsystem hardware to operate in.
It is much easier to control a HWIL system environment than a live system in an open air environment. Individual parameters can be controlled in the laboratory. The hardware performs the normal role that it would have operating in a real environment. Runs can be repeated, parameters can be changed as required, and the hardware is never destroyed in the test. Examples of HWIL simulations include dynamic engine test-benches, control systems, development test platforms, cockpit mockups, avionics integration systems, operational flight program simulations, and flight simulators.
Using HWIL, T&E can be conducted early in the development process, even before the product is completely developed. The degree of realism provided is dependent on the test requirements and based on the product maturity level discussed in the previous lesson. LVC simulations, especially HWIL, can be used to include a real-world environment and components into early maturity testing. That is why testing is done — to determine if components work individually and also together as a system. A realistic, simulated representation of the intended operating environment can be injected into the test. This gives the product developers and test planners early insight into how the components will work together as a system in a realistic setting. This early insight is especially important in allowing the developer to correct problems and faults before the product design is finalized.
There are also circumstances where the majority of T&E can only be accomplished by M&S. For example, there is no option but to rely on M&S to verify that a robotic Rover system landing on Mars is going to be successful. Test designers cannot create all of the environmental conditions in an Earth-based test, such as Mars’s gravity, magnetic field, atmosphere, and pressure. A significant portion of the T&E conducted for such missions are performed in HWIL tests using modeling and simulation to replicate environments encountered by the rover on Mars. These tests will generally use real Rover Hardware-in-the-Loop connected to various M&S laboratories to provide the needed realistic environment.
http://www.youtube.com/watch?v=F4bULefbHkg Go to this link to see HWIL T&E demonstration in automobile testing. Note the car industry wants to test a prototype in a safe environment before going on the road. They constructed a HWIL (called "HIL" in this video) environment in which a prototype dashboard can be tested by being connected to a computer that can simulate all the real-time car dynamics and vehicle control signals that will interact with the dashboard displays.
What Is a Systems Integration Laboratory?
As was explained in the lesson "What Is Test and Evaluation?," many commercial companies think in terms of using T&E to ensure that a product works as intended and is maturing in accordance with an expected schedule. Those companies conduct integrated testing early to expose weaknesses in a product’s design. The intent is to find out about problems early in the design and production of prototype systems, when they are easier and cheaper to fix. The communications company AT&T, for instance, refers to this as a "break it big early" philosophy. To reduce the risk of problems later in production, a concept they call "late-cycle churn," Boeing used a new technique to validate the 777-200 airliner’s maturity in a controlled setting — a Systems Integration Laboratory (SIL).15
Think of a Systems Integration Laboratory (SIL) as Hardware-in-the-Loop (HWIL) on steroids. A SIL is a risk-reduction facility where the complete product or system, including software and hardware, can be integrated and tested prior to building the first production prototype. The SIL will often start out with constructive models, which are replaced by HWIL and actual subsystem hardware as the system matures over time. The SIL provides a test facility that is a cross between a pure simulation and the final system. The SIL uses as many actual operating sub-systems, such as hydraulic sub-system, engine and power train, flight controls, and computer resources, as is technically and economically feasible.
The SIL enables the system to be tested around the clock, through a range of normal and extreme operating conditions in a very cost-effective manner. Boeing made extensive investments in their SIL so that they could test all of the 777-200’s main components in simulated flight conditions (Figure below).
Boeing linked over 60 geographically separated laboratories into the SIL. The laboratory combined actual 777-200 HWIL sub-systems, such as avionics, electrical system, and cockpit flight controls, with simulated flight conditions. Each simulated test flight recorded measurements from all systems, giving engineers the accurate data needed to investigate system operation and interaction. Test problems were recorded for each flight, entered into a tracking system, and processed as a “real” airplane flight discrepancy report.16
In this way, Boeing used the problems discovered during these simulated test flights to improve the aircraft, saving time and money. Boeing officials claim that the accuracy of the computer-generated information is critical to the credibility of such a laboratory and that Boeing’s large base of real-world, credible data was vital to the laboratory’s success. Ultimately, the laboratory "flew" about 2,000 test hours on the 777-200 and greatly enhanced the efficiency of subsequent actual flight tests. Flight testing still revealed problems, but the number of new problems was low, not significant enough to cause "late-cycle churn" or other problems in later development. Boeing was able to analyze early problems and identify potential solutions that were validated in the Systems Integration Laboratory before being incorporated on the aircraft. The monthly test-flying hour rates of the 777-200 airplane exceeded all previous programs, yet the number of new problems found on the airplane was low. Using a well-engineered SIL enabled the developer, Boeing, to reduce "late-cycle churn" and the risk to their product, saving time and money and ultimately allowing them to build a better airliner.17, 18
A Boeing 777-200 cockpit mockup included HWIL as part of a broader SIL. Using a SIL, Boeing included aggressive integrated T&E from the very beginning of the aircraft’s design and development. This approach was so effective that the Federal Aviation Administration certified the initial aircraft for overseas flight on the basis of Boeing’s T&E results. The certification normally requires two years of flight service. (Courtesy of Boeing)
http://www.youtube.com/watch?v=4KWEjWV2W90 Go to this link to see a System Integration Laboratory application used in developing new aircraft.
What Is Distributed Testing?
All the categories of live, virtual, and constructive models and simulations, including HWIL and SILs, can be connected or linked. In many cases, this is done with systems and capabilities, such as a SIL, that are not co-located. By sharing information through a Wide-Area Network (WAN), LVC capabilities can be linked around a campus, around a city, or around the world to form a distributed environment. When this is done to support T&E, it is referred to as distributed testing.
Distributed testing can be used to link sub-systems, systems, or system-of-systems to provide a distributed environment. Distributed testing can be used to integrate subsystems that are being developed or already exist at geographically separated facilities. It can also be used in lieu of some large-scale "open air" tests using actual, live, operational hardware for the systems involved. Conducting distributed LVC testing compliments live-only testing and provides the means for rapid integration of components and systems early in a product's developmental life cycle. It also provides an efficient means of adding realism to T&E by providing systems and capabilities not otherwise available, or by including separate but interrelated systems.
Distributed testing is particularly suited to some T&E, such as assessing a data exchange requirement between components, sub-systems, or systems, or within a system-of-systems. However, distributed testing will not be appropriate for all T&E. For example, system performance testing, product reliability testing, and other testing that does not include other systems or sub-systems will not likely use distributed methodologies.
How to Plan for Distributed Live, Virtual, and Constructive Simulations in T&E
Product developers should plan for T&E across the product’s entire developmental cycle to ensure that the product, its components, and its systems meet the appropriate maturity levels. M&S should be part of that planning to identify high-payoff areas in which to apply M&S and distributed LVC testing to conserve scarce resources. From early distributed tests using HWIL and SILs to test rehearsals using distributed LVC simulations, distributed testing can help early identification of problems and provide a cost-effective means to check out "live" test scenarios and thus reduce risk of failure for live-only testing. Distributed LVC simulations can provide a realistic environment for the system under test when it is impractical or too costly to use real-world assets. This is especially true for testing a system that is part of a system-of-systems, which will be discussed further in the next lesson.
Computer-generated test scenarios and capabilities, as well as LVC simulations of the system, can support distributed T&E by creating and enhancing realistic test environments. HWIL simulations enable testers and product developers to interact early with components and systems. Distributed LVC T&E can be used to identify and resolve issues of technical risk that require more focused testing. LVC simulations provide mechanisms for planning, rehearsing, optimizing, and executing complex tests. Integrated, distributed testing might also provide a means for examining why results of a physical test might deviate from pre-test predictions.
- A model is a representation of an actual or conceptual system.
- A simulation is an implementation of a model over time.
- The categories of simulations include:
- Live: real people, real systems.
- Virtual: real people, simulated systems.
- Constructive: simulated people, simulated systems.
- Verification and validation (V&V) provides confidence that the model or simulation used in T&E works the way it was designed and is an adequate representation of the "real world" for its intended use in a particular test.
- Hardware-in-the-Loop (HWIL) is a hybrid simulation that includes actual system or component hardware in conjunction with other LVC applications. It is not limited to a specific system or laboratory.
- A Systems Integration Laboratory (SIL) is a risk-reduction facility where the complete product or system, including software and hardware, can be integrated and tested prior to building the first production prototype.
- Distributed testing is a process for linking various geographically separated live, virtual, and constructive sites and capabilities together in a distributed environment.
1. What is the difference between a model and a simulation?
2. What are live simulations?
3. What are virtual simulations?
4. What are constructive simulations?
5. Briefly compare verification and validation.
6. Describe Hardware-in-the-Loop (HWIL).
7. Describe a Systems Integration Laboratory (SIL).
Further Reading/Supplemental Links
http://www.youtube.com/watch?v=ISiia1syhTM Not all HWIL testing is done by major companies on million dollar products.
http://www.youtube.com/watch?v=riBSh-4fxoc An example of a simulator, a virtual driving simulator. It includes selectable driving conditions, such as weather, traffic, urban buildings, and even a seat belt warning alarm, and uses a modified computer flat screen.
http://www.youtube.com/watch?v=HNC2SKzYZTM Another example of a virtual driving simulator. This is an emergency vehicle simulator that includes a more realistic "wrap around" screen configuration that is able to replicate accidents.
http://www.youtube.com/watch?v=KxpjiVDVo8Y This link will show you a virtual sinus surgery simulator. The operator manipulates the simulated operating probe and then sees what a doctor would see with a real probe and a real patient. How would you validate and verify this virtual simulation?
Points to Consider
For this lesson, points to consider are provided as an aid to the instructor to stimulate critical thinking among the students. These questions have no right or wrong answers, but may help further the student’s understanding of the material. Suggested responses are provided to help the instructor guide the discussion.
- In Test and Evaluation, why is the verification and validation (V&V) process for Models and Simulations so important? (Suggested response: Verification and validation provides a structured approach to develop M&S tools for the test process. It ensures that the M&S used will represent real world and that they don’t introduce errors or misrepresentations of the systems or the environments. This addresses the very basic questions all testers must ask themselves at some point in the T&E process, “How good is my data?”)
- When are distributed LVC methodologies not appropriate in T&E? (Suggested response: The most obvious conditions include system performance testing and reliability testing. You may be able to use M&S to help plan for a test of a new light bulb (i.e. how many bulbs you will need, how many times you should turn them on/off, and how long you should plan to leave them on). But to actually test the performance and reliability of a light bulb, you will need to use actual hardware. And, of course, a pizza taster's job would be very difficult to model or simulate.)
- What are some ramifications of developing a LVC distributed environment? (Suggested response: Developing a live, virtual, and constructive environment can be expensive, especially the initial development. But if it is an accurate representation, it will rapidly pay back those costs by reducing the expense in future testing. As the test planners become familiar with the LVC environment, they can use it for concept development for a whole new system to see how it operates in that environment. Developers can do that even before they have the hardware needed for the new concept.)
- Do LVC simulations have to be co-located to be effectively used in T&E? (Suggested response: It may seem counterintuitive, but no, models and simulations do not have to be co-located with the system under test. In fact, one of the primary advantages of using LVC simulations is the fact that they can be geographically separated, or distributed, and still provide the capabilities and environments needed for testing. This distributed LVC concept will be examined in the lesson "Introduction to T&E of System-of-Systems and Interoperability.")
1 DoD 5000.59-M “DoD Modeling and Simulation Glossary”
2 DoD 5000.59-M “DoD Modeling and Simulation Glossary”
3 DoD 5000.59-M “DoD Modeling and Simulation Glossary”
4 DoD 5000.59-M “DoD Modeling and Simulation Glossary”
5 DoD 5000.59-M “DoD Modeling and Simulation Glossary”
6 Defense Acquisition Guidebook, July 29, 201, Chapter 9, Test and Evaluation.
7 DoD 5000.59-M “DoD Modeling and Simulation Glossary”
8 DoD 5000.59-M “DoD Modeling and Simulation Glossary”
9 DoD 5000.59-M “DoD Modeling and Simulation Glossary”
10 DoD 5000.59-M “DoD Modeling and Simulation Glossary”
11 DoD 5000.59-M “DoD Modeling and Simulation Glossary”
12 DoD 5000.59-M “DoD Modeling and Simulation Glossary”
13 Defense Acquisition Guidebook, July 29, 201, Chapter 9, Test and Evaluation
14 DoD 5000.59-M “DoD Modeling and Simulation Glossary”
15 United States General Accounting Office, General Practices: A More Constructive Test Approach is Key to Better Weapon System Outcomes, July 2002, http://www.gao.gov/archive/2000/ns00199.pdf
16 A flight discrepancy report is a form used by pilots to describe malfunctions and problems an aircraft encounters during a flight. Maintenance personnel then use this report to identify what repairs must be made to the aircraft before it flies again. Not all discrepancy reports are serious, as they can identify that a minor light bulb has burned out and needs replacing. Or they can be very serious and cause the grounding of the aircraft, such as an engine failure during flight.
17 United States General Accounting Office, General Practices: A More Constructive Test Approach is Key to Better Weapon System Outcomes, July 2002, http://www.gao.gov/archive/2000/ns00199.pdf
18 Defense Acquisition Guidebook, July 29, 2010, Chapter 9, Test and Evaluation