- Explain the limitations and advantages of Test and Evaluation (T&E) in a distributed environment
- Describe how distributed live, virtual, and constructive (LVC) T&E can:
- Mitigate risk in product development
- Improve T&E capabilities
- Explain why distributed LVC T&E is not meant to replace live testing
- Describe the importance of "test-fix-test"
- Explain the advantages of scalable distributed T&E events
scalable: In electronics (including hardware, communication, and software), scalability is the ability of a system, network, or process to handle growing amounts of work in a graceful manner, or its ability to be enlarged to accommodate that growth. In the context of distributed T&E, we will refer to scalability as the ability to control the size and complexity of a distributed LVC event. It can be very small, to accommodate only two connected systems, or very large and complex, encompassing many systems and even multiple systems-of-systems.1
Check Your Understanding
In the three previous lessons, the student has been exposed to all the various "parts" needed to plan for and conduct distributed testing using LVC simulations. They should now have a basic understanding of the complexities and advantages of using such a T&E methodology. The concepts and details we have provided were not intended to make the student an expert on either T&E or distributed testing. But there should be a good enough foundation to understand when distributed LVC should be considered in T&E planning.
Testing using distributed LVC simulations is not meant as a solution to all T&E requirements. Nor will it correct all the "late-cycle churn" problems in developing a new product. However, there are distinct advantages and practical applications that can provide tremendous benefits. In this final lesson, we will introduce you to various considerations and specific advantages to the test planners, the developer, and ultimately the customer.
Limitations to Conventional Live Test and Evaluation Methodologies
Conventional Test and Evaluation (T&E) with actual and complete systems can provide for testing in representative and realistic real-world environments. However, this usually requires building a large and expensive T&E event. In many cases, testing in such a live scenario will be subject to limiting factors, such as:
- Systems that are interdependent with a system-of-systems might not be readily available because they are in high demand elsewhere, do not exist in adequate numbers yet, or do not exist at all.
- There may be limited availability of related systems needed to complement a system’s realistic environment.
- Systems may not be co-located at the test site.
- Fully complete, actual systems are not built yet.
In fact, in most cases, needed systems and capabilities are not co-located at the desired T&E venue. Relying solely on live systems for T&E is often impractical, usually very expensive, and sometimes simply impossible.
What Are the Advantages of Distributed Testing with Live, Virtual, and Constructive Models and Simulations?
A distributed T&E environment can be built using a mixture of live, Hardware-in-the-Loop (HWIL), virtual, and constructive capabilities, all connected together. That is, the various testing components can be connected on a network to form a distributed operating environment for the system under test (SUT) linking all its enabling systems. In this distributed environment using LVC representations, the testing components and systems need not be co-located. This distributed test approach will allow the product’s developer to customize the T&E methodology to capitalize on the particular advantages of each capability as it is needed. Also, components and systems not previously available can be fully integrated into the T&E process.
Distributed test events need not be solely large scenario-driven events, nor incorporate the LVC assets available all at once. One event may include live and virtual capabilities, another may include virtual and constructive, while a third may link multiple HWIL or Systems Integration Laboratory (SIL) facilities with a live operator. Smaller-scale testing can be done using a distributed infrastructure to provide technical risk reduction prior to linking the environment of all interrelated systems in a larger test. Linking the various systems and components of a large system or system-of-systems using distributed test capabilities may prove to be the simplest, quickest, and cheapest way to avoid the pitfall of conducting T&E assessments based solely on individual, system-particular performance parameters.
How Does Distributed Live, Virtual, and Constructive T&E Mitigate Risk?
In previous lessons, we have shown how various companies and developers consider reducing what has been called "late-cycle churn" in the design and manufacturing process for new products. As we discussed in the lesson "What Is Test and Evaluation?," the Pentium® Pro and Pentium® III are much more complex and have significantly improved capabilities over their predecessors, yet due to improved T&E capabilities they were developed in the same amount of time as the original Pentium® microprocessor. Some consider the concept of "break it early" to be important in finding problems with a product or system so they can be corrected before the design is finalized. The problem with the concept of "breaking it early" is how to best do that. Testing using only actual, live systems may be logistically impossible, can be very expensive, and can intrude on production schedules. In some cases, it may be impossible if the system under test has not yet been completely built. A distributed LVC environment can be developed at a fraction of the cost of testing with live systems. This in turn offers two distinct advantages for both test planners and the developer in mitigating risk for a new product:
- Distributed testing allows for early and continuous T&E to be integrated into the product’s developmental process. This idea is called "test early, test often" and allows developers to embed testing into all phases of a product's design, development, and production. As we have shown in previous lessons, conducting T&E very early in a product’s maturity cycle by linking distributed components or using HWIL and SILs allows developers to correct problems early, when they are cheaper and easier to fix.
- Distributed LVC simulations can be built to provide a realistic environment for the product or system under test, including all the necessary and interrelated systems. That is, the system under test’s planned or expected real-world operating conditions can be replicated in a distributed testing environment that can provide the characteristics, components, systems, and other capabilities needed to test technical and user performance. Providing this realistic environment early and often allows the developer to learn how their product may behave in actual operating conditions well before the product’s design is finalized, when problems are easier and cheaper to fix.
These two advantages become even more apparent when one considers not just individual systems, but the rapidly expanding network-centric environment and the need to conduct T&E in ever more complex systems-of-systems. The ability to "test early, test often" in realistic environments will become a driving force behind distributed testing.
How Does Distributed Live, Virtual, and Constructive T&E Improve Testing?
There are many aspects to using distributed LVC simulations to improve T&E.
- Representations of the system under test (e.g., HWIL, SILs) can be integrated into the testing process before the actual system under test (SUT) is built.
- Representations of other systems with which the SUT must interact can be integrated to test for interoperability.
- Systems and components can be integrated into the test of a live system without having to move the additional systems and components to the test site.
- Environmental models can be linked into the test to provide a realistic real-world environment. This can include bringing weather, challenging terrain, safety hazards, and other challenges into the test.
- In many cases, distributed test events are easy and relatively cheap to repeat. This allows the developer to collect much more data on the technical specifications and performance of a product.
- Due to the more controlled LVC environment, there is a significantly reduced risk of damaging the system under test, test equipment, or people.
- The test can proceed without damaging the environment, thus mitigating environmental restrictions on testing.
- Distributed test events allow for enhanced cross-flow of test data between distributed LVC T&E systems and facilities.
Conducting a technical or user test on a pizza, as described in the lesson "What Is Test and Evaluation?" may be time-consuming, but it is a relatively simple process. Testing today’s complicated systems and systems-of-systems, such as a new airliner flying in the national airspace system, presents much more of a challenge to developers and test planners. Ultimately, LVC representations can be used to evaluate the capabilities of a system or system-of-systems in a realistic distributed environment. This distributed environment can be developed at a fraction of the cost of testing with all live components and provides the capability to evaluate technical and operational performance for individual systems and systems-of-systems. This way, technical T&E can incorporate realistic environments much earlier in the product’s design phase. User T&E planners can also use distributed LVC T&E methodologies to verify early that the system under test works both "stand alone" and in the expected system-of-systems environment. Early integrated testing will allow the developers to find problems early in a product’s development, while they are still relatively easy and inexpensive to correct. Testing can then continue across the entire developmental cycle of a new product.
Why Is the "Live" So Important to T&E?
In order to convince him or herself, the agencies that oversee the industry (e.g., the Federal Aviation Administration or the Environmental Protection Agency), and the customer that the system performs as intended in the expected environment, the developer will be compelled to test the system in the "real world" using the actual system with other, related actual systems. Some T&E planners may want to consider using a Modeling and Simulation (M&S) methodology only, with no live systems. Regardless of how well a model or LVC simulation is constructed, it is still a representation. It will be very difficult to convince the decision maker that the system under test will function as intended in the real world using only M&S. For credible T&E, there must be some level of testing with a real system, a real user, and the expected, actual environment. In short, T&E planners must do "live" testing using real people and real systems to ensure the product has no serious faults or safety issues before it is released for use by the customers. Because of the ability to inject the "live" into testing, distributed LVC testing can provide for earlier identification of problems and reduce the amount of live testing, thus reducing cost and risk.
Distributed LVC testing cannot eliminate the risk of a major fault in the product, but can be used to build up to a live test and greatly reduce the risk before the live test occurs. A good example of this occurred recently, when the U.S. Air Force was adding a new digital communication capability to portions of their bomber fleet. Prior to conducting an actual flight test, the Air Force tested the interoperability of the new communication system by linking an actual bomber aircraft that was on the ground but manned with real pilots to a virtual simulator of an airborne communications aircraft at a separate base, also manned with real operators. A data collection facility at yet another base was also linked to complete the test. The Air Force testers discovered problems in the communications software, which the developers fixed within hours. The testers then ran the test again to verify that the communication system performed as intended with the fixes applied. All this was done with a combination of live and virtual systems separated by hundreds of miles without requiring a single flight. Conducting the live and virtual distributed testing provided for a significant reduction in risk to follow-on live flight tests of the new communications system, and saved the expense of multiple flight tests.
What Is "Test-Fix-Test?"
The above Air Force example leads us right into the concept of test-fix-test. The complexity and expense of today’s net-centric systems clearly demonstrate the need to test early and often throughout the development process and the life of the system. Test-fix-test refers to an iterative process in which the new system is subjected to a test, deficiencies are immediately corrected, the corrections are tested, and then the system continues on to another development and test cycle.
Table below shows the cost of fixing a defect depending on the stage during development it was found. For example, if a problem in the design is found only post-release, then it would cost 25 to 100 times more to fix than if it had already been found by the design phase review.2 In addition to the safety aspect, think of the expense of a nation-wide recall of automobiles due to a faulty design of the brakes or an accelerator switch when thousands and thousands have been sold to customers. Successful test-fix-test generally involves early and continuous testing of the new system from the early developmental stage through fielding to customers, to dramatically reduce a product’s "late-cycle churn." This is exactly the impact a distributed SIL can have on developing new products like the Boeing 777-200.
Early testing of a system’s capability to operate in its intended environment will allow designers and system engineers to identify and correct fundamental issues with performance and interoperability before they are built into the system. That will obviously reduce development time and cost and help designers build a better product, faster, and at a lower cost!
Cost to Fix Defect versus Time Detected
Cost to fix a defect
Advantages of Scalable Distributed Live, Virtual, and Constructive T&E Events
Distributed LVC methodologies offer T&E planners the opportunity to participate in scalable LVC events tailored to the test requirements. Distributed test events can be planned to support varying degrees of complexity, capabilities, requirements, funding, system maturity, and scheduling limitations. This includes experiments, early technical testing, and even follow-on user testing that might consist of:
- A distributed system-to-system event consists of connecting two systems or sub-systems together to test their interaction. In a distributed system-to-system event, test planners have the maximum ability to inject their test requirements into the event and to control test conditions and execution. These generally small, inexpensive events can be an excellent venue for "test-fix-test" and early risk reduction for the system/product, in order to assess the interaction and data exchange between systems or components. It allows the test planners to schedule and execute the event as often as needed, with minimal costs and without having to move the systems or sub-systems to the same place. This can be done using HWIL or a SIL to represent the system under test.
- A distributed network-centric event is a T&E event that uses an environment of LVC models and simulations. The size and complexity of the T&E event and the capabilities provided will naturally depend on the T&E requirements. Test planners could use a relatively small event to provide the most realistic environment possible, from concept exploration through follow-on T&E when it is too costly to bring all the player systems together at a single place. This would inject a real-world influence early in the product's design and development stage. A larger distributed LVC event could be an excellent venue to provide an assessment of system-of-systems interoperability issues early in the acquisition process. That is especially true when there are not adequate numbers of systems under test or support assets required for live testing. An even more complex network-centric distributed T&E environment would provide interactions between systems in a realistic system-of-systems environment. An important advantage of such an event is the ability to provide a realistic operating environment. This could include such factors as weather, rough terrain, heat, cold, adverse air quality (such as smoke and haze), and poor lighting conditions. In this case, virtual and constructive capabilities could supplement live systems to assess interoperability and reduce risk at a fraction of the time and cost of a live on-site test event. These larger events could naturally include various HWIL facilities and even multiple SILs.3
Are There Disadvantages to Distributed Live, Virtual, and Constructive T&E?
As with every aspect of T&E, not all solutions will be appropriate for every situation. Distributed LVC simulation would certainly not be appropriate for testing the pizzas or permanent wave machines that were discussed in the lesson "What Is Test and Evaluation?" Using distributed LVC T&E is certainly not free. The various systems and components must be brought on line, which may be difficult with differing communication systems. Distributed testing requires detailed test planning, so all the factors affecting the test must be understood and addressed before the test execution. For many distributed LVC T&E events, there may not be a completed product or real system, and in some cases no actual users. This may complicate the problems the developer may have testing the product or system in use by actual customers. But these same limitations apply to conventional testing. While not a panacea, and not appropriate for all situations, distributed LVC T&E has proven in many situations to help developers build a better product quicker and cheaper.
What Is the Bottom Line?
Conducting T&E using distributed LVC simulations can save the product developers time and money, reduce risk, and ultimately produce a better product! It is not appropriate for every situation, but given that a product developer does not want to sell the customer a product that does not work, they will need to test the product to ensure that the product works as intended in the conditions the customer will actually use it. Conducting T&E with live-only systems will become increasingly more difficult in the ever-changing network-centric, system-of-systems world we live in. Using LVC simulations in a distributed test environment can enable developers to test early and often in a product's development cycle and so identify and fix problems when they are much easier and cheaper to correct. A distributed LVC test environment presents a unique capability for testing net-centric systems-of-systems.
- There are limitations to testing with actual live, complete systems.
- Advantages of using distributed live, virtual, and constructive simulations in test and evaluation include being able to more easily customize individual events based on requirements and capabilities.
- Risk in developing a new product can be mitigated by:
- Testing much earlier and more often in a product’s development. This idea is called "test early–test often."
- Providing a realistic environment for the product or system under test and all interrelated systems.
- T&E using distributed live, virtual, and constructive simulations improves the T&E process and enables early and continuous testing across the developmental life cycle.
- "Test-fix-test" refers to an iterative process in which the new system is subjected to a test, deficiencies are immediately corrected, and then the system continues on to another test cycle.
- Distributed test events can be scalable and planned to support varying degrees of complexity, capabilities, requirements, funding, system maturity, and scheduling limitations.
- The bottom line is that T&E using distributed LVC simulations can save the product developer time and money, reduce risk, and ultimately produce a better product!
1. List two limitations in conducting T&E with actual live systems.
2. How does distributed LVC T&E mitigate risk?
3. How does distributed LVC T&E improve testing capabilities?
4. What is "test-fix-test"?
5. Why is scalability important in distributed LVC T&E?
6. What is the bottom line in using distributed LVC simulations in T&E?
Further Reading/Supplemental Links
http://www.benmeadowcroft.com/reports/systemfailure/ Go to this link to view a report on the major causes of system failure. The continued testing of systems already in use by the customer is also important in being prepared for potential system failures.
Points to Consider
For this lesson, points to consider are provided as an aid to the instructor to stimulate critical thinking among the students. These questions have no right or wrong answers, but may help further the student’s understanding of the material. Suggested responses are provided to help the instructor guide the discussion.
- A study conducted by the National Institute of Standards and Technology in 2002 reported that software bugs cost the U.S. economy $59.5 billion annually. More than a third of this cost could be avoided if better software testing was performed.4 Why is it that some companies don’t embed T&E early into their product’s development? (Suggested response: An aggressive T&E program can appear to be expensive and time-consuming. There is no doubt that some test events, even distributed LVC events, can have a large "up-front" cost. While developers will ultimately reap time and cost savings, the "up-front" cost of T&E can be intimidating. Also, there can be an attitude with some developers that T&E only brings bad news. While that may be true in the short term, as T&E discovers faults and problems in a product, T&E should actually be considered a money maker. Discovering those faults and problems early in a product’s development can only reduce the cost to the developer and customer. T&E doesn’t cost, T&E pays!
- We have shown how early distributed testing provides for a testing capability without having to bring all the required test assets to one place. This will save developers time and money and reduce risk, especially for a system-of-systems in a net-centric environment. However, can a distributed LVC test capability provide for a more thorough evaluation? That is, can distributed methodologies improve the evaluation part of the test and evaluation process? (Suggested response: Most test planners will focus on the distributed aspect of data collection during a test. That is, distributed LVC methodologies allow for the tester to move data and not necessarily the people needed to collect and analyze the data. While not immediately obvious, the same potential holds true for the evaluation of the test data. When the data is available on a distributed network, the evaluators are able to access the data while not having to be co-located. This ability for distributed evaluation and even reporting is yet another advantage of using distributed LVC simulations for T&E.
- Distributed testing is gaining recognition and acceptance from developers and testers. However, while the concept has proven its worth, work remains to improve distributed test capabilities and to educate product developers and testers to use it appropriately. This chapter in the Modeling & Simulation Flexbook® digital resource is intended to address this very subject. In completing these four lessons, you now have a very good foundation in T&E and how distributed LVC simulations can be used.
1 André B. Bondi, ‘Characteristics of scalability and their impact on performance’, Proceedings of the 2nd international workshop on Software and performance, Ottawa, Ontario, Canada, 2000, ISBN 1-58113-195-X, pages 195 – 203. As cited in Wikipedia article, Scalability, Jan 2012, http://en.wikipedia.org/wiki/Scalability.
2 McConnell, Steve (2004). Code Complete (2nd ed.). Microsoft Press. pp. 29. As cited in Wikipedia article, Software Testing, Jan 2012, http://en.wikipedia.org/wiki/Software_testing, Jan 2012.
3 Ferguson, Chip, Making the Case for Distributed Testing, ITEA Journal, Sept 2010
4 NIST. Economic Impacts of Inadequate Infrastructure for Software Testing, May 2002