Testing and Certification#
Key Message - Testing and Certification
Testing and certification is a critical enabler of grid modernization with benefits to industry ranging from operational trustworthiness to investor confidence. Yet the availability of testing and certification programs is limited.
Smart grid standards often include many user-selected options. This optionality typically allows for non-interoperability even among products conforming to the same standard. This means that the current industry focus on certifying conformance to individual standards is inadequate to assure interoperability.
Interoperability Profiles that describe the communications protocol and data model requirements necessary to achieve a specific set of physical functions are possible solutions to the interoperability challenge.
Interoperability profiles are not new standards, but instead describe a subset of requirements from existing standards that—when implemented and verified through testing and certification—would ensure interoperability across devices and systems.
The modern electric grid is often described as a “system of systems” (SOS) spanning multiple technology domains, involving thousands of organizations, and hundreds of standards. Smart grid devices, systems and applications require extensive data exchange and need well-defined interfaces to transfer and translate this data between points across the grid. Interoperability is necessary to provide seamless functional performance across systems that enables many benefits of the smart grid.
The Role of Testing and Certification#
Testing and Certification (T&C) programs provide common processes that are used to demonstrate conformance with a standard. When accepted and used across industries, these testing and certification processes support interoperability between devices and systems that span equipment vintage and manufacturer. Completing a T&C program allows vendors to offer products certified to a standard, and affords customers a level of trust that products will work as intended when deployed.
Testing and certification value#
Warning
What is Section 4.3.5?
As described in Section 4.3.5, the T&C value proposition benefits all grid stakeholders. The following list describes these benefits :
- Customers benefit by ensuring standards and performance requirements are implemented appropriately and consistently across purchased equipment.
- Manufacturers and Vendors benefit from the establishment of clear performance requirements, which reduces implementation costs for new standards.
- Regulators benefit because interoperability T&C maximizes the benefits of new grid technology investments they approve through regulatory proceedings.
Current practice#
NIST worked with sector experts to create best practice guidelines for the development of T&C programs for smart grid systems and devices. The foundational products are the Interoperability Process Reference Manual (IPRM) standard [134] and accompanying User’s Guide [140]. The IPRM standard defines a process by which industry stakeholders may procure, test, and assert interoperability between disparate vendors of smart grid products built to specific standards.
Levels of Interoperability#
There are many levels of interoperability, each with its own industry definitions. This is important for T&C because most standards, tests, and certifications have been created by industry to deliver a desired function and interoperability level.
A practical way to characterize interoperability is though the integration concept. From plug- and-play on one end of the spectrum to point-to-point integration on the other, each interoperability level describes the ease with which communication can be achieved across devices or systems, and the time and expense associated with overcoming any communications gaps. Figure 1 provides a cartoon representation of three interoperability levels, where plug-and-play allows for direct communication at little expense to the device operator and point-to-point integration requires development of custom interfaces that take considerable time and money to develop.
Types of Testing Processes#
The development of tesing and certification programs involves many steps, which are shown in Figure 2.
Conformance and interoperability testing#
Conformance testing ensures products conform to requirements detailed in a specification, while interoperability testing ensures that products from different vendors can communicate in any way in the same system. To do that, a common interpretation of a standard must be defined in an implementation agreement. Protocol Implementation Confornmance Statement (PICS) enables the development of a test plan from the requirements identified in the implementing agreement. This often leads to an implementation profile which can be used as the basis for interoperability testing requirements.
Interoperability testing is important because it can expose compatibility issues for devices from different vendors operating within a closed system. The key takeaway on conformance est interoperability testing is that they are both necessary to enable interoperability of smart grid devices and systems.
Certification regimes#
There are three certification types described in the IPRM standard: first-party certification, which is when a manufacturer attests that a product meets the standard requirements, second-party certification is when a user tests and certifies when a product meets the standard’s requirements, and third-party certification, which is done through an independent authority that includes a certification body and associated test lab.
Current Smart Grid Testing Initiatives#
NIST is involved in a number of initiatives designed to support development of testing and certification regimes across the sector, a few of which are described below.
Testing support#
Testing events and program formulations are key to developing a strong smart grid testing ecosystem but often lack sufficient organizer support. Enhanced support is essential to advance these activities beyond their current early stages.
Catalog of standards#
Due to the rapid growth in standards and the increased complexity of information, static lists became less useful. NIST has therefore worked with industry to develop the SEPA Catalog of Standards (COS), an online version of the smart grid-relevant standards lists published in prior Frameworks which is publicly accessible and is updated regularly through a consensus-based and transparent process.
Catalog of test programs#
The COS provides curated and useful information on standards relevant to smart grid development and deployment but does not, however, include information on associated test programs for these standards. A lack of readily accessible information on the availability of T&C programs is a barrier to further adoption and limits the interoperability benefits to industry that these techniques could provide.
To address this information gap, NIST conducted a landscape analysis which revealed that only a few interoperability standards have associated T&C programs and identified significant challenges in accessing these programs, even for experts. It suggested the industry would benefit from a central repository detailing available test programs.
Reference interoperability procurement language#
Procurement language is crucial to specifying product interoperability requirements so that integration issues can be addressed and mitigated before deployment. Without clear interoperability criteria in procurement, there’s a higher risk of products not working with existing systems or equipment.
A challenge in this space is that technical requirements for acquisitions by utilities and other customers are often focused on the functional specifications of the systems to be purchased, and do not include appropriate descriptions for the interoperability requirements. One solution is to develop reference language so that it is readily available to specify interoperability requirements during procurement.
NIST is collaborating with SEPA and the DOE’s GMLC to create guidelines and a list of criteria and metrics for interoperability in procurement language, aiming to improve stakeholder procurement practices.
Towards Interoperability Profiles#
As described previously, the latest landscape assessment reveals the availability of T&C programs for smart grid standards is quite limited (see Section Catalog of test programs). This situation could be improved by creating interoperability profiles with specific requirements and developing open-source test tools, both of which would increase stakeholder confidence in device performance.
Interoperability profiles#
Interoperability Profiles reduce implementation and testing complexity by curtailing and clarifying the range of interoperability requirements to a well-defined subset of those available through standards.
Interoperability Profiles would not replace or be considered standards, but instead would clarify standards-based interoperability implementation requirements for all stakeholders. Interoperability Profiles could therefore take many different forms based on the technology and underlying standards, and by defining the elements of the standard to be utilized for specific application environments would give all stakeholders greater confidence in asset functionality.
The basic set of elements for an Interoperability Profile include elements like asset description, performance specifications, communication protocols, and information models. They address interoperability failures by limiting the parts of the data model used, ensuring devices communicate effectively. An Interoperability Profile with a narrow set of implementation requirements could be more easily tested for certification, and eventually could be listed by vendors that support it or could be used in procurement specifications by end users.
Example of an interoperability profile#
The core elements of the Interoperability Profile approach have already been successfully demonstrated for smart inverters. California Rule 21 [205] and IEEE 1547-2018 [206] both define the specifications for interconnection and interoperability of distributed energy resources with associated electric power system interfaces. The standards include physical performance specifications, communication protocols, and require data elements.
While the inverter physical performance requirements are clear, the relatively large number of potential communications protocols and data model implementations could limit the ability to test for and certify device interoperability under the IEEE 1547 standard.
California Rule 21 also establishes rules for interconnection of inverter-based DER to the grid and demonstrates the application of an Interoperability Profile on existing standards by narrowing the degrees of freedom and complexity for implementing the required communication.
Interoperability profiles work plan#
As discussed in the Section Interoperability profiles and visually represented in Figure 6 below, Interoperability Profiles draw from existing standards to clarify the requirements that will ensure interoperability to support specific functionality of grid-connected assets. The process for developing an Interoperability Profile includes selecting system interfaces and developing a use case to define functional requirements, and then creating a series of documents that describe specific interface requirements and provide implementation guidance.
The Interoperability Profile will describe how to achieve the interoperability and functionality requirements of the use case by specifying the appropriate configuration options from each standard described in the application guide.
Open-source test tool development#
A key component of the T&C process is the test harness, which is created by translated test cases into automated scripts that can be executed to evaluate interoperability functions. Test harnesses create efficiency by automating the test process, but creating test harnesses is often the costliest part of the test program and requires a different skillset than traditional standards development.
Once a complete Interoperability Profile is available, the smart grid community can then assess the viability of developing the relevant test harness in an open-source environment. The benefit of this approach is that it would allow for broad participation in the test harness development process, and thereby reduce entry barriers for T&C activities.
Conclusions and Future Work#
Meaningful interoperability across the power sector is difficult to achieve and — at this time — virtually impossible to ensure. Testing and certification programs are key to enhancing interoperability but are limited in availability and scope. Test programs must also evolve beyond their current practice focusing on standards-driven conformance assessment into a model that supports the types of tests which would better guarantee interoperability of device and system. Interoperability Profiles and open-source test harnesses are two tools that, if properly developed, would facilitate expansion of interoperability focused T&C programs.