Specification Projects under the Eclipse Foundation Specification Process MUST produce a Technology Compatibility Kit (TCK) that delivers on the promise of enabling multiple compatible implementations.
This document defines:
It is the role of the TCK to ensure both compatibility of implementations and portability of applications.
TCK tests must be written following all rules and restrictions that applications must follow including, but not limited to, specification rules, trademark guidelines and license terms. Tests that do not follow these rules and restrictions as they pertain to applications may be deemed invalid and excluded using the Challenge process.
Projects MUST produce a final candidate binary that includes the EFTL license. The Sparkplug Specification Committee will sign and promote project TCK binary for distribution via Eclipse infrastructure on final approval. This is the TCK binary usable for self-certification when one desires to make a claim of compatibility, allowing for the use of the Sparkplug brands.
Both TCK binaries MUST contain the following
We recommend that the TCK documentation include
TCK binaries MAY contain
Release available via a release on project GitHub releases page (or equivalent)
Specifications are the sole source of truth and considered overruling the TCK in all senses. In the course of implementing a specification and attempting to pass the TCK, implementations may come to the conclusion that one or more tests or assertions do not conform to the specification, and therefore MUST be excluded from the certification requirements.
Requests for tests to be excluded are referred to as Challenges. This section identifies who can make challenges to the TCK, what challenges to the TCK may be submitted, how these challenges are submitted, how and to whom challenges are addressed.
Any implementor may submit a challenge to one or more tests in the TCK as it relates to their implementation. Implementor means the entity as a whole in charge of producing the final certified release. Challenges filed MUST represent the consensus of that entity.
Any test case (e.g., test class, @Test method), test case configuration (e.g., deployment descriptor), test beans, annotations, and other resources considered part of the TCK may be challenged.
The following scenarios are considered in scope for test challenges:
The following scenarios are considered out of scope for test challenges and will be immediately closed if filed:
Challenges MUST be filed via the specification project’s issue tracker using the label challenge and include the following information:
Challenges can be resolved by a specification project lead, or a project challenge triage team, after a consensus of the specification project committers is reached or attempts to gain consensus fails. Specification projects may exercise lazy consensus, voting or any practice that follows the principles of Eclipse Foundation Development Process.
The failure to resolve a Challenge might prevent an implementation from going to market; Challenges SHOULD be given a high priority by the specification project and resolved in a timely manner. Two weeks or less SHOULD be considered the ideal period of time to resolve a challenge. Challenges may go longer as needed, but as a rule SHOULD avoid months.
If consensus cannot be reached by the specification project for a prolonged period of time, the default recommendation is to exclude the tests and address the dispute in a future revision of the specification.
A consensus that a test produces invalid results will result in the exclusion of that test from certification requirements, and an immediate update and release of an official distribution of the TCK including the new exclude list. The associated challenge issue MUST be closed with an accepted label to indicate it has been resolved.
The specification project may approve (user) workarounds for an accepted TCK challenge (as alternative to excluding TCK tests).
When a challenge issue is rejected, it MUST be closed with a label of invalid to indicate it has been rejected. The appeal process for challenges rejected on technical terms is outlined in Escalation Appeal. If, however, an implementer feels the TCK challenge process was not followed, an appeal issue MUST be filed with the specification project’s issue tracker using the label challenge-appeal. A project lead MUST escalate the issue with the Sparkplug Specification Committee via email (sparkplug-spec@eclipse.org). The committee will evaluate the matter purely in terms of due process. If the appeal is accepted, the original TCK challenge issue will be reopened and a label of appealed-challenge added, along with a discussion of the appeal decision, and the challenge-appeal issue will be closed. If the appeal is rejected, the challenge-appeal issue MUST be closed with a label of invalid.
* Spec lead should email issue to sparkplug-spec@eclipse.org
Excludes MUST be included in the TCK project release in a format that is compatible with the testing framework in use so that as the excludes are updated, the affected tests are automatically removed from the test suite.
Excluded tests should be added back in for every major Sparkplug release by emptying the test exclude list for every Specification developed in the respective major release.
Requests for improvement to tests MUST simply be created as issues with a label of enhancement in the specification project’s TCK issue tracker.
Sparkplug is a self-certification ecosystem. If you wish to have your product featured in the official list of compatible products, a listing request as defined in this section is required.
There are additional requirements that MUST be met by any organization wishing to use the “Sparkplug Compatible” logo or Sparkplug website for promotion. Any request for listing from an organization not meeting the requirements will be held until such time as the requirements are met. See the Sparkplug Trademark Guidelines for more information.
An approved listing request is a statement from the Specification Project that you have met the intended TCK requirements and is just one of the requirements for logo usage.
Requests to be acknowledged as a certified implementation MUST be filed creating an issue in the dedicated repository managed by the specification project and include the following information:
To simplify the process, an issue template is available in the relevant repository.
While certification is on your honor, the community MUST be able to see your test results summary. At a minimum a results summary MUST:
An optional “Test List Page” showing all tests run may be linked from the Summary Page. The Summary Page URL is the URL that MUST be included in any Certification Requests.
The following are explicitly not requirements:
Implementers may supply this information and provide support for how to run a TCK against their product, but it is not required.
As noted, listing requests do not in themselves grant rights to use the “Sparkplug Compatible” logo. Approval that the TCK requirements have been met is a prerequisite for obtaining logo usage permission. The required approval processes is:
All specification project members are encouraged to review the request and associated supporting materials. Reviewers of a listing request MUST carefully check the validity of all required data, in particular:
Any committer on the specification project may vote against the certification request on the basis that the clearly defined requirements of the TCK process have not been met. This means that if there is a (-1) vote, lazy consensus is no longer an option and a majority vote MUST take place.
Certification requests that are reviewed and found to meet the requirements will be marked accepted by closing an issue with an accepted label. A pointer/link to the issue MUST then be emailed to tck@eclipse.org, as required by the Eclipse Foundation Technology Compatibility Kit License.
Listing requests that are reviewed and found to NOT meet the requirements will be marked as such by closing an issue with an invalid label along with the requirements that were not met. A new certification issue MUST be created with the updated requirements to attempt the certification request again.
If there is a concern that a TCK process issue has not been resolved satisfactorily, the Eclipse Development Process Grievance Handling procedure SHOULD be followed to escalate the resolution. Note that this is not a mechanism to attempt to handle implementation specific issues.
The only time tests may be added to a TCK are in a major or minor release. A service release which updates the excluded list MUST not have test additions or updates.
The process for releasing a point revision of a TCK entails filing an issue in the sparkplug/specifications repository with the following details: