A bit about conformance testing

Posted on

When I participated in a software project where we develop operating system I was responsible to testing different parts of operating system and applications that worked on that operating system. And I want to say that operating system must conform many standards to be compatible with exited software applications: TCP/IP, POSIX.1-2017, USB 1.1/2.0/3.0, IEEE 754, OpenGL etc. I spend a lot of time to find and build a list of existed opensource test suites that can help with conformance testing.

Conformance testing is an element of conformity assessment, and also known as compliance testing, or type testing — is testing or other activities that determine whether a process, product, or service complies with the requirements of a specification, technical standard, contract, or regulation.

To save time for other people I decided to publish a list of conformance test suites.

Standard Testsuite
WebCL WebCL conformance tests
WebRTC WebRTC conformance tests
SQL sqllogictest
SQL sqlancer
SQL go-sqlancer
SQL sqltest
SQL Structured Query Language (SQL) Test Suite
SQL Hypothesis + lark-grammars
QUIC: A UDP-Based Multiplexed and Secure Transport quic-tracker
Web ACID
Web Test suites for Web platform specs, Test the Web Forward
HTML 5 HTML 5 Test
HTML 5 HTML Benchmarks
RFC 4918: HTTP Extensions for Web Distributed Authoring and Versioning (WebDAV) http://www.webdav.org/neon/litmus/
WebSocket autobahn
ECMAScript test262
XServer XTS - X Test Suite
OpenGL The SGI OpenGL conformance tests
OpenGL Piglit
OpenCL OpenCL conformance tests
OpenVX OpenVX conformance tests
OpenGL OpenGL conformance tests, https://www.khronos.org/webgl/wiki/Testing/Conformance
RISC https://github.com/riscv/riscv-compliance
USB 2.0 USB20CV
USB 2.0 USBHTT
USB 3.2 USB32CV
USB TTCN3 + titan.TestPorts.USB
UEFI UEFI Self-Certification Test
UEFI FWTS
ACPI FWTS
Samba Pike SMB Test Framework
Samba Windows Protocol TestSuites
Kerberos Windows Protocol TestSuites
RDP Windows Protocol TestSuites
A Stateless Transport Tunneling Protocol for Network Virtualization (STT) stt
RFC 4960: Stream Control Transmission Protocol sctp-tests
RFC 4666: Signaling System 7 (SS7) Message Transfer Part 3 (MTP3) - User Adaptation Layer (M3UA) m3ua-testtool
RFC 3331: Signaling System 7 (SS7) Message Transfer Part 2 (MTP2) - User Adaptation Layer m2ua-testtool
RFC 3686: Signalling Connection Control Part User Adaptation Layer (SUA) sua-testtool
RFC 4165: Signaling System 7 (SS7) Message Transfer Part 2 (MTP2) - User Peer-to-Peer Adaptation Layer (M2PA) m2pa-testtool
HTTP/1.1 HTTPLint
RFC 7540: Hypertext Transfer Protocol Version 2 (HTTP/2) http2-test
RFC 3261: SIP: Session Initiation Protocol TTCN-3 test suite
RFC 1035: Domain Names - Implementation And Specification DNS Compliance Tests
TLS SSL and TLS protocol test suite
IPsec TTCN3 + titan.ProtocolModules.IPsec
TCP Some Testing Tools for TCP Implementors
TCP Network Semantics
TCP packetdrill testcases
UDP packetdrill testcases
TCP TTCN-3 and intel/net-test-suites, zephyrproject-rtos/net-tools, echo server
SMTP smtpscript
MQTT Eclipse IoT-Testware
POSIX.1 https://github.com/managarm/mlibc/tree/master/tests
POSIX.1 openposixsuite
POSIX.1 libc benchmark
POSIX.1 https://git.musl-libc.org/cgit/libc-testsuite/
POSIX.1 (regex) https://github.com/kokke/tiny-regex-c/tree/master/tests
POSIX.1 IEEE 1003.3-1991 - IEEE Standard for Information Technology–Test Methods for Measuring Conformance to POSIX®
POSIX.1 https://wiki.haskell.org/Regex_Posix
POSIX.1 https://github.com/mpieuchot/libpthread-regress
POSIX.1 https://sortix.org/os-test/
POSIX.1 Open Posix Tests
POSIX.1 POSIX regex, re_bench
POSIX.1 PCTS:151-2, POSIX Test Suite
POSIX.1 https://git.musl-libc.org/cgit/libc-testsuite/
POSIX.1 https://github.com/linux-test-project/ltp/tree/master/testcases/openposixtestsuite/conformance/interfaces
POSIX.1 malloc benchmark
POSIX.1 mimalloc tests
IEEE 754 fpspy (paper)
IEEE 754 Floating-Point Arithmetic Test Programs
IEEE 754 fpgen
Zoned Device ATA Command Set (ZAC) https://zonedstorage.io/tests/zbc-tests/
RFC 1738: Uniform Resource Locators (URL) Hypothesis + lark-grammars
RFC 2396: Uniform Resource Identifiers (URI): Generic Syntax Hypothesis + lark-grammars
RFC 2397: The “data” URL scheme Hypothesis + lark-grammars
RFC 5545: Internet Calendaring and Scheduling Core Object Specification (iCalendar) Hypothesis + lark-grammars
RFC 822: Standard for the Format of ARPA Internet Text Messages Hypothesis + lark-grammars
Robots Exclusion Protocol Hypothesis + lark-grammars
Robots Exclusion Protocol robotstxt-spec-test
TAP 13 Hypothesis + lark-grammars
YAML Hypothesis + lark-grammars
YAML Hypothesis + lark-grammars
JSON Schema JSON-Schema-Test-Suite
TOML toml-test
RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format https://github.com/nst/JSONTestSuite
Markdown mdtest, MarkdownTest
S3 S3 compatibility tests
OPDS (Open Publication Distribution System) OPDS Test Catalog
Теги: softwaretestingfeeden