Conformance and compliance testing
Conformance testing is an element of conformity assessment, and also known as compliance testing, or type testing - is testing or other activities that determine whether a process, product, or service complies with the requirements of a specification, technical standard, contract, or regulation.
Web standards
QUIC: A UDP-Based Multiplexed and Secure Transport
HTML (HyperText Markup Language)
WebSocket
WebCL
https://github.com/KhronosGroup/WebCL-conformance
WebRTC
https://webrtc.org/testing/conformance/
ECMAScript
ECMAScript Language Specification - https://tc39.es/ecma262/ test262
RFC 2616: Hypertext Transfer Protocol – HTTP/1.1
RFC 7540: Hypertext Transfer Protocol Version 2 (HTTP/2)
There are two test suites: http2-test and h2spec.
RFC 4918: HTTP Extensions for Web Distributed Authoring and Versioning (WebDAV)
http://www.webdav.org/neon/litmus/
RFC 1738: Uniform Resource Locators (URL)
Hypothesis + lark-grammars
RFC 2396: Uniform Resource Identifiers (URI): Generic Syntax
Hypothesis + lark-grammars
RFC 2397: The “data” URL scheme
Hypothesis + lark-grammars
RFC 5545: Internet Calendaring and Scheduling Core Object Specification (iCalendar)
Hypothesis + lark-grammars
Robots Exclusion Protocol
A file robots.txt used by websites primarily to manage crawler traffic to
site. For a long time format has differences in descriptions provided by Google
and Yandex in their documentations for webmasters. For example Yandex
describes
an option “Crawl-delay” but there is no such option in documentation provided
by Google.
In 2019 Google had an attempt to formalize
the Robots Exclusion Protocol Specification and they published a
draft, that was expired in
Jan 2, 2020. However proposed specification has a grammar that can be used for
testing parsers and producers on robots.txt files. For such purposes once can
use Hypothesis and lark-grammars or
robotstxt-spec-test.
Structured Query Language (SQL)
- sqllogictest
- sqlancer
- go-sqlancer
- sqltest
- Structured Query Language (SQL) Test Suite
- Hypothesis + lark-grammars
POSIX
- AT&T Research regex(3) regression tests
- https://github.com/pjd/pjdfstest
- https://github.com/managarm/mlibc/tree/master/tests
- Open Posix Tests
- libc benchmark
- https://git.musl-libc.org/cgit/libc-testsuite/
- Regex: https://github.com/kokke/tiny-regex-c/tree/master/tests
- IEEE 1003.3-1991 - IEEE Standard for Information Technology - Test Methods for Measuring Conformance to POSIX®
- POSIX regex
- re_bench
- https://github.com/mpieuchot/libpthread-regress
- https://sortix.org/os-test/
- PCTS:151-2, POSIX Test Suite
- Linux Test Project: https://github.com/linux-test-project/ltp/tree/master/testcases/openposixtestsuite/conformance/interfaces
- malloc benchmark
- mimalloc tests
Video
- XServer XTS - X Test Suite
- OpenGL The SGI OpenGL conformance tests
- OpenGL Piglit
- OpenCL OpenCL conformance tests
- OpenVX OpenVX conformance tests
- OpenGL OpenGL conformance tests, https://www.khronos.org/webgl/wiki/Testing/Conformance
Network
Samba
Kerberos
RDP (Remote Desktop Protocol)
memcached
A Stateless Transport Tunneling Protocol for Network Virtualization (STT)
RFC 4960: Stream Control Transmission Protocol
Signaling System 7 (SS7)
RFC 4666: Signaling System 7 (SS7) Message Transfer Part 3 (MTP3) - User Adaptation Layer (M3UA)
RFC 3331: Signaling System 7 (SS7) Message Transfer Part 2 (MTP2) - User Adaptation Layer
RFC 3686: Signalling Connection Control Part User Adaptation Layer (SUA)
RFC 4165: Signaling System 7 (SS7) Message Transfer Part 2 (MTP2) - User Peer-to-Peer Adaptation Layer (M2PA)
RFC 3261: SIP: Session Initiation Protocol
RFC 1035: Domain Names - Implementation And Specification
IPSec
- TTCN3 + titan.ProtocolModules.IPsec
TCP (Transmission Control Protocol)
- Some Testing Tools for TCP Implementors
- Network Semantics
- packetdrill testcases
- TTCN-3 and intel/net-test-suites, zephyrproject-rtos/net-tools, echo server
UDP
SMTP
SMTP smtpscript
MQTT
Data Formats
IEEE 754
Zoned Device ATA Command Set (ZAC)
https://zonedstorage.io/tests/zbc-tests/
RFC 822: Standard for the Format of ARPA Internet Text Messages
Hypothesis + lark-grammars
TAP (Test Anything Protocol)
TAP 13 TAP 14 Hypothesis + lark-grammars
YAML
YAML, Hypothesis + lark-grammars
RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format
https://json.org JSON Schema https://github.com/nst/JSONTestSuite JSON-Schema-Test-Suite
https://github.com/json-schema-org/JSON-Schema-Test-Suite
S3
OPDS
OPDS (Open Publication Distribution System) OPDS Test Catalog
Markdown
Markdown is a simple widely-used text markup format. It was intended to be as easy-to-read and easy-to-write as is feasible. Initially it had a limited syntax, that was later was extended by additional syntax (like footnotes, math support etc). There are two testsuites that allows to test parsers and generators of texts with Markdown markup: mdtest and MarkdownTest.
Hardware
ACPI
RISC
There is a RISC-V compliance testsuite - https://github.com/riscv/riscv-compliance.
There is a MicroTesk. MicroTESK is a reconfigurable and extendable framework for generating test programs for functional verification of microprocessors. MicroTESK allows automatically constructing test program generators based on formal specifications of microprocessor architectures. MicroTESK supports a wide range of architectures including RISC, CISC, VLIW, and DSP. MicroTESK supports online test program generation.
USB (Universal Serial Bus)
- USB 2.0 USB20CV
- USB 2.0 USBHTT
- USB 3.2 USB32CV
- USB TTCN3 + titan.TestPorts.USB
UEFI