Built-In Test (BIT) System Benefits

Overview

The Built-In Test (BIT) system is a comprehensive, modular, and extensible testing framework designed for Linux-based embedded systems. It provides automated hardware and software diagnostics through a plugin-based architecture, ensuring system reliability, integrity, and operational readiness.


Key Benefits

🛡️ Enhanced System Reliability

  • Proactive fault detection before failures impact operations
  • Continuous monitoring of critical system resources
  • Early warning system for potential hardware degradation

⚡ Reduced Downtime

  • Automated diagnostics eliminate manual testing overhead
  • Scheduled test execution runs in the background
  • Immediate fault notification enables rapid response

📊 Comprehensive Coverage

  • Hardware validation (PCI, USB, GPIO, Serial, CAN, Ethernet)
  • System resource monitoring (CPU, Memory, Disk)
  • Data path verification (TCP, CAN bus, Serial ports)

🔌 Plug-and-Play Architecture

  • Dynamic plugin loading at runtime
  • Configuration-driven test parameters
  • No core system modifications required to add new tests

📋 Systemd Integration

  • Runs as a native Linux service
  • Automatic startup on boot
  • Standard service management (systemctl start/stop/status)

Test Types & Coverage

The BIT system implements three primary test categories aligned with industry-standard Built-In Test methodologies:

Power-On BIT (PBIT)

Tests executed once at system startup to verify hardware integrity before normal operations begin.

Continuous BIT (CBIT)

Tests that run periodically during normal operations to detect runtime faults and resource exhaustion.

Factory BIT (FBIT)

Comprehensive hardware tests used during manufacturing and maintenance to validate all system interfaces.


Test Summary Table

Power-On BIT (PBIT) Tests

Test NameDescriptionFrequency
pbit_bsp_versionValidates BSP version matches expected configurationOnce at startup
pbit_canValidates CAN bus interface availabilityOnce at startup
pbit_checksumVerifies file checksums for integrityOnce at startup
pbit_cpu_coresValidates expected CPU core countOnce at startup
pbit_cpu_usageChecks initial CPU utilization is within limitsOnce at startup
pbit_disk_healthValidates disk SMART health statusOnce at startup
pbit_disk_usageChecks disk space availability at bootOnce at startup
pbit_dmesg_checkScans dmesg for critical errorsOnce at startup
pbit_ethernetValidates Ethernet interface availabilityOnce at startup
pbit_file_checksumVerifies critical system file integrityOnce at startup
pbit_firewall_configurationValidates firewall rules are correctly configuredOnce at startup
pbit_gpioValidates GPIO pin availability and stateOnce at startup
pbit_gpu_loadingChecks GPU driver and initial loadingOnce at startup
pbit_memory_usageValidates available memory at startupOnce at startup
pbit_pci_whitelistValidates PCI devices against approved whitelistOnce at startup
pbit_permissions_verificationValidates file/directory permissionsOnce at startup
pbit_power_testChecks power supply status and voltagesOnce at startup
pbit_selinux_apparmor_statusValidates security module statusOnce at startup
pbit_serial_portsValidates serial port availabilityOnce at startup
pbit_ssh_configurationValidates SSH security configurationOnce at startup
pbit_syslog_analysisScans syslog for critical startup errorsOnce at startup
pbit_temperatureChecks initial thermal readingsOnce at startup
pbit_usb_whitelistValidates USB devices against approved whitelistOnce at startup

Continuous BIT (CBIT) Tests

Test NameDescriptionFrequency
cbit_bsp_versionMonitors BSP version consistencyConfigurable
cbit_canMonitors CAN bus health and error countersConfigurable
cbit_checksumPeriodic file integrity verificationConfigurable
cbit_cpu_coresMonitors CPU core availabilityConfigurable
cbit_cpu_usageMonitors CPU utilization over rolling windowEvery 1s
cbit_disk_healthMonitors disk SMART attributesConfigurable
cbit_disk_usageMonitors disk usage against thresholdsEvery 30s
cbit_dmesgMonitors kernel message bufferConfigurable
cbit_dmesg_checkContinuous dmesg error detectionConfigurable
cbit_ethernetMonitors Ethernet link status and errorsConfigurable
cbit_ethernet_statusMonitors Ethernet connection stateConfigurable
cbit_firewall_configurationMonitors firewall rule integrityConfigurable
cbit_gpioMonitors GPIO state changesConfigurable
cbit_gpu_loadingMonitors GPU utilizationConfigurable
cbit_memory_usageTracks memory consumption patternsConfigurable
cbit_pci_whitelistMonitors PCI device changesConfigurable
cbit_permissionsMonitors file permission changesConfigurable
cbit_permissions_verificationContinuous permission validationConfigurable
cbit_power_consumptionMonitors power draw and efficiencyConfigurable
cbit_selinux_apparmor_statusMonitors security module stateConfigurable
cbit_serial_portsMonitors serial port availabilityConfigurable
cbit_ssh_configurationMonitors SSH configuration changesConfigurable
cbit_syslog_analysisContinuous syslog error monitoringConfigurable
cbit_temperatureMonitors thermal readings and trendsConfigurable
cbit_usb_whitelistMonitors USB device changesConfigurable

Factory BIT (FBIT) Tests

Test NameDescriptionFrequency
fbit_can_dataTests CAN bus transmit/receive functionalityConfigurable
fbit_gpio_dataValidates GPIO input/output functionalityConfigurable
fbit_pciPCI device enumeration and verificationConfigurable
fbit_serial_dataValidates serial port loopback communicationConfigurable
fbit_ssdSSD read/write performance validationConfigurable
fbit_system_dataSystem information collectionConfigurable
fbit_tcp_dataTests Ethernet data paths using iPerfConfigurable
fbit_usbUSB device read/write verificationConfigurable
fbit_videoVideo output verificationConfigurable

Total: 57 Built-In Tests covering hardware validation, system monitoring, and security verification.


Plugin Architecture

The BIT system is built on a powerful plugin architecture that enables seamless extensibility without modifying core system code.

How It Works

The BIT system uses Zenoh as its messaging middleware, enabling real-time test result distribution to monitoring clients and seamless integration with GVA (Generic Vehicle Architecture) systems via protocol bridging.

Message Flow Architecture

flowchart LR subgraph Server["🖥️ BIT Server"] BM["BIT Manager"] PBIT["PBIT Plugins"] CBIT["CBIT Plugins"] FBIT["FBIT Plugins"] end subgraph Zenoh["🌐 Zenoh Network"] ZS["Zenoh Session"] end subgraph Clients["📱 BIT Clients"] GUI["BIT GUI"] CLI["BIT CLI"] end subgraph Gateway["🔗 BIT Gateway"] ZDG["Zenoh-DDS Bridge"] end subgraph GVA["🎖️ GVA Domain (DDS)"] HUMS["Health & Usage
Monitoring Service"] end PBIT -->|TestResult| BM CBIT -->|TestResult| BM FBIT -->|TestResult| BM BM -->|"Zenoh Publish
bit/{hostname}/PBIT"| ZS BM -->|"Zenoh Publish
bit/{hostname}/CBIT"| ZS BM -->|"Zenoh Publish
bit/{hostname}/FBIT"| ZS ZS -->|"Subscribe
bit/+/+"| GUI ZS -->|"Subscribe
bit/+/+"| CLI ZS -->|"Subscribe
bit/+/+"| ZDG ZDG -->|"DDS Publish
GVA::HealthStatus"| HUMS

Detailed Message Sequence

sequenceDiagram participant Plugin as Test Plugin participant BM as BIT Manager participant Zenoh as Zenoh Router participant GUI as BIT GUI participant Gateway as BIT Gateway participant HUMS as GVA HUMS (DDS) Note over Plugin,HUMS: Test Execution & Result Distribution Plugin->>BM: Execute Test BM->>BM: Encode BuiltInTestResult (Protobuf) BM->>Zenoh: Publish to bit/{hostname}/CBIT par Parallel Distribution Zenoh-->>GUI: Forward Result GUI->>GUI: Display in Dashboard and Zenoh-->>Gateway: Forward Result Gateway->>Gateway: Zenoh → DDS Transform Gateway->>HUMS: Publish GVA::HealthStatus HUMS->>HUMS: Update Vehicle Health Record end Note over GUI,HUMS: Real-time monitoring on both systems

Protocol Bridge Details

ComponentProtocol InProtocol OutPurpose
BIT ManagerInternalZenohPublishes test results
BIT GUIZenohDisplays real-time results
BIT CLIZenohCommand-line monitoring
BIT GatewayZenohDDS (GVA)Bridges to vehicle systems
GVA HUMSDDSHealth & Usage Monitoring

Plugin Features

FeatureDescription
Dynamic LoadingPlugins are loaded as shared libraries (.so) at runtime
Trait-Based InterfaceConsistent TestRun and TestDetails traits for all plugins
Configuration FilesTOML-based configuration per test (/etc/bit/*.toml)
Version TrackingEach plugin reports version and build timestamp
Logging CallbacksUnified logging infrastructure across all tests
Run CountersAutomatic tracking of pass/fail statistics

Creating a New Plugin

Creating a custom BIT plugin is straightforward:

  1. Implement the TestRun trait:
impl TestRun for MyCustomTest {
    fn run(&mut self) {
        // Your test logic here
        self.base_test.status = TestStatus::Success;
    }
}
  1. Export the plugin interface:
#[no_mangle]
pub extern "C" fn create_test() -> Box<dyn TestRun> {
    Box::new(MyCustomTest::new())
}
  1. Create a configuration file (optional):
[my_custom_test]
frequency = 60
enabled = true
threshold = 90
  1. Build and deploy the shared library to /usr/local/lib/bit_manager/

Rust Language Safety Features

The BIT system is implemented in Rust, providing unparalleled safety guarantees critical for embedded and safety-critical systems.

Memory Safety

FeatureBenefit
No Null PointersOption<T> types prevent null pointer dereferences
No Buffer OverflowsBounds checking on all array/vector access
No Use-After-FreeOwnership system guarantees memory validity
No Data RacesCompile-time prevention of concurrent access bugs

Why Rust for BIT?

🔒 Compile-Time Safety

Rust catches entire categories of bugs at compile time that would cause runtime failures in C/C++:

  • Memory leaks
  • Buffer overflows
  • Race conditions
  • Null pointer dereferences

⚡ Zero-Cost Abstractions

High-level safety features compile to efficient machine code with no runtime overhead, matching C/C++ performance.

🔧 Fearless Concurrency

The ownership model enables safe multi-threaded test execution without data races:

// Thread-safe shared state with Arc<Mutex<T>>
let shared_state = Arc::new(Mutex::new(TestState::new()));

📦 Modern Dependency Management

  • Cargo package manager ensures reproducible builds
  • Automated dependency resolution
  • Built-in testing framework

🛡️ Type-Safe Configuration

Configuration parsing leverages Rust's type system to catch errors early:

let threshold: f32 = config.get("threshold")?;  // Type-checked at compile time

Safety Statistics

MetricRust vs C/C++
Memory safety bugsEliminated at compile time
CVE vulnerability classes prevented~70% of common vulnerabilities
Runtime null pointer crashesImpossible
Thread safety violationsCaught at compile time

Integration & Deployment

Systemd Service

# Enable automatic startup
sudo systemctl enable bit_manager

# Start the service
sudo systemctl start bit_manager

# Check status
sudo systemctl status bit_manager

Debian Package Installation

cargo deb -p bit_manager
sudo dpkg -i target/debian/bit_manager_*.deb

Configuration Paths

PathPurpose
/etc/bit/Test configuration files (TOML)
/usr/local/lib/bit_manager/Plugin shared libraries
/var/log/bit_manager/Log files

Inspection & Monitoring

The bit_inspect utility provides detailed information about loaded tests:

# List all available tests
bit_inspect

# Get detailed information about a specific test
bit_inspect cbit_disk_usage

Example Output:

Details:
  Long-Name              Disk utilization test
  Author                 Ross Newman <[email protected]>
  Description            Check the disk is not nearing full
  Status                 NotRun

Plugin Details:
  Plugin Name            cbit_disk_usage
  Version                1.0.0
  Date Built             2025-03-09 01:42:11
  Run Frequency          Periodic(30s)

Summary

The BIT system delivers:

Comprehensive hardware and software diagnostics
Extensible plugin architecture for custom tests
Memory-safe implementation in Rust
Industry-standard PBIT/CBIT/FBIT methodology
Native Linux/systemd integration
Configuration-driven test parameters
Real-time monitoring and logging