Built-In Test (BIT) System Benefits
Overview
The Built-In Test (BIT) system is a comprehensive, modular, and extensible testing framework designed for Linux-based embedded systems. It provides automated hardware and software diagnostics through a plugin-based architecture, ensuring system reliability, integrity, and operational readiness.
Key Benefits
🛡️ Enhanced System Reliability
- Proactive fault detection before failures impact operations
- Continuous monitoring of critical system resources
- Early warning system for potential hardware degradation
⚡ Reduced Downtime
- Automated diagnostics eliminate manual testing overhead
- Scheduled test execution runs in the background
- Immediate fault notification enables rapid response
📊 Comprehensive Coverage
- Hardware validation (PCI, USB, GPIO, Serial, CAN, Ethernet)
- System resource monitoring (CPU, Memory, Disk)
- Data path verification (TCP, CAN bus, Serial ports)
🔌 Plug-and-Play Architecture
- Dynamic plugin loading at runtime
- Configuration-driven test parameters
- No core system modifications required to add new tests
📋 Systemd Integration
- Runs as a native Linux service
- Automatic startup on boot
- Standard service management (
systemctl start/stop/status)
Test Types & Coverage
The BIT system implements three primary test categories aligned with industry-standard Built-In Test methodologies:
Power-On BIT (PBIT)
Tests executed once at system startup to verify hardware integrity before normal operations begin.
Continuous BIT (CBIT)
Tests that run periodically during normal operations to detect runtime faults and resource exhaustion.
Factory BIT (FBIT)
Comprehensive hardware tests used during manufacturing and maintenance to validate all system interfaces.
Test Summary Table
Power-On BIT (PBIT) Tests
| Test Name | Description | Frequency |
|---|---|---|
pbit_bsp_version | Validates BSP version matches expected configuration | Once at startup |
pbit_can | Validates CAN bus interface availability | Once at startup |
pbit_checksum | Verifies file checksums for integrity | Once at startup |
pbit_cpu_cores | Validates expected CPU core count | Once at startup |
pbit_cpu_usage | Checks initial CPU utilization is within limits | Once at startup |
pbit_disk_health | Validates disk SMART health status | Once at startup |
pbit_disk_usage | Checks disk space availability at boot | Once at startup |
pbit_dmesg_check | Scans dmesg for critical errors | Once at startup |
pbit_ethernet | Validates Ethernet interface availability | Once at startup |
pbit_file_checksum | Verifies critical system file integrity | Once at startup |
pbit_firewall_configuration | Validates firewall rules are correctly configured | Once at startup |
pbit_gpio | Validates GPIO pin availability and state | Once at startup |
pbit_gpu_loading | Checks GPU driver and initial loading | Once at startup |
pbit_memory_usage | Validates available memory at startup | Once at startup |
pbit_pci_whitelist | Validates PCI devices against approved whitelist | Once at startup |
pbit_permissions_verification | Validates file/directory permissions | Once at startup |
pbit_power_test | Checks power supply status and voltages | Once at startup |
pbit_selinux_apparmor_status | Validates security module status | Once at startup |
pbit_serial_ports | Validates serial port availability | Once at startup |
pbit_ssh_configuration | Validates SSH security configuration | Once at startup |
pbit_syslog_analysis | Scans syslog for critical startup errors | Once at startup |
pbit_temperature | Checks initial thermal readings | Once at startup |
pbit_usb_whitelist | Validates USB devices against approved whitelist | Once at startup |
Continuous BIT (CBIT) Tests
| Test Name | Description | Frequency |
|---|---|---|
cbit_bsp_version | Monitors BSP version consistency | Configurable |
cbit_can | Monitors CAN bus health and error counters | Configurable |
cbit_checksum | Periodic file integrity verification | Configurable |
cbit_cpu_cores | Monitors CPU core availability | Configurable |
cbit_cpu_usage | Monitors CPU utilization over rolling window | Every 1s |
cbit_disk_health | Monitors disk SMART attributes | Configurable |
cbit_disk_usage | Monitors disk usage against thresholds | Every 30s |
cbit_dmesg | Monitors kernel message buffer | Configurable |
cbit_dmesg_check | Continuous dmesg error detection | Configurable |
cbit_ethernet | Monitors Ethernet link status and errors | Configurable |
cbit_ethernet_status | Monitors Ethernet connection state | Configurable |
cbit_firewall_configuration | Monitors firewall rule integrity | Configurable |
cbit_gpio | Monitors GPIO state changes | Configurable |
cbit_gpu_loading | Monitors GPU utilization | Configurable |
cbit_memory_usage | Tracks memory consumption patterns | Configurable |
cbit_pci_whitelist | Monitors PCI device changes | Configurable |
cbit_permissions | Monitors file permission changes | Configurable |
cbit_permissions_verification | Continuous permission validation | Configurable |
cbit_power_consumption | Monitors power draw and efficiency | Configurable |
cbit_selinux_apparmor_status | Monitors security module state | Configurable |
cbit_serial_ports | Monitors serial port availability | Configurable |
cbit_ssh_configuration | Monitors SSH configuration changes | Configurable |
cbit_syslog_analysis | Continuous syslog error monitoring | Configurable |
cbit_temperature | Monitors thermal readings and trends | Configurable |
cbit_usb_whitelist | Monitors USB device changes | Configurable |
Factory BIT (FBIT) Tests
| Test Name | Description | Frequency |
|---|---|---|
fbit_can_data | Tests CAN bus transmit/receive functionality | Configurable |
fbit_gpio_data | Validates GPIO input/output functionality | Configurable |
fbit_pci | PCI device enumeration and verification | Configurable |
fbit_serial_data | Validates serial port loopback communication | Configurable |
fbit_ssd | SSD read/write performance validation | Configurable |
fbit_system_data | System information collection | Configurable |
fbit_tcp_data | Tests Ethernet data paths using iPerf | Configurable |
fbit_usb | USB device read/write verification | Configurable |
fbit_video | Video output verification | Configurable |
Total: 57 Built-In Tests covering hardware validation, system monitoring, and security verification.
Plugin Architecture
The BIT system is built on a powerful plugin architecture that enables seamless extensibility without modifying core system code.
How It Works
The BIT system uses Zenoh as its messaging middleware, enabling real-time test result distribution to monitoring clients and seamless integration with GVA (Generic Vehicle Architecture) systems via protocol bridging.
Message Flow Architecture
Monitoring Service"] end PBIT -->|TestResult| BM CBIT -->|TestResult| BM FBIT -->|TestResult| BM BM -->|"Zenoh Publish
bit/{hostname}/PBIT"| ZS BM -->|"Zenoh Publish
bit/{hostname}/CBIT"| ZS BM -->|"Zenoh Publish
bit/{hostname}/FBIT"| ZS ZS -->|"Subscribe
bit/+/+"| GUI ZS -->|"Subscribe
bit/+/+"| CLI ZS -->|"Subscribe
bit/+/+"| ZDG ZDG -->|"DDS Publish
GVA::HealthStatus"| HUMS
Detailed Message Sequence
Protocol Bridge Details
| Component | Protocol In | Protocol Out | Purpose |
|---|---|---|---|
| BIT Manager | Internal | Zenoh | Publishes test results |
| BIT GUI | Zenoh | — | Displays real-time results |
| BIT CLI | Zenoh | — | Command-line monitoring |
| BIT Gateway | Zenoh | DDS (GVA) | Bridges to vehicle systems |
| GVA HUMS | DDS | — | Health & Usage Monitoring |
Plugin Features
| Feature | Description |
|---|---|
| Dynamic Loading | Plugins are loaded as shared libraries (.so) at runtime |
| Trait-Based Interface | Consistent TestRun and TestDetails traits for all plugins |
| Configuration Files | TOML-based configuration per test (/etc/bit/*.toml) |
| Version Tracking | Each plugin reports version and build timestamp |
| Logging Callbacks | Unified logging infrastructure across all tests |
| Run Counters | Automatic tracking of pass/fail statistics |
Creating a New Plugin
Creating a custom BIT plugin is straightforward:
- Implement the
TestRuntrait:
impl TestRun for MyCustomTest {
fn run(&mut self) {
// Your test logic here
self.base_test.status = TestStatus::Success;
}
}- Export the plugin interface:
#[no_mangle]
pub extern "C" fn create_test() -> Box<dyn TestRun> {
Box::new(MyCustomTest::new())
}- Create a configuration file (optional):
[my_custom_test]
frequency = 60
enabled = true
threshold = 90- Build and deploy the shared library to
/usr/local/lib/bit_manager/
Rust Language Safety Features
The BIT system is implemented in Rust, providing unparalleled safety guarantees critical for embedded and safety-critical systems.
Memory Safety
| Feature | Benefit |
|---|---|
| No Null Pointers | Option<T> types prevent null pointer dereferences |
| No Buffer Overflows | Bounds checking on all array/vector access |
| No Use-After-Free | Ownership system guarantees memory validity |
| No Data Races | Compile-time prevention of concurrent access bugs |
Why Rust for BIT?
🔒 Compile-Time Safety
Rust catches entire categories of bugs at compile time that would cause runtime failures in C/C++:
- Memory leaks
- Buffer overflows
- Race conditions
- Null pointer dereferences
⚡ Zero-Cost Abstractions
High-level safety features compile to efficient machine code with no runtime overhead, matching C/C++ performance.
🔧 Fearless Concurrency
The ownership model enables safe multi-threaded test execution without data races:
// Thread-safe shared state with Arc<Mutex<T>>
let shared_state = Arc::new(Mutex::new(TestState::new()));📦 Modern Dependency Management
- Cargo package manager ensures reproducible builds
- Automated dependency resolution
- Built-in testing framework
🛡️ Type-Safe Configuration
Configuration parsing leverages Rust's type system to catch errors early:
let threshold: f32 = config.get("threshold")?; // Type-checked at compile timeSafety Statistics
| Metric | Rust vs C/C++ |
|---|---|
| Memory safety bugs | Eliminated at compile time |
| CVE vulnerability classes prevented | ~70% of common vulnerabilities |
| Runtime null pointer crashes | Impossible |
| Thread safety violations | Caught at compile time |
Integration & Deployment
Systemd Service
# Enable automatic startup
sudo systemctl enable bit_manager
# Start the service
sudo systemctl start bit_manager
# Check status
sudo systemctl status bit_managerDebian Package Installation
cargo deb -p bit_manager
sudo dpkg -i target/debian/bit_manager_*.debConfiguration Paths
| Path | Purpose |
|---|---|
/etc/bit/ | Test configuration files (TOML) |
/usr/local/lib/bit_manager/ | Plugin shared libraries |
/var/log/bit_manager/ | Log files |
Inspection & Monitoring
The bit_inspect utility provides detailed information about loaded tests:
# List all available tests
bit_inspect
# Get detailed information about a specific test
bit_inspect cbit_disk_usageExample Output:
Details:
Long-Name Disk utilization test
Author Ross Newman <[email protected]>
Description Check the disk is not nearing full
Status NotRun
Plugin Details:
Plugin Name cbit_disk_usage
Version 1.0.0
Date Built 2025-03-09 01:42:11
Run Frequency Periodic(30s)Summary
The BIT system delivers:
✅ Comprehensive hardware and software diagnostics
✅ Extensible plugin architecture for custom tests
✅ Memory-safe implementation in Rust
✅ Industry-standard PBIT/CBIT/FBIT methodology
✅ Native Linux/systemd integration
✅ Configuration-driven test parameters
✅ Real-time monitoring and logging