DevSecOps

Rustinel: Open-Source Endpoint Detection Engine for Linux, Windows, and macOS

Mohamed Magdy

Mohamed Magdy

— 5 min read
Share
Rustinel: Open-Source Endpoint Detection Engine for Linux, Windows, and macOS
Rustinel endpoint detection architecture showing telemetry collection and rule evaluation across Windows, Linux, and macOS.

The Detection Gap: Transparency Matters

Commercial endpoint detection and response (EDR) tools are black boxes. You buy a binary, run it with elevated privileges, and trust it works. You don't know what rules it's actually using. You don't know if detections are accurate or if it's generating false positives. You can't modify rules to match your environment.

Rustinel flips that model. It's a transparent endpoint detection engine where you see exactly what's happening, control the rules, and can inspect or modify the detection logic. Open source. Auditable. Extensible.

The goal is simple: give blue teams, researchers, and detection engineers a detection engine they can inspect, run, test, and extend without trusting a vendor's black box.

What Rustinel Does

Rustinel collects native host telemetry and evaluates it against detection rules in real time. The pipeline:

  1. Collect: Gather OS telemetry from native APIs (ETW on Windows, eBPF on Linux, ESF on macOS)
  2. Normalize: Convert platform-specific events into a unified model (ECS format)
  3. Detect: Evaluate events against Sigma rules, YARA patterns, and IOC files
  4. Alert: Write ECS NDJSON formatted alerts to logs
  5. Respond (optional): Terminate malicious processes

Key differentiator: Rustinel runs with elevated privileges and processes untrusted rule files. It's built assuming rules and IOCs could be malicious, with security boundaries in place.

Telemetry Collection by Platform

Windows: Event Tracing for Windows (ETW)

Rustinel uses ETW to capture kernel-level events without a kernel driver. No driver means easier deployment and fewer security risks. Events captured include:

  • Process creation and termination
  • File system operations
  • Network connections
  • Registry modifications
  • DLL loading and module events
  • Thread creation

Similar to Sysmon's telemetry, but natively via Windows APIs.

Linux: eBPF (Extended Berkeley Packet Filter)

eBPF programs run in the kernel with minimal overhead, capturing:

  • Process creation, execution, and termination
  • Network connections (TCP/UDP)
  • File system operations (open, read, write)
  • System calls from user space
  • Network packet inspection

Rustinel runs with minimal capabilities (CAP_BPF, CAP_NET_ADMIN, CAP_SYS_RESOURCE), not full root. This reduces the blast radius if compromised.

macOS: Endpoint Security Framework

macOS Endpoint Security Framework (ESF) provides:

  • Process execution events
  • File operations
  • Network connections
  • Code execution authorization points

⚠️ macOS support is experimental pending the Endpoint Security client entitlement.

Detection Rules: Sigma, YARA, IOCs

Rustinel evaluates three types of detection logic:

Sigma Rules

Sigma is a rule format for security event patterns. Rustinel parses YAML Sigma rules and evaluates them against normalized events. Example:

title: Suspicious Process Creation
logsource:
  product: linux
  service: sysmon
detection:
  selection:
    EventID: 1
    ParentImage|endswith: '/tmp'
  condition: selection

Sigma rules are platform-agnostic; the same rule works on Windows, Linux, and macOS because Rustinel normalizes events.

YARA Rules

YARA pattern matching for malware detection. Rustinel evaluates YARA rules against:

  • Running process memory (optional memory scanning)
  • Files on disk
  • Network traffic (future)

IOC Files

Indicator of Compromise lists (IP addresses, domains, file hashes, process names). Rustinel matches events against:

  • Network IOCs (source/destination IPs, domains)
  • File IOCs (hashes, paths, file names)
  • Process IOCs (process names, command lines)

Hot-Reload Rules Without Restart

Edit a Sigma rule, update an IOC file, or modify YARA patterns while Rustinel is running. The engine detects changes, recompiles the affected detector, and swaps it in atomically. No downtime. No missed events.

# Rustinel is running
vim /opt/rustinel/rules/sigma/custom.yml

# Changes detected and applied automatically

This is critical for production environments where restarting detection is expensive.

Installation and Quick Start

Linux

# Download release
curl -LO https://github.com/Karib0u/rustinel/releases/latest/download/rustinel-1.0.2-x86_64-unknown-linux-musl.tar.gz
tar xzf rustinel-1.0.2-x86_64-unknown-linux-musl.tar.gz
cd rustinel-1.0.2-x86_64-unknown-linux-musl

# Run (requires elevated privileges for eBPF)
sudo ./rustinel run

Windows

# Download release
Invoke-WebRequest -Uri "https://github.com/Karib0u/rustinel/releases/latest/download/rustinel-1.0.2-x86_64-pc-windows-msvc.zip" -OutFile rustinel.zip
Expand-Archive rustinel.zip -DestinationPath .
cd rustinel-1.0.2-x86_64-pc-windows-msvc

# Run (requires administrator)
.\rustinel.exe run

macOS

# Download release (Apple Silicon)
curl -LO https://github.com/Karib0u/rustinel/releases/latest/download/rustinel-1.0.2-aarch64-apple-darwin.tar.gz
tar xzf rustinel-1.0.2-aarch64-apple-darwin.tar.gz
cd rustinel-1.0.2-aarch64-apple-darwin

# Run as root
sudo ./rustinel run

From Source

git clone https://github.com/Karib0u/rustinel.git
cd rustinel
cargo build --release  # Rust 1.92+ required
sudo ./target/release/rustinel run

Alert Output: ECS NDJSON

Alerts are written to logs/alerts.json.<date> in ECS (Elastic Common Schema) NDJSON format. Each line is a valid JSON event:

{"@timestamp":"2026-06-08T10:30:45.123Z","event":{"action":"network-connection","provider":"rustinel"},"process":{"pid":1234,"name":"curl"},"destination":{"ip":"8.8.8.8","port":443},"detection":{"rule":"suspicious_dns_query"}}

This format integrates directly with:

  • Elasticsearch/Kibana
  • Splunk
  • Splunk Enterprise Security
  • OpenSearch
  • Other SIEM platforms

Rustinel includes demo configurations for Elastic and Splunk.

Detection Capabilities: What It Sees

Rustinel is a transparent detection engine, not a silver bullet. Here's what you get:

Strong detections:

  • Process creation patterns (parent-child relationships)
  • Network connections (destination filtering, suspicious ports)
  • Privilege escalation attempts
  • Credential access (keyloggers, password dumpers)
  • Persistence mechanisms (cron jobs, startup scripts, registry modification)
  • Lateral movement (unusual process spawning, remote command execution)

Limitations:

  • No kernel driver = can't detect rootkits that hide at kernel level
  • User-space malware can manipulate its own events before Rustinel sees them
  • Memory scanning requires additional configuration and privileges
  • Depends on rule quality; bad rules = missed detections

Architecture: Minimal Privilege

On Linux, Rustinel runs with specific capabilities instead of full root:

CAP_BPF          # Load and attach eBPF programs
CAP_NET_ADMIN    # Attach tc/xdp programs, read socket metadata
CAP_SYS_RESOURCE # Raise memlock limit required by eBPF maps

This reduces the blast radius if the Rustinel process is compromised. An attacker would have limited privilege escalation paths compared to a full-root process.

Roadmap: What's Coming

Near term:

  • DNS response enrichment (parse DNS answers for query results)
  • AUTH-mode prevention (block process execution before it happens via Endpoint Security Framework)
  • Periodic YARA sweeps (background scans of running processes independent of creation events)
  • Memory scanning improvements

Detection gaps being addressed:

  • Multi-interface network capture (instead of single interface)
  • YARA memory scanning for process memory inspection
  • More example Sigma rules for common detection scenarios

Use Cases

Blue Teams

Run Rustinel in your lab or production environment to:

  • Detect malware and attack patterns in real time
  • Understand what your endpoints are actually doing
  • Test and tune Sigma rules before deploying to commercial tools
  • Correlate detections across Windows, Linux, and macOS endpoints

Researchers

  • Analyze malware behavior across platforms
  • Build custom detection rules for novel attacks
  • Study endpoint detection blindspots
  • Contribute back to the open-source detection community

Detection Engineers

  • Build Sigma rules in a transparent environment
  • Test detections without waiting for vendor support
  • Modify rules to match your environment's specific behaviors
  • Hot-reload rules without restarting detection

Threat Hunters

  • Run ad-hoc detection jobs on endpoints
  • Correlate events manually using ECS NDJSON output
  • Integrate with your SIEM without proprietary connectors
  • Audit exactly what's being collected and detected

Solo-Maintained, Best-Effort Support

Rustinel is maintained by Karib0u, a security researcher with expertise in malware analysis, DFIR, and threat hunting. It's a solo project with no dedicated security team. Response times are best-effort: expect acknowledgement within a few days, resolution within a few weeks.

Report security issues for:

  • Rule or IOC file parsing leading to code execution or privilege escalation
  • eBPF program issues exploitable from user space
  • Detection bypass via crafted process or network events

Community and Future

Rustinel has 364 GitHub stars and active community engagement. Feedback from users:

"Exactly what I was looking for. Already deployed it and testing it."
"What you built with the ETW telemetry + Sigma/YARA combo is exactly what we need right now."
"Great work, using it in my lab and really liking it."

The project demonstrates that open-source, transparent endpoint detection is viable and valuable. As more organizations move away from black-box tools, projects like Rustinel show an alternative: see the code, understand the detection, own the rules.

References

Read more