Node Manager/Data Processor

1. Real-time File-to-WebSocket Streaming

  File Watching with Rust NIF:

  - filewatchernif (native/filewatchernif/src/lib.rs:1-1216) uses Linux inotify for ultra-low

  latency file monitoring

  - Watches hourly blockchain data files (e.g., /hl/data/node_fills_by_block/hourly/20250825/14)

  - Detects file modifications in real-time and reads only new data (tracks offset: lib.rs:716-767)

  - Sends data immediately to Elixir via NIF messaging (lib.rs:751-763)

  Data Flow Architecture:

  1. File Events → Rust NIF (lib.rs:666-687 handles MODIFY events)

  2. NIF → FileProducer (file_producer.ex:60-104 processes file data)

  3. FileProducer → Broadway Pipeline (batches and processes)

  4. Data Handlers → Phoenix PubSub (broadcasts to topics)

  5. PubSub → WebSocket Channels (fills_channel.ex:95-118 streams to clients)

  Latency Tracking:

  The system tracks end-to-end latency from file write to client delivery

  (fills_channel.ex:100-108), measuring microsecond-level performance.

  2. Memory-Mapped File Querying

  ABCI State Memory Mapping:

  - abci_reader NIF (native/abci_reader/src/lib.rs:1-1573) loads MessagePack blockchain state files

  into memory

  - Zero-copy access to account data, orderbooks, and contract state

  - Instant queries without database ingestion (lib.rs:48-71 - account info, lib.rs:427-453 -

  orderbook data)

  File Memory Mapping Engine:

  - file_mmap_nif (native/file_mmap_nif/src/engine.rs:1-553) creates memory-mapped indexes of

  blockchain files

  - Block-level indexing (engine.rs:307-374) builds BTreeMap for O(log n)

   lookups

  - Multi-file spanning (engine.rs:445-480) queries across different data types simultaneously

  - Persistent caching (engine.rs:124-170) saves indexes to disk for instant startup

  Query Performance:

  - Direct memory access to specific blocks without parsing entire files

  - Range queries (engine.rs:509-520) efficiently traverse block ranges

  - Multi-type queries return all data types for a given block (engine.rs:445-480)

  Key Performance Features:

  - Sub-millisecond file detection via native inotify

  - Incremental file reading (only new data, tracked by byte offset)

  - Zero-copy memory mapping for instant blockchain state queries

  - Concurrent processing with configurable Broadway pipelines

  - Smart buffering (file_producer.ex:149-182) balances latency vs throughput

  - Filtered subscriptions (fills_channel.ex:16-86) reduce bandwidth via targeted PubSub topics

  The architecture enables microsecond-latency streaming from file writes to WebSocket clients,

  while providing instant access to historical blockchain data via memory-mapped file queries.

https://github.com/skedzior/Hackathon