AstraLog-HPC
Loading...
Searching...
No Matches
Collector Class Reference

Reads sensor packets from MQTT, validates and batches them, and submits completed batches to the remote Engine. More...

#include <Collector.hpp>

Collaboration diagram for Collector:
Collaboration graph

Public Member Functions

 Collector ()
 Reads configuration from environment variables and initializes all subcomponents.
 
 ~Collector ()=default
 
void run ()
 Connects to the broker and blocks until 'q' is pressed or a fatal error occurs.
 
void stop ()
 Drains buffered data, disconnects from the broker, and joins the submitter thread.
 

Private Member Functions

void retrieve_api_key ()
 Reads API_KEY from the environment and stores it in api_key.
 
void process_packet (const std::string &payload)
 Validates the payload, writes it to the current batch, and flushes to disk when the batch is full.
 
void finalize_drain ()
 Flushes any buffered data and enqueues it for submission; no-op after the first call.
 
void submitter_loop ()
 Worker thread that dequeues batch files and submits them via SCP/SSH, keeping the MQTT callback non-blocking.
 
void retriever_loop ()
 Worker thread that periodically downloads results and SLURM logs from the cluster.
 

Private Attributes

std::string api_key
 API key for authenticating with the MQTT broker.
 
std::string batch_filepath
 Path where batches will be temporarily saved.
 
std::string sensors_filepath
 Path to the sensors configuration.
 
std::string rules_filepath
 Path to the rules configuration.
 
std::atomic< bool > fatal_error_ {false}
 Set to true by the MQTT or submitter thread to trigger shutdown.
 
int batch_size
 Packet count per batch; computed as optimal_batch_size() = 10 × num_sensors.
 
std::string log_path
 Path to the collector's log file.
 
std::thread submitter_thread_
 Thread running submitter_loop.
 
std::thread retriever_thread_
 Thread running retriever_loop.
 
std::queue< std::string > job_queue_
 Batch file paths waiting for submission.
 
std::mutex queue_mutex_
 Protects job_queue_ and stopping_.
 
std::condition_variable queue_cv_
 Wakes the submitter thread when a batch is enqueued or stopping_ is set.
 
std::condition_variable retriever_cv_
 Wakes the retriever thread early when stopping_ is set; shares queue_mutex_.
 
bool stopping_ = false
 Signals submitter_loop to exit once job_queue_ is empty; guarded by queue_mutex_.
 
std::atomic< bool > draining_ {false}
 Set when shutdown begins; tells process_packet to flush after the current row closes.
 
std::atomic< bool > drain_done_ {false}
 Set once finalize_drain has run; causes process_packet to discard late packets.
 
std::mutex drain_mutex_
 Guards the drain_cv_ wait in stop().
 
std::condition_variable drain_cv_
 Notified by finalize_drain to unblock stop()'s drain wait.
 
std::unique_ptr< Connector > broker_connector
 Manages the MQTT connection.
 
std::unique_ptr< BatchWriter > batch_writer
 Buffers incoming packets and writes complete batches to disk.
 
std::unique_ptr< JobScheduler > job_scheduler
 Dispatches batches to the remote Engine via SCP/SSH.
 
std::unique_ptr< mqtt::async_client > mqtt_client
 The Paho MQTT asynchronous client.
 
mqtt::connect_options connOpts
 Options for the MQTT connection.
 
std::unique_ptr< Validator > validator
 Validates incoming packets against expected sensors.
 

Detailed Description

Reads sensor packets from MQTT, validates and batches them, and submits completed batches to the remote Engine.

Constructor & Destructor Documentation

◆ Collector()

Collector::Collector ( )

Reads configuration from environment variables and initializes all subcomponents.

Exceptions
astralog::ConfigErrorif any required environment variable is missing or invalid.
Here is the call graph for this function:

◆ ~Collector()

Collector::~Collector ( )
default

Member Function Documentation

◆ finalize_drain()

void Collector::finalize_drain ( )
private

Flushes any buffered data and enqueues it for submission; no-op after the first call.

Here is the caller graph for this function:

◆ process_packet()

void Collector::process_packet ( const std::string &  payload)
private

Validates the payload, writes it to the current batch, and flushes to disk when the batch is full.

Parameters
payloadThe raw JSON payload received from the broker.
Here is the call graph for this function:
Here is the caller graph for this function:

◆ retrieve_api_key()

void Collector::retrieve_api_key ( )
private

Reads API_KEY from the environment and stores it in api_key.

Exceptions
astralog::ConfigErrorif the required environment variable is missing.
Here is the caller graph for this function:

◆ retriever_loop()

void Collector::retriever_loop ( )
private

Worker thread that periodically downloads results and SLURM logs from the cluster.

Interval is read from RETRIEVE_INTERVAL_SECONDS (default: 60 s).

Here is the caller graph for this function:

◆ run()

void Collector::run ( )

Connects to the broker and blocks until 'q' is pressed or a fatal error occurs.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ stop()

void Collector::stop ( )

Drains buffered data, disconnects from the broker, and joins the submitter thread.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ submitter_loop()

void Collector::submitter_loop ( )
private

Worker thread that dequeues batch files and submits them via SCP/SSH, keeping the MQTT callback non-blocking.

Here is the caller graph for this function:

Member Data Documentation

◆ api_key

std::string Collector::api_key
private

API key for authenticating with the MQTT broker.

◆ batch_filepath

std::string Collector::batch_filepath
private

Path where batches will be temporarily saved.

◆ batch_size

int Collector::batch_size
private

Packet count per batch; computed as optimal_batch_size() = 10 × num_sensors.

◆ batch_writer

std::unique_ptr<BatchWriter> Collector::batch_writer
private

Buffers incoming packets and writes complete batches to disk.

◆ broker_connector

std::unique_ptr<Connector> Collector::broker_connector
private

Manages the MQTT connection.

◆ connOpts

mqtt::connect_options Collector::connOpts
private

Options for the MQTT connection.

◆ drain_cv_

std::condition_variable Collector::drain_cv_
private

Notified by finalize_drain to unblock stop()'s drain wait.

◆ drain_done_

std::atomic<bool> Collector::drain_done_ {false}
private

Set once finalize_drain has run; causes process_packet to discard late packets.

◆ drain_mutex_

std::mutex Collector::drain_mutex_
private

Guards the drain_cv_ wait in stop().

◆ draining_

std::atomic<bool> Collector::draining_ {false}
private

Set when shutdown begins; tells process_packet to flush after the current row closes.

◆ fatal_error_

std::atomic<bool> Collector::fatal_error_ {false}
private

Set to true by the MQTT or submitter thread to trigger shutdown.

◆ job_queue_

std::queue<std::string> Collector::job_queue_
private

Batch file paths waiting for submission.

◆ job_scheduler

std::unique_ptr<JobScheduler> Collector::job_scheduler
private

Dispatches batches to the remote Engine via SCP/SSH.

◆ log_path

std::string Collector::log_path
private

Path to the collector's log file.

◆ mqtt_client

std::unique_ptr<mqtt::async_client> Collector::mqtt_client
private

The Paho MQTT asynchronous client.

◆ queue_cv_

std::condition_variable Collector::queue_cv_
private

Wakes the submitter thread when a batch is enqueued or stopping_ is set.

◆ queue_mutex_

std::mutex Collector::queue_mutex_
private

Protects job_queue_ and stopping_.

◆ retriever_cv_

std::condition_variable Collector::retriever_cv_
private

Wakes the retriever thread early when stopping_ is set; shares queue_mutex_.

◆ retriever_thread_

std::thread Collector::retriever_thread_
private

Thread running retriever_loop.

◆ rules_filepath

std::string Collector::rules_filepath
private

Path to the rules configuration.

◆ sensors_filepath

std::string Collector::sensors_filepath
private

Path to the sensors configuration.

◆ stopping_

bool Collector::stopping_ = false
private

Signals submitter_loop to exit once job_queue_ is empty; guarded by queue_mutex_.

◆ submitter_thread_

std::thread Collector::submitter_thread_
private

Thread running submitter_loop.

◆ validator

std::unique_ptr<Validator> Collector::validator
private

Validates incoming packets against expected sensors.


The documentation for this class was generated from the following files: