Reads sensor packets from MQTT, validates and batches them, and submits completed batches to the remote Engine.
More...
#include <Collector.hpp>
|
| | Collector () |
| | Reads configuration from environment variables and initializes all subcomponents.
|
| |
| | ~Collector ()=default |
| |
| void | run () |
| | Connects to the broker and blocks until 'q' is pressed or a fatal error occurs.
|
| |
| void | stop () |
| | Drains buffered data, disconnects from the broker, and joins the submitter thread.
|
| |
|
| void | retrieve_api_key () |
| | Reads API_KEY from the environment and stores it in api_key.
|
| |
| void | process_packet (const std::string &payload) |
| | Validates the payload, writes it to the current batch, and flushes to disk when the batch is full.
|
| |
| void | finalize_drain () |
| | Flushes any buffered data and enqueues it for submission; no-op after the first call.
|
| |
| void | submitter_loop () |
| | Worker thread that dequeues batch files and submits them via SCP/SSH, keeping the MQTT callback non-blocking.
|
| |
| void | retriever_loop () |
| | Worker thread that periodically downloads results and SLURM logs from the cluster.
|
| |
|
| std::string | api_key |
| | API key for authenticating with the MQTT broker.
|
| |
| std::string | batch_filepath |
| | Path where batches will be temporarily saved.
|
| |
| std::string | sensors_filepath |
| | Path to the sensors configuration.
|
| |
| std::string | rules_filepath |
| | Path to the rules configuration.
|
| |
| std::atomic< bool > | fatal_error_ {false} |
| | Set to true by the MQTT or submitter thread to trigger shutdown.
|
| |
| int | batch_size |
| | Packet count per batch; computed as optimal_batch_size() = 10 × num_sensors.
|
| |
| std::string | log_path |
| | Path to the collector's log file.
|
| |
| std::thread | submitter_thread_ |
| | Thread running submitter_loop.
|
| |
| std::thread | retriever_thread_ |
| | Thread running retriever_loop.
|
| |
| std::queue< std::string > | job_queue_ |
| | Batch file paths waiting for submission.
|
| |
| std::mutex | queue_mutex_ |
| | Protects job_queue_ and stopping_.
|
| |
| std::condition_variable | queue_cv_ |
| | Wakes the submitter thread when a batch is enqueued or stopping_ is set.
|
| |
| std::condition_variable | retriever_cv_ |
| | Wakes the retriever thread early when stopping_ is set; shares queue_mutex_.
|
| |
| bool | stopping_ = false |
| | Signals submitter_loop to exit once job_queue_ is empty; guarded by queue_mutex_.
|
| |
| std::atomic< bool > | draining_ {false} |
| | Set when shutdown begins; tells process_packet to flush after the current row closes.
|
| |
| std::atomic< bool > | drain_done_ {false} |
| | Set once finalize_drain has run; causes process_packet to discard late packets.
|
| |
| std::mutex | drain_mutex_ |
| | Guards the drain_cv_ wait in stop().
|
| |
| std::condition_variable | drain_cv_ |
| | Notified by finalize_drain to unblock stop()'s drain wait.
|
| |
| std::unique_ptr< Connector > | broker_connector |
| | Manages the MQTT connection.
|
| |
| std::unique_ptr< BatchWriter > | batch_writer |
| | Buffers incoming packets and writes complete batches to disk.
|
| |
| std::unique_ptr< JobScheduler > | job_scheduler |
| | Dispatches batches to the remote Engine via SCP/SSH.
|
| |
| std::unique_ptr< mqtt::async_client > | mqtt_client |
| | The Paho MQTT asynchronous client.
|
| |
| mqtt::connect_options | connOpts |
| | Options for the MQTT connection.
|
| |
| std::unique_ptr< Validator > | validator |
| | Validates incoming packets against expected sensors.
|
| |
Reads sensor packets from MQTT, validates and batches them, and submits completed batches to the remote Engine.
◆ Collector()
Reads configuration from environment variables and initializes all subcomponents.
- Exceptions
-
◆ ~Collector()
| Collector::~Collector |
( |
| ) |
|
|
default |
◆ finalize_drain()
| void Collector::finalize_drain |
( |
| ) |
|
|
private |
Flushes any buffered data and enqueues it for submission; no-op after the first call.
◆ process_packet()
| void Collector::process_packet |
( |
const std::string & |
payload | ) |
|
|
private |
Validates the payload, writes it to the current batch, and flushes to disk when the batch is full.
- Parameters
-
| payload | The raw JSON payload received from the broker. |
◆ retrieve_api_key()
| void Collector::retrieve_api_key |
( |
| ) |
|
|
private |
Reads API_KEY from the environment and stores it in api_key.
- Exceptions
-
◆ retriever_loop()
| void Collector::retriever_loop |
( |
| ) |
|
|
private |
Worker thread that periodically downloads results and SLURM logs from the cluster.
Interval is read from RETRIEVE_INTERVAL_SECONDS (default: 60 s).
◆ run()
Connects to the broker and blocks until 'q' is pressed or a fatal error occurs.
◆ stop()
Drains buffered data, disconnects from the broker, and joins the submitter thread.
◆ submitter_loop()
| void Collector::submitter_loop |
( |
| ) |
|
|
private |
Worker thread that dequeues batch files and submits them via SCP/SSH, keeping the MQTT callback non-blocking.
◆ api_key
| std::string Collector::api_key |
|
private |
API key for authenticating with the MQTT broker.
◆ batch_filepath
| std::string Collector::batch_filepath |
|
private |
Path where batches will be temporarily saved.
◆ batch_size
| int Collector::batch_size |
|
private |
Packet count per batch; computed as optimal_batch_size() = 10 × num_sensors.
◆ batch_writer
Buffers incoming packets and writes complete batches to disk.
◆ broker_connector
| std::unique_ptr<Connector> Collector::broker_connector |
|
private |
Manages the MQTT connection.
◆ connOpts
| mqtt::connect_options Collector::connOpts |
|
private |
Options for the MQTT connection.
◆ drain_cv_
| std::condition_variable Collector::drain_cv_ |
|
private |
Notified by finalize_drain to unblock stop()'s drain wait.
◆ drain_done_
| std::atomic<bool> Collector::drain_done_ {false} |
|
private |
Set once finalize_drain has run; causes process_packet to discard late packets.
◆ drain_mutex_
| std::mutex Collector::drain_mutex_ |
|
private |
Guards the drain_cv_ wait in stop().
◆ draining_
| std::atomic<bool> Collector::draining_ {false} |
|
private |
Set when shutdown begins; tells process_packet to flush after the current row closes.
◆ fatal_error_
| std::atomic<bool> Collector::fatal_error_ {false} |
|
private |
Set to true by the MQTT or submitter thread to trigger shutdown.
◆ job_queue_
| std::queue<std::string> Collector::job_queue_ |
|
private |
Batch file paths waiting for submission.
◆ job_scheduler
Dispatches batches to the remote Engine via SCP/SSH.
◆ log_path
| std::string Collector::log_path |
|
private |
Path to the collector's log file.
◆ mqtt_client
| std::unique_ptr<mqtt::async_client> Collector::mqtt_client |
|
private |
The Paho MQTT asynchronous client.
◆ queue_cv_
| std::condition_variable Collector::queue_cv_ |
|
private |
Wakes the submitter thread when a batch is enqueued or stopping_ is set.
◆ queue_mutex_
| std::mutex Collector::queue_mutex_ |
|
private |
Protects job_queue_ and stopping_.
◆ retriever_cv_
| std::condition_variable Collector::retriever_cv_ |
|
private |
Wakes the retriever thread early when stopping_ is set; shares queue_mutex_.
◆ retriever_thread_
| std::thread Collector::retriever_thread_ |
|
private |
Thread running retriever_loop.
◆ rules_filepath
| std::string Collector::rules_filepath |
|
private |
Path to the rules configuration.
◆ sensors_filepath
| std::string Collector::sensors_filepath |
|
private |
Path to the sensors configuration.
◆ stopping_
| bool Collector::stopping_ = false |
|
private |
Signals submitter_loop to exit once job_queue_ is empty; guarded by queue_mutex_.
◆ submitter_thread_
| std::thread Collector::submitter_thread_ |
|
private |
Thread running submitter_loop.
◆ validator
| std::unique_ptr<Validator> Collector::validator |
|
private |
Validates incoming packets against expected sensors.
The documentation for this class was generated from the following files: