Accumulates incoming packets into per-timestamp rows and flushes complete batches to disk.
More...
#include <BatchWriter.hpp>
|
| | BatchWriter (int batch_size, std::string batch_filepath, const std::string &sensors_filepath, const std::string &rules_filepath) |
| | Constructs a new BatchWriter.
|
| |
| | ~BatchWriter ()=default |
| |
| void | receive (const std::string ×tamp, const std::string &sensor_id, const std::string &packet) |
| | Adds the packet to the current row; fills the slot with an error packet if packet is empty.
|
| |
| int | get_batch_size () const |
| | Returns the configured maximum batch size.
|
| |
| int | get_num_proc () const |
| | Number of MPI tasks to request: ~1 per 4 sensors, at least one.
|
| |
| bool | is_batch_full () const |
| | Returns true if the buffer has reached the batch size limit.
|
| |
| bool | row_in_progress () const |
| | Returns true when some but not all sensors have reported for the current timestamp.
|
| |
| bool | has_buffered () const |
| | Returns true if there are completed rows buffered but not yet flushed.
|
| |
| std::string | flush () |
| | Writes the buffered rows to a numbered batch file and clears the buffer.
|
| |
|
| std::string | make_error_packet (const std::string ×tamp, const std::string &sensor_id) const |
| | Returns a JSON error packet (value "E") for the given timestamp and sensor.
|
| |
| void | close_row () |
| | Flushes the current row to buffer_, padding missing sensors with error packets.
|
| |
|
| CurrentRow | current_row_ |
| | The row currently being filled.
|
| |
| std::vector< std::string > | sensor_list_ |
| | Ordered list of all configured sensor IDs.
|
| |
| std::unordered_set< std::string > | sensor_set_ |
| | Hash set of sensor IDs for O(1) membership checks.
|
| |
| size_t | num_sensors_ = 0 |
| | Total number of configured sensors.
|
| |
| std::vector< std::string > | buffer_ |
| | Completed rows waiting to be flushed to disk.
|
| |
| int | batch_size_ |
| | Maximum number of packets (not rows) before a flush is triggered.
|
| |
| int | batch_counter_ = 0 |
| | Counter for generating unique batch file names.
|
| |
| std::string | batch_filepath_ |
| | Base file path for batch output files.
|
| |
| std::unique_ptr< ConfigReader > | configurator_ |
| | Computes the default batch size from the configuration.
|
| |
Accumulates incoming packets into per-timestamp rows and flushes complete batches to disk.
A row groups all sensor readings for one timestamp; it closes once every configured sensor has reported. Missing sensors receive an error packet. A batch is ready when batch_size individual packets (num_sensors × desired_rows) have been accumulated.
◆ BatchWriter()
| BatchWriter::BatchWriter |
( |
int |
batch_size, |
|
|
std::string |
batch_filepath, |
|
|
const std::string & |
sensors_filepath, |
|
|
const std::string & |
rules_filepath |
|
) |
| |
|
inline |
Constructs a new BatchWriter.
- Parameters
-
| batch_size | The maximum number of individual packets (sensor readings) to buffer before flushing; equals num_sensors * desired_rows. |
| batch_filepath | The base file path where batches will be written. |
| sensors_filepath | The path to the sensors configuration file. |
| rules_filepath | The path to the rules configuration file. |
◆ ~BatchWriter()
| BatchWriter::~BatchWriter |
( |
| ) |
|
|
default |
◆ close_row()
| void BatchWriter::close_row |
( |
| ) |
|
|
private |
Flushes the current row to buffer_, padding missing sensors with error packets.
Called when either all sensors have reported or a new timestamp arrives before the previous row is complete (sensor skipped a tick).
◆ flush()
| std::string BatchWriter::flush |
( |
| ) |
|
Writes the buffered rows to a numbered batch file and clears the buffer.
- Returns
- The absolute path of the generated batch file.
◆ get_batch_size()
| int BatchWriter::get_batch_size |
( |
| ) |
const |
|
inline |
Returns the configured maximum batch size.
- Returns
- The batch size limit.
◆ get_num_proc()
| int BatchWriter::get_num_proc |
( |
| ) |
const |
|
inline |
Number of MPI tasks to request: ~1 per 4 sensors, at least one.
The engine shards sensors across ranks and uses OpenMP within each. Stays on one node, so the count is capped by its CPU budget (NODE_CORES / CPUS_PER_TASK); the script exports OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK so ranks don't oversubscribe. To go multi-node: scale the cap by num_nodes, pass –nodes to sbatch, and make the core counts env-configurable (they differ per partition).
- Returns
- The number of MPI tasks to request.
◆ has_buffered()
| bool BatchWriter::has_buffered |
( |
| ) |
const |
|
inline |
Returns true if there are completed rows buffered but not yet flushed.
- Returns
- True if the buffer is not empty, false otherwise.
◆ is_batch_full()
| bool BatchWriter::is_batch_full |
( |
| ) |
const |
|
inline |
Returns true if the buffer has reached the batch size limit.
- Returns
- True if full and ready to flush, false otherwise.
◆ make_error_packet()
| std::string BatchWriter::make_error_packet |
( |
const std::string & |
timestamp, |
|
|
const std::string & |
sensor_id |
|
) |
| const |
|
private |
Returns a JSON error packet (value "E") for the given timestamp and sensor.
◆ receive()
| void BatchWriter::receive |
( |
const std::string & |
timestamp, |
|
|
const std::string & |
sensor_id, |
|
|
const std::string & |
packet |
|
) |
| |
Adds the packet to the current row; fills the slot with an error packet if packet is empty.
The row is flushed into the buffer once all configured sensors have reported for the current timestamp.
- Parameters
-
| timestamp | The timestamp of the packet. |
| sensor_id | The ID of the sensor that generated the packet. |
| packet | The validated JSON packet string, or empty for an invalid reading. |
◆ row_in_progress()
| bool BatchWriter::row_in_progress |
( |
| ) |
const |
|
inline |
Returns true when some but not all sensors have reported for the current timestamp.
Used at shutdown to avoid flushing a partial row.
- Returns
- True if a row is in progress, false otherwise.
◆ batch_counter_
| int BatchWriter::batch_counter_ = 0 |
|
private |
Counter for generating unique batch file names.
◆ batch_filepath_
| std::string BatchWriter::batch_filepath_ |
|
private |
Base file path for batch output files.
◆ batch_size_
| int BatchWriter::batch_size_ |
|
private |
Maximum number of packets (not rows) before a flush is triggered.
◆ buffer_
| std::vector<std::string> BatchWriter::buffer_ |
|
private |
Completed rows waiting to be flushed to disk.
◆ configurator_
Computes the default batch size from the configuration.
◆ current_row_
The row currently being filled.
◆ num_sensors_
| size_t BatchWriter::num_sensors_ = 0 |
|
private |
Total number of configured sensors.
◆ sensor_list_
| std::vector<std::string> BatchWriter::sensor_list_ |
|
private |
Ordered list of all configured sensor IDs.
◆ sensor_set_
| std::unordered_set<std::string> BatchWriter::sensor_set_ |
|
private |
Hash set of sensor IDs for O(1) membership checks.
The documentation for this class was generated from the following files: