AstraLog-HPC
Loading...
Searching...
No Matches
BatchWriter Class Reference

Accumulates incoming packets into per-timestamp rows and flushes complete batches to disk. More...

#include <BatchWriter.hpp>

Collaboration diagram for BatchWriter:
Collaboration graph

Classes

struct  CurrentRow
 Tracks the state of the row currently being built. More...
 

Public Member Functions

 BatchWriter (int batch_size, std::string batch_filepath, const std::string &sensors_filepath, const std::string &rules_filepath)
 Constructs a new BatchWriter.
 
 ~BatchWriter ()=default
 
void receive (const std::string &timestamp, const std::string &sensor_id, const std::string &packet)
 Adds the packet to the current row; fills the slot with an error packet if packet is empty.
 
int get_batch_size () const
 Returns the configured maximum batch size.
 
int get_num_proc () const
 Number of MPI tasks to request: ~1 per 4 sensors, at least one.
 
bool is_batch_full () const
 Returns true if the buffer has reached the batch size limit.
 
bool row_in_progress () const
 Returns true when some but not all sensors have reported for the current timestamp.
 
bool has_buffered () const
 Returns true if there are completed rows buffered but not yet flushed.
 
std::string flush ()
 Writes the buffered rows to a numbered batch file and clears the buffer.
 

Private Member Functions

std::string make_error_packet (const std::string &timestamp, const std::string &sensor_id) const
 Returns a JSON error packet (value "E") for the given timestamp and sensor.
 
void close_row ()
 Flushes the current row to buffer_, padding missing sensors with error packets.
 

Private Attributes

CurrentRow current_row_
 The row currently being filled.
 
std::vector< std::string > sensor_list_
 Ordered list of all configured sensor IDs.
 
std::unordered_set< std::string > sensor_set_
 Hash set of sensor IDs for O(1) membership checks.
 
size_t num_sensors_ = 0
 Total number of configured sensors.
 
std::vector< std::string > buffer_
 Completed rows waiting to be flushed to disk.
 
int batch_size_
 Maximum number of packets (not rows) before a flush is triggered.
 
int batch_counter_ = 0
 Counter for generating unique batch file names.
 
std::string batch_filepath_
 Base file path for batch output files.
 
std::unique_ptr< ConfigReader > configurator_
 Computes the default batch size from the configuration.
 

Detailed Description

Accumulates incoming packets into per-timestamp rows and flushes complete batches to disk.

A row groups all sensor readings for one timestamp; it closes once every configured sensor has reported. Missing sensors receive an error packet. A batch is ready when batch_size individual packets (num_sensors × desired_rows) have been accumulated.

Constructor & Destructor Documentation

◆ BatchWriter()

BatchWriter::BatchWriter ( int  batch_size,
std::string  batch_filepath,
const std::string &  sensors_filepath,
const std::string &  rules_filepath 
)
inline

Constructs a new BatchWriter.

Parameters
batch_sizeThe maximum number of individual packets (sensor readings) to buffer before flushing; equals num_sensors * desired_rows.
batch_filepathThe base file path where batches will be written.
sensors_filepathThe path to the sensors configuration file.
rules_filepathThe path to the rules configuration file.

◆ ~BatchWriter()

BatchWriter::~BatchWriter ( )
default

Member Function Documentation

◆ close_row()

void BatchWriter::close_row ( )
private

Flushes the current row to buffer_, padding missing sensors with error packets.

Called when either all sensors have reported or a new timestamp arrives before the previous row is complete (sensor skipped a tick).

Here is the call graph for this function:
Here is the caller graph for this function:

◆ flush()

std::string BatchWriter::flush ( )

Writes the buffered rows to a numbered batch file and clears the buffer.

Returns
The absolute path of the generated batch file.
Here is the caller graph for this function:

◆ get_batch_size()

int BatchWriter::get_batch_size ( ) const
inline

Returns the configured maximum batch size.

Returns
The batch size limit.

◆ get_num_proc()

int BatchWriter::get_num_proc ( ) const
inline

Number of MPI tasks to request: ~1 per 4 sensors, at least one.

The engine shards sensors across ranks and uses OpenMP within each. Stays on one node, so the count is capped by its CPU budget (NODE_CORES / CPUS_PER_TASK); the script exports OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK so ranks don't oversubscribe. To go multi-node: scale the cap by num_nodes, pass –nodes to sbatch, and make the core counts env-configurable (they differ per partition).

Returns
The number of MPI tasks to request.

◆ has_buffered()

bool BatchWriter::has_buffered ( ) const
inline

Returns true if there are completed rows buffered but not yet flushed.

Returns
True if the buffer is not empty, false otherwise.
Here is the caller graph for this function:

◆ is_batch_full()

bool BatchWriter::is_batch_full ( ) const
inline

Returns true if the buffer has reached the batch size limit.

Returns
True if full and ready to flush, false otherwise.
Here is the caller graph for this function:

◆ make_error_packet()

std::string BatchWriter::make_error_packet ( const std::string &  timestamp,
const std::string &  sensor_id 
) const
private

Returns a JSON error packet (value "E") for the given timestamp and sensor.

Here is the caller graph for this function:

◆ receive()

void BatchWriter::receive ( const std::string &  timestamp,
const std::string &  sensor_id,
const std::string &  packet 
)

Adds the packet to the current row; fills the slot with an error packet if packet is empty.

The row is flushed into the buffer once all configured sensors have reported for the current timestamp.

Parameters
timestampThe timestamp of the packet.
sensor_idThe ID of the sensor that generated the packet.
packetThe validated JSON packet string, or empty for an invalid reading.
Here is the call graph for this function:
Here is the caller graph for this function:

◆ row_in_progress()

bool BatchWriter::row_in_progress ( ) const
inline

Returns true when some but not all sensors have reported for the current timestamp.

Used at shutdown to avoid flushing a partial row.

Returns
True if a row is in progress, false otherwise.
Here is the caller graph for this function:

Member Data Documentation

◆ batch_counter_

int BatchWriter::batch_counter_ = 0
private

Counter for generating unique batch file names.

◆ batch_filepath_

std::string BatchWriter::batch_filepath_
private

Base file path for batch output files.

◆ batch_size_

int BatchWriter::batch_size_
private

Maximum number of packets (not rows) before a flush is triggered.

◆ buffer_

std::vector<std::string> BatchWriter::buffer_
private

Completed rows waiting to be flushed to disk.

◆ configurator_

std::unique_ptr<ConfigReader> BatchWriter::configurator_
private

Computes the default batch size from the configuration.

◆ current_row_

CurrentRow BatchWriter::current_row_
private

The row currently being filled.

◆ num_sensors_

size_t BatchWriter::num_sensors_ = 0
private

Total number of configured sensors.

◆ sensor_list_

std::vector<std::string> BatchWriter::sensor_list_
private

Ordered list of all configured sensor IDs.

◆ sensor_set_

std::unordered_set<std::string> BatchWriter::sensor_set_
private

Hash set of sensor IDs for O(1) membership checks.


The documentation for this class was generated from the following files: