AstraLog-HPC
Loading...
Searching...
No Matches
AstraLog-HPC

Spacecraft telemetry pipeline for the Software Engineering for HPC exam. It receives sensor data over MQTT, validates it locally, and offloads batch rule-evaluation jobs to the CINECA Galileo100 (G100) HPC cluster.

Project Information

  • Selected Track: Full

Team Members, Roles, and Effort

  • Giulia Cavoletti — Rule Factory Engineer. Activities: Implemented RulesFactory, designed the Strategy pattern for rule evaluation, developed rule logic, and wrote unit tests for the rules components. Effort: ~100 hours.
  • Federico Pizzolato — Deployment & Job Scheduling Manager. Activities: Managed SLURM job scheduling, implemented SSH/SCP communication with CINECA G100, and built the CI/CD pipeline automation and bash deployment scripts. Effort: ~100 hours.
  • Paolo Potì — Packet Quality Supervisor. Activities: Implemented the Validator, schema validations, JSON parsing with simdjson, MQTT integration, and packet validation unit tests. Effort: ~100 hours.
  • Emma Scalvedi — AstraLog Engine Constructor. Activities: Implemented the Engine logic, BatchWriter, OutputManager, unit tests for the engine and the collector, wrote and the integration test. Effort: ~100 hours.

Technologies Used

  • Programming Language: C++20
  • Libraries & Frameworks: Paho MQTT C/C++, simdjson, jsoncpp, OpenMP, MPI (OpenMPI), and GoogleTest.
  • Build System: CMake (3.20+)

Architecture & Code Organization

Two executables, split by where the work belongs:

  • Collector (runs on the operator machine, connected to the login node) — subscribes to the MQTT broker, validates each packet, accumulates valid packets into batches, and submits one SLURM job per batch to G100.
  • Engine (runs as a SLURM job on G100) — loads a batch plus the sensor/rule config, evaluates the rules in parallel, and writes results back.
MQTT broker ──> Collector ──(scp + sbatch)──> G100 ──> Engine ──(scp back)──> output/
│ │
├─ Validator (schema / type checks) ├─ RulesFactory (Strategy pattern)
├─ BatchWriter (per-timestamp rows) └─ OutputManager (CSV / alarms / state)
└─ JobScheduler (ssh/scp, SLURM)

The repository layout directly reflects the modular software architecture defined in the requirement analysis and specification document. The code is organized into src/ and include/ directories, with sub-components matching the main architecture blocks (Collector, Engine, Validator, RulesFactory).

Distribution and Parallelization

As a group of four students, we implemented a hybrid parallelization strategy in the Engine running on the G100 cluster:

  • MPI (Inter-node): We shard the different sensors across multiple MPI ranks.
  • OpenMP (Intra-node): Within each rank, the evaluation of rules for a single sensor's data is parallelized using OpenMP threads. After the primary rules are evaluated locally, the outcomes are combined across ranks using MPI reductions, and rank 0 runs the cross-sensor correlation rules.

Testing Strategy

We adopted a bottom-up strategy for our testing, starting from the leaves of the "uses" hierarchy. The rationale behind this choice is that a bottom-up approach does not need stubs. However, it typically requires more drivers: one for each module (as in unit testing).

In practice, our testing pipeline followed these steps:

  1. Single Components: We first developed and tested the individual leaf components (ValidatorTests, RulesFactoryTests, CollectorTests) in isolation using their own drivers.
  2. MPI / Subsystems: Next, we tested the parallelization layer and the engine subsystem (EngineTests), ensuring MPI and OpenMP worked correctly across simulated ranks.
  3. End-to-End (E2E): Finally, we integrated the working subsystems into the final one, performing E2ETests that cover the full pipeline from the MQTT publisher to the final output generation.

Pipeline Description

Collector. Every incoming packet is validated immediately (malformed JSON, schema errors, type mismatches against sensors.yaml); validation stays local because the data rate is far below one CPU's capacity. Valid packets are batched one-per-line by BatchWriter; when a batch fills, JobScheduler ships {batch, sensors.yaml, rules.json, job.slurm} to the data-mover and submits via the login node. Config is re-sent with every batch so rule/sensor changes take effect.

Engine. Reads the batch and config plus cross-batch state (stateful.txt, stepdiff.txt). RulesFactory instantiates rule objects and RuleOrdering groups them per sensor. Sensors are sharded across MPI ranks and evaluated in parallel with OpenMP within each rank; the primary-rule outcomes are then combined across ranks (MPI) and rank 0 runs the correlation rules per timestamp. Output goes to valid_data.csv / alarms.log, and cross-batch state is persisted at batch end.

Difficulties Faced

  • Overcome: Dealing with the interactive CINECA SSH certificate for CI/CD automation. G100 mandates interactive 2FA (browser OTP), meaning no CI runner can obtain a cert unattended. We solved this by pushing automation up to the boundary: a human provisions the certificate to open a ~12h deploy window, and the CI reuses it securely via GitHub Secrets while it remains valid.
  • Overcome: Parallelizing the rule evaluation required careful synchronization across nodes. We resolved this by using MPI reductions to safely gather intermediate states before evaluating correlation rules.
  • Overcome: Setting up the test architecture and the bottom-up strategy. Initializing the testing environment for MPI and ensuring that all subsystems were correctly isolated before integration required careful design of the test drivers and mock data files. We eventually succeeded by rigorously separating the test execution phase by phase.

Generative AI Usage & Challenges

Developing this application presented both technical and human challenges. Splitting the work was necessary, but interacting modules meant working on other people's code and solving their bugs, which represented a considerable cognitive expense. To keep everything organized, we simulated a real development workflow using pull requests with at least two mandatory reviewers to avoid bug propagation.

In this process, we utilized Large Language Models (LLMs) including Gemini 3.1 Pro, Claude Sonnet 4.6, and Claude Opus 4.8, alongside agents like Claude Code and GitHub Copilot. AI proved highly beneficial for polishing the English in our report, filling gaps in our C++ knowledge, and speeding up the learning curve for external libraries like simdjson and paho-mqtt. Furthermore, we systematically requested GitHub Copilot code reviews on our PRs, which substantially aided the human reviewers.

However, the use of AI introduced its own challenges. Copilot required strict human supervision since it lacked access to our domain assumptions and specifications, leading to false positives that required critical evaluation. We discovered that evaluating AI-generated code often requires more cognitive effort than writing it yourself. Additionally, due to the limited context window, we had to carefully select what code to provide in the prompt rather than dumping the entire codebase.

A major pitfall we encountered is that modern AI models produce very confident, well-formulated outputs even when incorrect. This overconfidence makes it harder to convince yourself that the model is wrong, creating extra work to debug "self-amplifying" false positives. Ultimately, we concluded that the most effective way to use AI is to express our intent and domain knowledge as clearly as possible in the prompt, ensuring that the model speeds up the writing process rather than replacing our critical thinking.

Repository layout

src/ Engine, Collector, Validator, RulesFactory sources
include/ public headers (mirrors src/)
config/ sensors.yaml, rules.json, job_template.slurm, cluster_known_hosts
scripts/ deploy + cert/known_hosts helpers
container.def Apptainer/Singularity image definition
tests/ GoogleTest suites
docs/ Doxygen config + architecture page (Mermaid diagrams)

Build

Requires CMake 3.20+ and a C++20 compiler.

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)
ctest --test-dir build # run the test suite

Container

The Engine runs on G100 from an Apptainer/Singularity image built from container.def (Ubuntu base + Paho + the compiled binaries):

apptainer build container.sif container.def

Deploy to G100

The CINECA SSH certificate is interactive (browser OTP) and lives ~12 h, so it must be issued by a human; the scripts only use an existing cert.

One-time: scripts/make_known_hosts.sh pins the load-balanced pool host keys into config/cluster_known_hosts.

Each session:

step-cli ssh login '<you>@mail.polimi.it' --provisioner cineca-hpc # browser OTP
scripts/push_cert.sh # push cert to repo secrets → opens the CI deploy window
scripts/deploy.sh ./container.sif # (optional) upload the image now from your machine

CI auto-deploy. scripts/push_cert.sh uploads the cert to repo secrets and records CERT_PUSHED_AT. While that flag is fresh (scripts/cert_flag_ok.sh), every push to main rebuilds and redeploys automatically (.github/workflows/build_ct.yml); once it goes stale, the deploy step skips itself and CI stays green.

Run the Collector

cp .env.example .env # fill in secrets/paths
src/Collector/run.sh # local

run.sh sources .env and takes SSH auth from the ssh-agent (or a file cert via CLUSTER_KEY_PATH). Press q to stop; results land in output/.