|
AstraLog-HPC
|
Spacecraft telemetry pipeline for the Software Engineering for HPC exam. It receives sensor data over MQTT, validates it locally, and offloads batch rule-evaluation jobs to the CINECA Galileo100 (G100) HPC cluster.
Two executables, split by where the work belongs:
The repository layout directly reflects the modular software architecture defined in the requirement analysis and specification document. The code is organized into src/ and include/ directories, with sub-components matching the main architecture blocks (Collector, Engine, Validator, RulesFactory).
As a group of four students, we implemented a hybrid parallelization strategy in the Engine running on the G100 cluster:
We adopted a bottom-up strategy for our testing, starting from the leaves of the "uses" hierarchy. The rationale behind this choice is that a bottom-up approach does not need stubs. However, it typically requires more drivers: one for each module (as in unit testing).
In practice, our testing pipeline followed these steps:
ValidatorTests, RulesFactoryTests, CollectorTests) in isolation using their own drivers.EngineTests), ensuring MPI and OpenMP worked correctly across simulated ranks.E2ETests that cover the full pipeline from the MQTT publisher to the final output generation.Collector. Every incoming packet is validated immediately (malformed JSON, schema errors, type mismatches against sensors.yaml); validation stays local because the data rate is far below one CPU's capacity. Valid packets are batched one-per-line by BatchWriter; when a batch fills, JobScheduler ships {batch, sensors.yaml, rules.json, job.slurm} to the data-mover and submits via the login node. Config is re-sent with every batch so rule/sensor changes take effect.
Engine. Reads the batch and config plus cross-batch state (stateful.txt, stepdiff.txt). RulesFactory instantiates rule objects and RuleOrdering groups them per sensor. Sensors are sharded across MPI ranks and evaluated in parallel with OpenMP within each rank; the primary-rule outcomes are then combined across ranks (MPI) and rank 0 runs the correlation rules per timestamp. Output goes to valid_data.csv / alarms.log, and cross-batch state is persisted at batch end.
Developing this application presented both technical and human challenges. Splitting the work was necessary, but interacting modules meant working on other people's code and solving their bugs, which represented a considerable cognitive expense. To keep everything organized, we simulated a real development workflow using pull requests with at least two mandatory reviewers to avoid bug propagation.
In this process, we utilized Large Language Models (LLMs) including Gemini 3.1 Pro, Claude Sonnet 4.6, and Claude Opus 4.8, alongside agents like Claude Code and GitHub Copilot. AI proved highly beneficial for polishing the English in our report, filling gaps in our C++ knowledge, and speeding up the learning curve for external libraries like simdjson and paho-mqtt. Furthermore, we systematically requested GitHub Copilot code reviews on our PRs, which substantially aided the human reviewers.
However, the use of AI introduced its own challenges. Copilot required strict human supervision since it lacked access to our domain assumptions and specifications, leading to false positives that required critical evaluation. We discovered that evaluating AI-generated code often requires more cognitive effort than writing it yourself. Additionally, due to the limited context window, we had to carefully select what code to provide in the prompt rather than dumping the entire codebase.
A major pitfall we encountered is that modern AI models produce very confident, well-formulated outputs even when incorrect. This overconfidence makes it harder to convince yourself that the model is wrong, creating extra work to debug "self-amplifying" false positives. Ultimately, we concluded that the most effective way to use AI is to express our intent and domain knowledge as clearly as possible in the prompt, ensuring that the model speeds up the writing process rather than replacing our critical thinking.
Requires CMake 3.20+ and a C++20 compiler.
The Engine runs on G100 from an Apptainer/Singularity image built from container.def (Ubuntu base + Paho + the compiled binaries):
The CINECA SSH certificate is interactive (browser OTP) and lives ~12 h, so it must be issued by a human; the scripts only use an existing cert.
One-time: scripts/make_known_hosts.sh pins the load-balanced pool host keys into config/cluster_known_hosts.
Each session:
CI auto-deploy. scripts/push_cert.sh uploads the cert to repo secrets and records CERT_PUSHED_AT. While that flag is fresh (scripts/cert_flag_ok.sh), every push to main rebuilds and redeploys automatically (.github/workflows/build_ct.yml); once it goes stale, the deploy step skips itself and CI stays green.
run.sh sources .env and takes SSH auth from the ssh-agent (or a file cert via CLUSTER_KEY_PATH). Press q to stop; results land in output/.