RE: https://hachyderm.io/@MalditoBarbudo/115603904212424537
Found the problem. It seems that if you have a long pipeline that takes long in the first run (in order of days) and a systemd timer that starts the pipeline every 24 hours, if for some reason the pipeline is still running when the timer triggers, both pipelines will fail. The newly started one because there is a pipeline still running (everything ok there). The running pipeline will continue running but silently failing to record the finished targets and branches from that point.
The error was mine because I didn't disable the timer when regenerating the whole pipeline, but the silent error, especially in long pipelines scares me. First time I saw this was after waiting 5 days for a pipeline to finish, with all the tar_progress and tar_meta telling me that there were no errors, just to discover I had to repeat 80% of the pipeline again after finishing :(
#rstats #targets #pipelines