summaryrefslogtreecommitdiff
path: root/buffer/2024-06-17.md
diff options
context:
space:
mode:
authorJasper Ras <jras@hostnet.nl>2024-11-23 11:30:28 +0100
committerJasper Ras <jras@hostnet.nl>2024-11-23 11:30:28 +0100
commit1f1423fdb8d4e2928768ffa33a6ad2b59c75e82e (patch)
tree67d94f0d821a053aca0e7c8a7ffb6c369047aa14 /buffer/2024-06-17.md
parent93d36010a9d54bca1fdcaf1f4c99e6981aaf41bc (diff)
lowercase dirs
Diffstat (limited to 'buffer/2024-06-17.md')
-rw-r--r--buffer/2024-06-17.md10
1 files changed, 10 insertions, 0 deletions
diff --git a/buffer/2024-06-17.md b/buffer/2024-06-17.md
new file mode 100644
index 0000000..144e133
--- /dev/null
+++ b/buffer/2024-06-17.md
@@ -0,0 +1,10 @@
+Investigated the last remaining error with a missing reply queue which was the likely suspect of some alerts of compute service down etc. After some digging in Kibana i noticed a compute node that was logging MessagingTimeouts. I restarted the compute service on this node and it seems to have resolved the errors.
+
+Both system test in CPH8 and LEJ1 were failing. CPH8 likely due to me restarting neutron-server earlier that morning in an attempt to fix the reply queue error.
+LEJ1 had been failing since 14th of June. My guess is due to some maintenance related issue. Both system tests are running smoothly again after cleaning up their artifacts.
+
+Meanwhile, maintenance on VPS1-LEJ1 is still on-going and causing headaches. Another password refresh of the access node caused a migration to fail during post-cleanup and I had to set the host and node of the instance to the correct node directly via MySQL. I should apply Rutgers' suggestion of saving some variables in the OS_\* env vars, rather sooner than later.
+
+Tested retention backups being scheduled, and deployed it to prod.
+
+Investigate whether we can listen for notifications by Nova so that we can listen for server deletions and remove them from periodic backups. There should be no need to delete backups for the server, as the retention policy will have them deleted after a while. \ No newline at end of file