diff options
Diffstat (limited to '2 areas/Werk')
-rw-r--r-- | 2 areas/Werk/Maintain backup-service/Backup verwijderen faalt.md | 52 | ||||
-rw-r--r-- | 2 areas/Werk/antagonist-migratie.md | 27 | ||||
-rw-r--r-- | 2 areas/Werk/feelings.md | 4 | ||||
-rw-r--r-- | 2 areas/Werk/find-knowledge.md | 4 | ||||
-rw-r--r-- | 2 areas/Werk/group.one-infra.md | 3 | ||||
-rw-r--r-- | 2 areas/Werk/interview-candidates.md | 7 | ||||
-rw-r--r-- | 2 areas/Werk/maintain-openstack/image-updater.md | 6 | ||||
-rw-r--r-- | 2 areas/Werk/maintain-openstack/operational/maintenance-log.md | 3 | ||||
-rw-r--r-- | 2 areas/Werk/maintain-openstack/operational/storage-load-2024-12-05.md | 12 | ||||
-rw-r--r-- | 2 areas/Werk/maintain-openstack/ovn.md | 1 | ||||
-rw-r--r-- | 2 areas/Werk/maintain-openstack/vps2-lej1 compute mix.md | 5 | ||||
-rw-r--r-- | 2 areas/Werk/reflection.md | 0 | ||||
-rw-r--r-- | 2 areas/Werk/self-reflection/2024.md | 26 | ||||
-rw-r--r-- | 2 areas/Werk/self-reflection/2025.md | 1 | ||||
-rw-r--r-- | 2 areas/Werk/self-reflection/what.md | 8 | ||||
-rw-r--r-- | 2 areas/Werk/team.md | 10 |
16 files changed, 169 insertions, 0 deletions
diff --git a/2 areas/Werk/Maintain backup-service/Backup verwijderen faalt.md b/2 areas/Werk/Maintain backup-service/Backup verwijderen faalt.md new file mode 100644 index 0000000..6af3202 --- /dev/null +++ b/2 areas/Werk/Maintain backup-service/Backup verwijderen faalt.md @@ -0,0 +1,52 @@ +#work #backup-service #bug + +--- +# Summary +We openen libvirt connection alleen tijdens het starten van het proces. Daarna niet meer. Dus connection was closed door iets -> elk opvolgend request faalt. + +**Oplossing** +Connections e.d openen per request. + +# Onderzoek +```shell +2025-01-06 11:00:42.760 3562 INFO goba.cmd.agent [None req-3ac13f90-ebe4-482b-82ef-fded0df9be87 - - - - -] action='delete' type='backup' task_uuid='1957362f-44ec-475b-a5a1-96b53aa8be60' +2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent [-] internal error: client socket is closed: libvirt.libvirtError: internal error: client socket is closed +2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent Traceback (most recent call last): +2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent File "/usr/lib/python3/dist-packages/goba/cmd/agent.py", line 137, in execute +2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent execute_fn(ctx) +2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent File "/usr/lib/python3/dist-packages/goba/cmd/agent.py", line 192, in execute_fn +2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent backup.delete(req, self.storage, self.libvirt) +2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent File "/usr/lib/python3/dist-packages/goba/backup.py", line 332, in delete +2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent if not libvirt_client.is_running(instance_id): +2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent File "/usr/lib/python3/dist-packages/goba/adapters/libvirt.py", line 134, in is_running +2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent dom_state = self._get_domain(instance_id).state()[0] +2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent File "/usr/lib/python3/dist-packages/libvirt.py", line 3146, in state +2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent raise libvirtError('virDomainGetState() failed') +2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent libvirt.libvirtError: internal error: client socket is closed +2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent +``` + +``` +[jasras@n14.compute.vps1-lej1 ~]$ systemctl status goba +● goba.service - Group.one OpenStack Backup Agent + Loaded: loaded (/lib/systemd/system/goba.service; enabled; vendor preset: enabled) + Active: active (running) since Thu 2024-12-12 06:26:50 UTC; 3 weeks 4 days ago + Docs: https://gitlab.group.one/groupvps/group-one-backup-agent + Main PID: 3562 (goba) + Tasks: 86 (limit: 4915) + Memory: 166.9M + CPU: 2h 23min 28.981s + CGroup: /system.slice/goba.service + ├─ 3562 /usr/bin/python3 /usr/bin/goba --config-file /etc/goba/goba.conf + └─14368 /usr/bin/python3 /usr/bin/privsep-helper --config-file /etc/goba/goba.conf --privsep_context goba.privsep.file_admin_pctxt --privsep_sock_path /tmp/tmpjfi8jt6b/privsep.sock +``` +Er draait hier nog een privsep-helper + +Na restart: +``` + CGroup: /system.slice/goba.service + └─30932 /usr/bin/python3 /usr/bin/goba --config-file /etc/goba/goba.conf +``` + +Tasks slagen nu wel. + diff --git a/2 areas/Werk/antagonist-migratie.md b/2 areas/Werk/antagonist-migratie.md new file mode 100644 index 0000000..c04a00b --- /dev/null +++ b/2 areas/Werk/antagonist-migratie.md @@ -0,0 +1,27 @@ +Migrate VMs + +Not managed by customers, only Antagonist managed +Most of it is backoffice, only few are front facing + +Ovirt (KVM) + + +Possible to change IPs? + +Difficulty: private networking between hosts + databases only ones in the private network + + Should be possible to switch IPs + + +Outsourced routers to DC provider, more difficult to integrate networks + +Most of the things can be redeployed, Rutger notes that maybe all they need is access + +We can use their left-over IP space + +Features +- requires internal network, so want to create networks/routers +- HA/automatic failover? -> Not yet +- Anti-affinity rules? Dont want all the LBs on the same (failing) node + diff --git a/2 areas/Werk/feelings.md b/2 areas/Werk/feelings.md new file mode 100644 index 0000000..c2126bf --- /dev/null +++ b/2 areas/Werk/feelings.md @@ -0,0 +1,4 @@ +Realisatie tevreden met baan. Veel vrijheid, niet al te gekke werkdruk, goed salaris. +Ik ben tevreden met mijn baan bij group.one, ik heb veel vrijheid en de werkdruk is niet al te hoog. Het salaris en toekomstperspectief ben ik blij mee.=g + +De externer entwicklers in PM beginnen vermoeiend te worden. Leveren geen of weinig informatie aan. Het zijn N00Bs.
\ No newline at end of file diff --git a/2 areas/Werk/find-knowledge.md b/2 areas/Werk/find-knowledge.md new file mode 100644 index 0000000..4d5a31c --- /dev/null +++ b/2 areas/Werk/find-knowledge.md @@ -0,0 +1,4 @@ +A lot if not all knowledge is kept in files and repositories usually codified. A critical skill is knowing what to and how to look for things. +Rather ask questions that help you to figure out or find something then ask directly for the result. + +This is a skill for work.
\ No newline at end of file diff --git a/2 areas/Werk/group.one-infra.md b/2 areas/Werk/group.one-infra.md new file mode 100644 index 0000000..f3d4013 --- /dev/null +++ b/2 areas/Werk/group.one-infra.md @@ -0,0 +1,3 @@ + +[[TODO]] +I want to have an overview of systems and what they are responsible for. Example, what is OneHOP or OneHome and what do they do. I plan to write that down in this document.
\ No newline at end of file diff --git a/2 areas/Werk/interview-candidates.md b/2 areas/Werk/interview-candidates.md new file mode 100644 index 0000000..08e2f89 --- /dev/null +++ b/2 areas/Werk/interview-candidates.md @@ -0,0 +1,7 @@ +- Ben je bekend met scrum, agile, kanban, jira etc? + + +Technische vragen: +threading, race conditions, deadlocks +message queue +- Kun je uitleggen hoe een binary search werkt
\ No newline at end of file diff --git a/2 areas/Werk/maintain-openstack/image-updater.md b/2 areas/Werk/maintain-openstack/image-updater.md new file mode 100644 index 0000000..274d16f --- /dev/null +++ b/2 areas/Werk/maintain-openstack/image-updater.md @@ -0,0 +1,6 @@ +https://gitlab.group.one/groupvps/openstack-image-updater + +Updater runs in CI. + +`images.yaml`; contains which images +`cloud_images.yaml`; can override params for all images in a specific cloud, sadly cannot override per specific image. diff --git a/2 areas/Werk/maintain-openstack/operational/maintenance-log.md b/2 areas/Werk/maintain-openstack/operational/maintenance-log.md new file mode 100644 index 0000000..79c8a7a --- /dev/null +++ b/2 areas/Werk/maintain-openstack/operational/maintenance-log.md @@ -0,0 +1,3 @@ +### 2024-12-10 +- manually updated instance record host/node for instance `10924c62-7f0f-4df1-9dd8-9108e3cb0764` +- suspended guest: virsh dompmwakeup and try again diff --git a/2 areas/Werk/maintain-openstack/operational/storage-load-2024-12-05.md b/2 areas/Werk/maintain-openstack/operational/storage-load-2024-12-05.md new file mode 100644 index 0000000..6931d2d --- /dev/null +++ b/2 areas/Werk/maintain-openstack/operational/storage-load-2024-12-05.md @@ -0,0 +1,12 @@ +Vraag uitgezet bij Allan voor meer informatie. +Allan: het komt al 1.5 maand voor; https://group-onecom.slack.com/archives/C02FT9KEFNH/p1729863978525299 + +Uit dat bericht blijkt dat Jerry zou kijken of het Acronis is. + +Jeroen gecontacteerd om ff te vragen of ze dat al gedaan hebben; jeroen denkt dat het niet door Acronis komt omdat dat vooral read ops zijn en verspreid over de nacht. + +Allan stuurt graph met kantteking: "seems to involve alot of LUNs": `https://prometheus2.env.vps1-cph8.one.com/graph?g0.expr=count(lun_write_ops%20%3E%20(lun_write_ops%20offset%2020m%20%2B%20100))&g0.tab=0&g0.display_mode=lines&g0.show_exemplars=0&g0.range_input=30m&g0.end_input=2024-12-05%2005%3A47%3A35&g0.moment_input=2024-12-05%2005%3A47%3A35` + +Jeroen dacht mogelijk mysql dumps van shared, maar zou ook verspreid moeten zijn: ging kijken wanneer die gdraaid hadden, niks meer over gehoord. + +O.b.v LUN graph heb ik wat LUNs gepakt en bleek allemaal managed vps te zijn; Jeroen op onderzoek naar wat daar binnen rond die tijd gebeurd. Verdenkt PSA crons.
\ No newline at end of file diff --git a/2 areas/Werk/maintain-openstack/ovn.md b/2 areas/Werk/maintain-openstack/ovn.md new file mode 100644 index 0000000..b70a4d8 --- /dev/null +++ b/2 areas/Werk/maintain-openstack/ovn.md @@ -0,0 +1 @@ +https://dani.foroselectronica.es/ovn-where-is-my-packet-665/g
\ No newline at end of file diff --git a/2 areas/Werk/maintain-openstack/vps2-lej1 compute mix.md b/2 areas/Werk/maintain-openstack/vps2-lej1 compute mix.md new file mode 100644 index 0000000..cb41fae --- /dev/null +++ b/2 areas/Werk/maintain-openstack/vps2-lej1 compute mix.md @@ -0,0 +1,5 @@ +VPS2-LEJ1 is a mixed bag of shared and local storage; +nodes 1-8 are BOTH shared and local storage +nodes 9-10 are exclusively shared storage + +.. wap
\ No newline at end of file diff --git a/2 areas/Werk/reflection.md b/2 areas/Werk/reflection.md new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/2 areas/Werk/reflection.md diff --git a/2 areas/Werk/self-reflection/2024.md b/2 areas/Werk/self-reflection/2024.md new file mode 100644 index 0000000..ba627c2 --- /dev/null +++ b/2 areas/Werk/self-reflection/2024.md @@ -0,0 +1,26 @@ +I need to become more goal-oriented: set a clear goal with a clear end and work towards that. +I'm currently working to improve my skill in note-taking to help achieve current and determine new goals. +I'm doing this by setting up a "second brain" using ideas from the book "Building a second brain" by Tiago Forte. + +--- + +**General Performance and Achievements** + Describe your overall performance throughout the year and highlight any achievements you are especially proud of + +Looking back I feel that my focus this year has been impacted due to a series of events happening outside of work in my private life, such as moving to a new house and the death of a close friend. Besides that I feel I was often doubting myself which made me indecisive at times. + +The backup service which I had mostly worked on developed far slower than I liked. Due to above but also because so many parts of this were completely new to me. I think that I could have done a better job at learning each of them by separating it into smaller "toy" projects and learning each part before putting it together. For example how libraries such as oslo.messaging and taskflow worked. Besides that I also feel I didn't always do a great job on including the rest of the team. +During the last quarter of this year I did feel I was able to improve and work with a clearer mind and work more closely together with Erik; communicating more frequently and effectively, especially with the snapshots. + +I feel proud that, even though it was not progressing as quickly or organized as I liked, I was able to set-up the backup service and put it to production and seeing few operational problems as of yet. I feel especially proud of how we were able to utilize the project to quickly implement the snapshot feature that was suddenly requested. + +**Opportunities to become even better** + Which areas do you perceive as your strongest? + +I think I'm always eager to develop a deep understanding of how things technically work, abstract thinking, I have a sense of responsibility and am always prepared to help out where needed. + + Which areas would you like to improve in? + +I am currently trying to improve my productivity by setting up a note taking system. I will continue trying to improve this. I plan to also do a little self "performance review" at the end of each week to help me keep track of things happening during the year. + +I think in a broad sense I need to become better at setting clear goals that have a clear end which I can work toward. Both in work related things such as projects, but also in personal things such as setting up a note taking system. diff --git a/2 areas/Werk/self-reflection/2025.md b/2 areas/Werk/self-reflection/2025.md new file mode 100644 index 0000000..e6d283e --- /dev/null +++ b/2 areas/Werk/self-reflection/2025.md @@ -0,0 +1 @@ +# week 01 diff --git a/2 areas/Werk/self-reflection/what.md b/2 areas/Werk/self-reflection/what.md new file mode 100644 index 0000000..154e0d6 --- /dev/null +++ b/2 areas/Werk/self-reflection/what.md @@ -0,0 +1,8 @@ +Do a weekly self reflection and keep them documented in this folder. I intend to use these for the yearly performance review. + +Things to ask myself every week: +- What went well? +- What didn't go well? +- Were there any achievements this week that i'm proud of? +- Did I learn something? +-
\ No newline at end of file diff --git a/2 areas/Werk/team.md b/2 areas/Werk/team.md new file mode 100644 index 0000000..191485e --- /dev/null +++ b/2 areas/Werk/team.md @@ -0,0 +1,10 @@ +# Technical Details +Maximum disk size local storage defined in QCOW2 image. +# SOP +DHCP ports gone -> openstack subnet set --no-dhcp, openstack subnet set --dhpc. (toggle) -> they should be recreated. (https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/2015377) +## Maintenance +Announcing maintenance: +* Maintenance that requires customer interaction: at least 14 days ahead of time +* Regular maintenance with few seconds downtime at most: at least 7 days ahead of time +* Emergency maintenance: as soon as possible and preferably before doing anything +Notify MCS: mcs-support@group.one |