From f9034731fa234a4b5efa1d2d6147fe1e798b6d36 Mon Sep 17 00:00:00 2001 From: Jasper Ras Date: Fri, 4 Apr 2025 14:31:53 +0200 Subject: vault backup: 2025-04-04 14:31:53 --- .obsidian/workspace.json | 72 ++++++++++---------- .trash/Acronis Managed Backup provisioning flow.md | 74 +++++++++++++++++++++ .trash/Ceph.md | 0 ...r is off I take most of his responsibilities.md | 1 - Acronis Managed Backup provisioning flow.md | 74 --------------------- Becoming a Senior.md | 2 + Booting a NixOS on OpenStack.md | 2 +- Ceph on devstack.md | 11 ---- EF00 partition.md | 4 +- EF02 boot partition.md | 7 -- EF02 partition.md | 7 ++ Gobs Ceph Client Key Access.md | 13 ++++ ...t will completely go DOWN due to OVN upgrade.md | 8 +++ Installing Ceph on devstack.md | 15 +++++ Installing a devstack plugin.md | 2 +- Managing Ceph on Devstack & OpenStack.md | 42 ++++++++++++ ...rvisor causes Freenet to shutdown EVERYTHING.md | 12 ++++ OVN upgrade.md | 40 +++++++++++ Overview of Ceph.md | 5 ++ Random notes on Freenet deployment.md | 8 +++ ...iting subselect into just an additional join.md | 32 +++++++++ ...e image os admin user to fix password resets.md | 77 ++++++++++++++++++++++ Using Libvirt to manage QEMU image bitmaps.md | 21 ++++++ 23 files changed, 397 insertions(+), 132 deletions(-) create mode 100644 .trash/Acronis Managed Backup provisioning flow.md create mode 100644 .trash/Ceph.md delete mode 100644 Acronis Managed Backup provisioning flow.md delete mode 100644 Ceph on devstack.md delete mode 100644 EF02 boot partition.md create mode 100644 EF02 partition.md create mode 100644 Gobs Ceph Client Key Access.md create mode 100644 I'm worried that Freenet will completely go DOWN due to OVN upgrade.md create mode 100644 Installing Ceph on devstack.md create mode 100644 Managing Ceph on Devstack & OpenStack.md create mode 100644 Network outage on single hypervisor causes Freenet to shutdown EVERYTHING.md create mode 100644 OVN upgrade.md create mode 100644 Random notes on Freenet deployment.md create mode 100644 Rewriting subselect into just an additional join.md create mode 100644 Updating the image os admin user to fix password resets.md create mode 100644 Using Libvirt to manage QEMU image bitmaps.md diff --git a/.obsidian/workspace.json b/.obsidian/workspace.json index d6443a3..f667601 100644 --- a/.obsidian/workspace.json +++ b/.obsidian/workspace.json @@ -4,67 +4,67 @@ "type": "split", "children": [ { - "id": "d7ac73dacb8efdbe", + "id": "b5921b94f3580090", "type": "tabs", "children": [ { - "id": "6bdd9a8324d3b0f0", + "id": "502f77e94f5ff48c", "type": "leaf", "state": { "type": "markdown", "state": { - "file": "Build custom modules as profiles to manage various NixOS hosts.md", + "file": "Managing Ceph on Devstack & OpenStack.md", "mode": "source", "backlinks": true, "source": false }, "icon": "lucide-file", - "title": "Build custom modules as profiles to manage various NixOS hosts" + "title": "Managing Ceph on Devstack & OpenStack" } }, { - "id": "84e32b3623ed3a00", + "id": "c10897c003c6bee7", "type": "leaf", "state": { "type": "markdown", "state": { - "file": "NixOS - Advantage of imperative container management.md", + "file": "OVN upgrade.md", "mode": "source", "backlinks": true, "source": false }, "icon": "lucide-file", - "title": "NixOS - Advantage of imperative container management" + "title": "OVN upgrade" } }, { - "id": "be0892c76b1d189a", + "id": "62d186e3d63b15e0", "type": "leaf", "state": { "type": "markdown", "state": { - "file": "Nix Inherit.md", + "file": "I'm worried that Freenet will completely go DOWN due to OVN upgrade.md", "mode": "source", "backlinks": true, "source": false }, "icon": "lucide-file", - "title": "Nix Inherit" + "title": "I'm worried that Freenet will completely go DOWN due to OVN upgrade" } }, { - "id": "1fb3c7b7af372a3c", + "id": "96a8cc05fb9592a6", "type": "leaf", "state": { "type": "markdown", "state": { - "file": "Distinction between imperative and declarative programming.md", + "file": "Network outage on single hypervisor causes Freenet to shutdown EVERYTHING.md", "mode": "source", "backlinks": true, "source": false }, "icon": "lucide-file", - "title": "Distinction between imperative and declarative programming" + "title": "Network outage on single hypervisor causes Freenet to shutdown EVERYTHING" } } ], @@ -99,7 +99,7 @@ "state": { "type": "search", "state": { - "query": "", + "query": "OVN upgrade", "matchingCase": false, "explainSearch": false, "collapseAll": false, @@ -219,35 +219,35 @@ "obsidian-excalidraw-plugin:New drawing": false } }, - "active": "1fb3c7b7af372a3c", + "active": "96a8cc05fb9592a6", "lastOpenFiles": [ - "Nix Inherit.md", - "Distinction between imperative and declarative programming.md", - "NixOS - Advantage of imperative container management.md", - "NixOS - Imperative container management.md", + "Making notes useful.md", + "Managing Ceph on Devstack & OpenStack.md", + "Network outage on single hypervisor causes Freenet to shutdown EVERYTHING.md", + "I'm worried that Freenet will completely go DOWN due to OVN upgrade.md", + "Random notes on Freenet deployment.md", + "OVN upgrade.md", + "Installing Ceph on devstack.md", + "Overview of Ceph.md", + "Gobs Ceph Client Key Access.md", + "Installing a devstack plugin.md", + "Rewriting subselect into just an additional join.md", + "Updating the image os admin user to fix password resets.md", "Freenet CPU steal graph.md", - "Test.md", + "Using Libvirt to manage QEMU image bitmaps.md", "Build custom modules as profiles to manage various NixOS hosts.md", - "EF02 boot partition.md", - "EF00 partition.md", - "Automatic configuration deployment with NixOS.md", - "Booting a NixOS on OpenStack.md", - "NixOS.md", - "Acronis Managed Backup provisioning flow.md", - "ESP", - "TODO's in Zettelkasten?.md", + "Nix Inherit.md", "Becoming a Senior.md", "A reason for becoming senior is that when Rutger is off I take most of his responsibilities.md", - "Overview of Ceph.md", - "Excalidraw/Intro to Ceph.excalidraw.md", - "Ceph on devstack.md", - "Excalidraw", - "Ideas versus facts.md", - "Installing a devstack plugin.md", - "Making notes useful.md", + "Acronis Managed Backup provisioning flow.md", + "Ceph.md", + "NixOS - Imperative container management.md", "NixOS - Declarative container management.md", "NixOS - Containers.md", "NixOS - Container networking.md", - "Practice summarizing and paraphrasing.md" + "EF02 partition.md", + "EF00 partition.md", + "ESP", + "Excalidraw" ] } \ No newline at end of file diff --git a/.trash/Acronis Managed Backup provisioning flow.md b/.trash/Acronis Managed Backup provisioning flow.md new file mode 100644 index 0000000..7f2b4f6 --- /dev/null +++ b/.trash/Acronis Managed Backup provisioning flow.md @@ -0,0 +1,74 @@ +--- +tags: + - work + - provisioning +--- +Sections marked with '!' are not found in Tevins confluence doc +## Create the tenant (customer) +CustomerManager::create($username); + +**Create tenant**: +- parent_id : parent tenant (find with api client id) (ParentFinder) +- kind : customer +- name: username + +**Bind application**: +- Tenant uuid +- BACKUP_UUID = '6e6d758d-8e74-3ae3-ac84-50eb0dff12eb' + +**Enable services** ! +- Sets "offering items" (OfferingItemName) + +**Activate customer**: +- Get pricing +- Update pricing mode to 'production' + +## Create the user (credentials) +CustomerManager::createUser(tenant, email, first_name, last_name) + +**Create user**: +- tenant id +- login : tenant username +- contact info + +**Update user policies**: + - tenant id + - trustee type: 'user' + - trustee id: user_id + - role id: 'backup_user' + +## Activate user + +**Set password**: +- user id +- password + +**Accept ToS** ! +- tenant uuid + +**Add backup plans** ! +- backup plan (AddPlans.php#53, see models/Backup.php) (see template) + - interval : 24 hrs (1 per day) + - post data command `{"command": "\/usr\/lib\/Acronis\/BackupAndRecovery\/webcppostcapture", "commandArguments": "{RESOURCE_ID}", "continueOnCommandError": false, "waitCommandComplete": true, "workingDirectory": ""}` + - pre data command `{"command": "\/usr\/lib\/Acronis\/BackupAndRecovery\/webcpprecapture", "commandArguments": "{RESOURCE_ID}", "continueOnCommandError": false, "waitCommandComplete": true, "workingDirectory": ""}` + +Make sure to store the ID! + +## Schedule backups +This can only finish successfully after Puppet has installed the Acronis Agent and that agent registered itself with Acronis. + +**Poll for a machine id** (that way we know the agent is registered) (CustomerManager.php#375) +- BackupConsole::getResources +- Filter on type == 'machine' +- getId + +We want 1 (our) backup plan active so therefore we: +**Revoke all plans** (CustomerManager.php#354) +- BackupConsole::getBackupPlans +- foreach BackupConsole::revokeBackupPlan + +**Add resources to plan** (CustomerManager.php#289) +- impersonate/authenticate +- apply backup plan: + - plan id + - machine id diff --git a/.trash/Ceph.md b/.trash/Ceph.md new file mode 100644 index 0000000..e69de29 diff --git a/A reason for becoming senior is that when Rutger is off I take most of his responsibilities.md b/A reason for becoming senior is that when Rutger is off I take most of his responsibilities.md index 442b817..d76b919 100644 --- a/A reason for becoming senior is that when Rutger is off I take most of his responsibilities.md +++ b/A reason for becoming senior is that when Rutger is off I take most of his responsibilities.md @@ -3,5 +3,4 @@ tags: - work - role --- - Whenever Rutger has vacation most people he interacts with go to me. That feels a bit weird being the "medior" in our team that also has a "senior". \ No newline at end of file diff --git a/Acronis Managed Backup provisioning flow.md b/Acronis Managed Backup provisioning flow.md deleted file mode 100644 index 7f2b4f6..0000000 --- a/Acronis Managed Backup provisioning flow.md +++ /dev/null @@ -1,74 +0,0 @@ ---- -tags: - - work - - provisioning ---- -Sections marked with '!' are not found in Tevins confluence doc -## Create the tenant (customer) -CustomerManager::create($username); - -**Create tenant**: -- parent_id : parent tenant (find with api client id) (ParentFinder) -- kind : customer -- name: username - -**Bind application**: -- Tenant uuid -- BACKUP_UUID = '6e6d758d-8e74-3ae3-ac84-50eb0dff12eb' - -**Enable services** ! -- Sets "offering items" (OfferingItemName) - -**Activate customer**: -- Get pricing -- Update pricing mode to 'production' - -## Create the user (credentials) -CustomerManager::createUser(tenant, email, first_name, last_name) - -**Create user**: -- tenant id -- login : tenant username -- contact info - -**Update user policies**: - - tenant id - - trustee type: 'user' - - trustee id: user_id - - role id: 'backup_user' - -## Activate user - -**Set password**: -- user id -- password - -**Accept ToS** ! -- tenant uuid - -**Add backup plans** ! -- backup plan (AddPlans.php#53, see models/Backup.php) (see template) - - interval : 24 hrs (1 per day) - - post data command `{"command": "\/usr\/lib\/Acronis\/BackupAndRecovery\/webcppostcapture", "commandArguments": "{RESOURCE_ID}", "continueOnCommandError": false, "waitCommandComplete": true, "workingDirectory": ""}` - - pre data command `{"command": "\/usr\/lib\/Acronis\/BackupAndRecovery\/webcpprecapture", "commandArguments": "{RESOURCE_ID}", "continueOnCommandError": false, "waitCommandComplete": true, "workingDirectory": ""}` - -Make sure to store the ID! - -## Schedule backups -This can only finish successfully after Puppet has installed the Acronis Agent and that agent registered itself with Acronis. - -**Poll for a machine id** (that way we know the agent is registered) (CustomerManager.php#375) -- BackupConsole::getResources -- Filter on type == 'machine' -- getId - -We want 1 (our) backup plan active so therefore we: -**Revoke all plans** (CustomerManager.php#354) -- BackupConsole::getBackupPlans -- foreach BackupConsole::revokeBackupPlan - -**Add resources to plan** (CustomerManager.php#289) -- impersonate/authenticate -- apply backup plan: - - plan id - - machine id diff --git a/Becoming a Senior.md b/Becoming a Senior.md index 5993f9e..64ab969 100644 --- a/Becoming a Senior.md +++ b/Becoming a Senior.md @@ -4,3 +4,5 @@ tags: - role --- This year I indicated that I would like to become senior. At the time Rutger gave a few reasons, mainly that my architectural skills are still lacking due to me going my own way during the development of gobs, making some (to Rutger) less than ideal decisions about API design and such. + +[[A reason for becoming senior is that when Rutger is off I take most of his responsibilities]] diff --git a/Booting a NixOS on OpenStack.md b/Booting a NixOS on OpenStack.md index 280b05b..d68a049 100644 --- a/Booting a NixOS on OpenStack.md +++ b/Booting a NixOS on OpenStack.md @@ -3,4 +3,4 @@ tags: - nixos - openstack --- -Booting a [[NixOS]] VM on openstack required me to partition a [[EF02 boot partition]] as well as an [[EF00 partition]] and install GRUB as a bootloader instead of systemd-boot. \ No newline at end of file +Booting a [[NixOS]] VM on openstack required me to partition a [[EF02 partition]] as well as an [[EF00 partition]] and install GRUB as a bootloader instead of systemd-boot. \ No newline at end of file diff --git a/Ceph on devstack.md b/Ceph on devstack.md deleted file mode 100644 index 5dcbc62..0000000 --- a/Ceph on devstack.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -tags: - - work - - devstack - - ceph ---- -[[Overview of Ceph]] - -https://github.com/openstack/devstack-plugin-ceph - -Added `enable_plugin devstack-plugin-ceph https://opendev.org/openstack/devstack-plugin-ceph` to local.conf. diff --git a/EF00 partition.md b/EF00 partition.md index ccd93e1..0511d20 100644 --- a/EF00 partition.md +++ b/EF00 partition.md @@ -4,4 +4,6 @@ tags: - boot - efi --- -This partition is the EFI system partition a.k.a ESP. UEFI firmware loads the bootloader from here. This partition as opposite to the [[EF02 boot partition]] contains all tools needed to boot, such as the boot loader and kernel images. \ No newline at end of file +This partition is the EFI system partition a.k.a ESP. UEFI firmware loads the bootloader from here. This partition as opposite to the [[EF02 partition]] contains all tools needed to boot, such as the boot loader and kernel images. + +Typically this partition is mounted as `/boot`. \ No newline at end of file diff --git a/EF02 boot partition.md b/EF02 boot partition.md deleted file mode 100644 index e3b1197..0000000 --- a/EF02 boot partition.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -tags: - - linux - - boot - - bios ---- -An EF02 partition is a small partition usually 1MB where the BIOS program looks for a bootloader such as GRUB. \ No newline at end of file diff --git a/EF02 partition.md b/EF02 partition.md new file mode 100644 index 0000000..e3b1197 --- /dev/null +++ b/EF02 partition.md @@ -0,0 +1,7 @@ +--- +tags: + - linux + - boot + - bios +--- +An EF02 partition is a small partition usually 1MB where the BIOS program looks for a bootloader such as GRUB. \ No newline at end of file diff --git a/Gobs Ceph Client Key Access.md b/Gobs Ceph Client Key Access.md new file mode 100644 index 0000000..d8f86c7 --- /dev/null +++ b/Gobs Ceph Client Key Access.md @@ -0,0 +1,13 @@ +--- +tags: + - work + - gobs + - ceph + - openstack +--- +[[Overview of Ceph]] + +Ceph credentials are typically stored in a keyring file under `/etc/ceph`. Nova has a a key there owned by nova/nova. +Instead of adding a new key I've added the `goba` user to the `nova` group and modified our hieradata to specify a mode which allows the group to read the keyring. + +https://gitlab.group.one/groupvps/hieradata/-/merge_requests/499/diffs diff --git a/I'm worried that Freenet will completely go DOWN due to OVN upgrade.md b/I'm worried that Freenet will completely go DOWN due to OVN upgrade.md new file mode 100644 index 0000000..fdf4aec --- /dev/null +++ b/I'm worried that Freenet will completely go DOWN due to OVN upgrade.md @@ -0,0 +1,8 @@ +--- +tags: + - work + - freenet +--- +[[OVN upgrade]] +[[Random notes on Freenet deployment]] +[[Freenet CPU steal graph]] \ No newline at end of file diff --git a/Installing Ceph on devstack.md b/Installing Ceph on devstack.md new file mode 100644 index 0000000..c3d4a83 --- /dev/null +++ b/Installing Ceph on devstack.md @@ -0,0 +1,15 @@ +--- +tags: + - devstack + - ceph + - openstack +--- +[[Overview of Ceph]] + +https://github.com/openstack/devstack-plugin-ceph + +Added `enable_plugin devstack-plugin-ceph https://opendev.org/openstack/devstack-plugin-ceph` to local.conf. + +Make sure we have [[Gobs Ceph Client Key Access]]. + +`sudo apt-get install python3-rados` on the docker image for goba. \ No newline at end of file diff --git a/Installing a devstack plugin.md b/Installing a devstack plugin.md index 1220798..afeb431 100644 --- a/Installing a devstack plugin.md +++ b/Installing a devstack plugin.md @@ -5,4 +5,4 @@ tags: --- To install a devstack plugin add the corresponding `enable_plugin [gitref]` under the `[[local|localrc]]` section. -This way I successfully installed [[Ceph on devstack]] +This way I successfully installed [[Installing Ceph on devstack]] diff --git a/Managing Ceph on Devstack & OpenStack.md b/Managing Ceph on Devstack & OpenStack.md new file mode 100644 index 0000000..cad83e5 --- /dev/null +++ b/Managing Ceph on Devstack & OpenStack.md @@ -0,0 +1,42 @@ +--- +tags: + - ceph + - devstack + - openstack +--- +[[Installing Ceph on devstack]] + +The pool names are: +- volumes for cinder volumes +- images for glance images +- vms for nova "local storage" rbd volumes +- backups for cinder volume backups + +https://docs.ceph.com/en/reef/rbd/rados-rbd-cmds/ + +``` +ubuntu@gobs-devstack:~$ sudo rbd ls volumes +volume-5b6cc251-8f3e-4572-a8b9-52efa390ebc3 +volume-6f5e81dd-40f3-42f9-aad3-afcf5696387c +volume-ac39d5be-9606-4711-948e-e76c035e2a25 +``` + +Specify the pool in the `info` command: +``` +ubuntu@gobs-devstack:~$ sudo rbd info volumes/volume-6f5e81dd-40f3-42f9-aad3-afcf5696387c +rbd image 'volume-6f5e81dd-40f3-42f9-aad3-afcf5696387c': + size 10 GiB in 2560 objects + order 22 (4 MiB objects) + snapshot_count: 0 + id: 17d385edc7a94 + block_name_prefix: rbd_data.17d385edc7a94 + format: 2 + features: layering, exclusive-lock, object-map, fast-diff + op_features: + flags: + create_timestamp: Mon Mar 31 12:56:46 2025 + access_timestamp: Mon Mar 31 12:56:46 2025 + modify_timestamp: Mon Mar 31 12:56:46 2025 + parent: volumes/volume-5b6cc251-8f3e-4572-a8b9-52efa390ebc3@snapshot-3d1f148a-5fe4-4df2-87c3-4272565b82c2 + overlap: 10 GiB +``` \ No newline at end of file diff --git a/Network outage on single hypervisor causes Freenet to shutdown EVERYTHING.md b/Network outage on single hypervisor causes Freenet to shutdown EVERYTHING.md new file mode 100644 index 0000000..d342ab9 --- /dev/null +++ b/Network outage on single hypervisor causes Freenet to shutdown EVERYTHING.md @@ -0,0 +1,12 @@ +--- +tags: + - work + - freenet +--- +[[Freenet CPU steal graph]] +[[Random notes on Freenet deployment]] +[[I'm worried that Freenet will completely go DOWN due to OVN upgrade]] + +Recently we had added some hypervisors but one of them had invalid MTU set on the bond. Due to... me testing there, so rutger didn't reboot it. +During maintenance we started migrating workload onto it and their network started failing. +As a response to this Freenet ***automatically*** hit the **PANIC** switch and systemctl stopped about everything in ALL AVAILABILITY ZONES. \ No newline at end of file diff --git a/OVN upgrade.md b/OVN upgrade.md new file mode 100644 index 0000000..8874ef4 --- /dev/null +++ b/OVN upgrade.md @@ -0,0 +1,40 @@ +--- +tags: + - work + - ovn + - upgrade +--- +https://docs.ovn.org/en/latest/intro/install/ovn-upgrades.html + +**Fail-safe upgrade procedure** +The ansible playbook does **NOT** encode the fail-safe procedure in such a way that it prevents us from skipping versions. It is **mandatory** that we ourselves correctly decide the version to upgrade to. + +**Is the issue resolved with group.one- prefixes on unit files and /etc/default?** +https://gitlab.group.one/groupvps/ovn-builder/-/merge_requests/11/diffs + +It seems to have been fixed using overrides in debian/rules that move stuff to the expected places after building. + +*After testing it looks like this is all done correctly now* + + +**Ansible playbook** +https://gitlab.group.one/groupvps/ansible/-/merge_requests/56/diffs + +- Makes a backup of OVN db's on networking nodes + - Creates files on the local system under /tmp + - Creates files on the remote system under /var/.ovn-backups +- Pins the controller version on both compute and networking nodes + - `ovs-vsctl set open_vswitch . external_ids:ovn-match-northd-version=true` +- Upgrades ovn-central, ovn-common and ovn-host on network nodes + - Installs packages + - restarts ovn-controller, ovn-northd, ovn-ovsdb-server-sb/nb + - Checks ovsdb-server is running + - Checks the socket file for existence +- Upgrades ovn controller on compute nodes + - Installs package + - restart ovn-controller + +**Invocation** +``` +ansible-playbook playbooks/failsafe_install_ovn.yml --limit vps_testpod --extra-vars "ovn_search_version=22.03.8+20250403.1251.35813e0b.systems.jammy1" +``` \ No newline at end of file diff --git a/Overview of Ceph.md b/Overview of Ceph.md index 0826f82..5a5505f 100644 --- a/Overview of Ceph.md +++ b/Overview of Ceph.md @@ -1,3 +1,8 @@ +--- +tags: + - ceph + - storage +--- Ceph provides Block Storage, Object Storage as well as shared filesystem. ![[Intro to Ceph.excalidraw]] diff --git a/Random notes on Freenet deployment.md b/Random notes on Freenet deployment.md new file mode 100644 index 0000000..e028053 --- /dev/null +++ b/Random notes on Freenet deployment.md @@ -0,0 +1,8 @@ +--- +tags: + - work + - freenet +--- +Uses [[Overview of Ceph]] as a volume backing store. Mehmet Top is the primary contact for this Ceph deployment. + +Their mbox VM's also connect to Quobyte storage. \ No newline at end of file diff --git a/Rewriting subselect into just an additional join.md b/Rewriting subselect into just an additional join.md new file mode 100644 index 0000000..c82d27e --- /dev/null +++ b/Rewriting subselect into just an additional join.md @@ -0,0 +1,32 @@ +--- +tags: + - mysql +--- +Optimizing a query with subselects into a "flat" query with just additional joins. This query finds rows that are missing for instances. + +``` +select * +from instance_system_metadata ism1 +where ism1.instance_uuid in ( + select i.uuid + from instances i + left join instance_system_metadata ism + on ism.instance_uuid = i.uuid and ism.key = "image_os_admin_user" + where i.deleted_at is null and ism.id is null +) +and ism1.key = "image_hw_qemu_guest_agent"; +``` + +Can be rewritten more nicely with an additional join +``` +select i.uuid +from instances i +left join instance_system_metadata ism_agent + on ism_agent.instance_uuid = i.uuid + and ism_agent.key = "image_hw_qemu_guest_agent" +left join instance_system_metadata ism_user + on ism_user.instance_uuid = i.uuid + and ism_user.key = "image_os_admin_user" +where i.deleted is null + and ism_user.key is null; +``` \ No newline at end of file diff --git a/Updating the image os admin user to fix password resets.md b/Updating the image os admin user to fix password resets.md new file mode 100644 index 0000000..48852de --- /dev/null +++ b/Updating the image os admin user to fix password resets.md @@ -0,0 +1,77 @@ +--- +tags: + - work + - nova +--- + +**instances** +``` ++-------------+-----------------------+------+-----+---------+----------------+ +| Field | Type | Null | Key | Default | Extra | ++-------------+-----------------------+------+-----+---------+----------------+ +| deleted_at | datetime | YES | | NULL | | +| uuid | varchar(36) | NO | UNI | NULL | | ++-------------+-----------------------+------+-----+---------+----------------+ +``` +**instance_system_metadata** +``` ++---------------+--------------+------+-----+---------+----------------+ +| Field | Type | Null | Key | Default | Extra | ++---------------+--------------+------+-----+---------+----------------+ +| created_at | datetime | YES | | NULL | | +| updated_at | datetime | YES | | NULL | | +| deleted_at | datetime | YES | | NULL | | +| id | int(11) | NO | PRI | NULL | auto_increment | +| instance_uuid | varchar(36) | NO | MUL | NULL | | +| key | varchar(255) | NO | | NULL | | +| value | varchar(255) | YES | | NULL | | +| deleted | int(11) | YES | | NULL | | ++---------------+--------------+------+-----+---------+----------------+ +``` + +**Insert new record** +``` +insert into instance_system_metadata (created_at, instance_uuid, `key`, `value`) values (NOW(), "4e675ad6-f01a-476f-8827-8300e540447c", "image_os_admin_user", "administrator" +``` + +**Updated existing values** +``` +update instance_system_metadata set `value` = "administrator", updated_at = NOW() where deleted_at is NULL and `key` = "image_os_admin_user" and instance_uuid in (select instance_uuid from instance_system_metadata where deleted_at is NULL and `key` = "image_os_t +ype" and `value` = "linux"); +``` + +**Which instances currently do not have image_os_admin_user** +``` +select i.uuid from instances i left join instance_system_metadata ism on ism.instance_uuid = i.uuid and ism.key = "image_os_admin_user" where i.deleted_at is null and ism.id is null +``` + +**Which instances of those above, do have qemu guest agent stuff set** +``` +select * from instance_system_metadata ism1 where ism1.instance_uuid in (select i.uuid from instances i left join instance_system_metadata ism on ism.instance_uuid = i.uuid and ism.key = "image_os_admin_user" where i.deleted_at is null and ism.id is null) and ism1.key = "image_hw_qemu_guest_agent"; +``` + +**Insert the image_os_admin_user for those records that miss it but do have qemu guest agent metadata** +``` +INSERT INTO instance_system_metadata (created_at, instance_uuid, `key`, `value`, deleted) +SELECT + NOW(), -- Current timestamp for created_at + i.uuid, -- The UUID of the instance needing the new key + 'image_os_admin_user', -- The fixed key you want to insert + 'administrator', -- The fixed value for the new key + 0 +FROM + instances i +INNER JOIN -- Use INNER JOIN to ensure the instance MUST have the guest agent key + instance_system_metadata ism_agent ON ism_agent.instance_uuid = i.uuid + AND ism_agent.key = 'image_hw_qemu_guest_agent' +LEFT JOIN -- Use LEFT JOIN to check for the PRESENCE of the admin user key + instance_system_metadata ism_admin ON ism_admin.instance_uuid = i.uuid + AND ism_admin.key = 'image_os_admin_user' +WHERE + i.deleted_at IS NULL -- Ensure the instance is not deleted + AND ism_admin.instance_uuid IS NULL; -- This is the crucial part: Select only rows where the LEFT JOIN did NOT find an existing 'image_os_admin_user' key +``` + +``` +insert into instance_system_metadata (created_at, instance_uuid, `key`, `value`, deleted) select NOW(), i.uuid, 'image_os_admin_user', 'administrator', 0 from instances i inner join instance_system_metadata ism_agent on ism_agent.instance_uuid = i.uuid and ism_agent.key = 'image_hw_qemu_guest_agent' left join instance_system_metadata ism_admin on ism_admin.instance_uuid = i.uuid and ism_admin.key = 'image_os_admin_user' where i.deleted_at is null and ism_admin.instance_uuid is null; +``` \ No newline at end of file diff --git a/Using Libvirt to manage QEMU image bitmaps.md b/Using Libvirt to manage QEMU image bitmaps.md new file mode 100644 index 0000000..92eef5f --- /dev/null +++ b/Using Libvirt to manage QEMU image bitmaps.md @@ -0,0 +1,21 @@ +--- +tags: + - qemu + - libvirt +--- +To query info about block devices, including bitmaps we can use a QMP command: +``` +sudo virsh qemu-monitor-command --pretty instance-00026133 '{"execute": "query-block", "arguments": {}}' +``` + +There are six primary bitmap-management API commands: +- `block-dirty-bitmap-add` +- `block-dirty-bitmap-remove` +- `block-dirty-bitmap-clear` +- `block-dirty-bitmap-disable` +- `block-dirty-bitmap-enable` +- `block-dirty-bitmap-merge` +And one related query command: +- `query-block` + +The `node` argument to bitmap cmds can be found as `node-name` in `query-block` results. \ No newline at end of file -- cgit v1.2.3 From eae6b57f17f5065a2bd3f130668043693f579cb4 Mon Sep 17 00:00:00 2001 From: Jasper Ras Date: Fri, 11 Apr 2025 16:02:40 +0200 Subject: vault backup: 2025-04-11 16:02:40 --- .obsidian/daily-notes.json | 4 + .obsidian/workspace.json | 89 ++++++++-------------- .trash/2025-04-10.md | 0 ...ith infinite choice and cannot make a choice.md | 1 + ... out to be one of us that forgot to clean up.md | 6 ++ .trash/Untitled 2.md | 0 Detroit-style TDD.md | 18 +++++ Device mapper.md | 30 ++++++++ Discovery Testing.md | 8 ++ High Gear Low Gear Testing.md | 14 ++++ How to (RIP) Grep.md | 17 +++++ ...f tags I use in this Vault and their purpose.md | 17 +++++ London-style TDD.md | 8 ++ London-style versus Detroit-style TDD.md | 12 +++ Managing Ceph on Devstack & OpenStack.md | 10 +++ OpenStack Tempest.md | 7 ++ TDD.md | 12 +++ Tempest Plugin.md | 18 +++++ ...olume type of a Cinder volume can be changed.md | 7 ++ ...real implementation gives feedback on design.md | 14 ++++ YAGNI.md | 8 ++ daily/10-Apr-2025.md | 18 +++++ daily/11-Apr-2025.md | 27 +++++++ 23 files changed, 286 insertions(+), 59 deletions(-) create mode 100644 .obsidian/daily-notes.json create mode 100644 .trash/2025-04-10.md create mode 100644 .trash/Blank Page Syndrom is when you start with infinite choice and cannot make a choice.md create mode 100644 .trash/I assumed that Mohammed made an oopsie but instead it turned out to be one of us that forgot to clean up.md create mode 100644 .trash/Untitled 2.md create mode 100644 Detroit-style TDD.md create mode 100644 Device mapper.md create mode 100644 Discovery Testing.md create mode 100644 High Gear Low Gear Testing.md create mode 100644 How to (RIP) Grep.md create mode 100644 List of tags I use in this Vault and their purpose.md create mode 100644 London-style TDD.md create mode 100644 London-style versus Detroit-style TDD.md create mode 100644 OpenStack Tempest.md create mode 100644 TDD.md create mode 100644 Tempest Plugin.md create mode 100644 The volume type of a Cinder volume can be changed.md create mode 100644 Writing a test double before the real implementation gives feedback on design.md create mode 100644 YAGNI.md create mode 100644 daily/10-Apr-2025.md create mode 100644 daily/11-Apr-2025.md diff --git a/.obsidian/daily-notes.json b/.obsidian/daily-notes.json new file mode 100644 index 0000000..3efcc00 --- /dev/null +++ b/.obsidian/daily-notes.json @@ -0,0 +1,4 @@ +{ + "folder": "/daily", + "format": "DD-MMM-Y" +} \ No newline at end of file diff --git a/.obsidian/workspace.json b/.obsidian/workspace.json index f667601..9818d14 100644 --- a/.obsidian/workspace.json +++ b/.obsidian/workspace.json @@ -8,67 +8,37 @@ "type": "tabs", "children": [ { - "id": "502f77e94f5ff48c", + "id": "8533accf5c2515ac", "type": "leaf", "state": { "type": "markdown", "state": { - "file": "Managing Ceph on Devstack & OpenStack.md", + "file": "daily/11-Apr-2025.md", "mode": "source", "backlinks": true, "source": false }, "icon": "lucide-file", - "title": "Managing Ceph on Devstack & OpenStack" + "title": "11-Apr-2025" } }, { - "id": "c10897c003c6bee7", + "id": "a06a83491e7620e5", "type": "leaf", "state": { "type": "markdown", "state": { - "file": "OVN upgrade.md", + "file": "Device mapper.md", "mode": "source", "backlinks": true, "source": false }, "icon": "lucide-file", - "title": "OVN upgrade" - } - }, - { - "id": "62d186e3d63b15e0", - "type": "leaf", - "state": { - "type": "markdown", - "state": { - "file": "I'm worried that Freenet will completely go DOWN due to OVN upgrade.md", - "mode": "source", - "backlinks": true, - "source": false - }, - "icon": "lucide-file", - "title": "I'm worried that Freenet will completely go DOWN due to OVN upgrade" - } - }, - { - "id": "96a8cc05fb9592a6", - "type": "leaf", - "state": { - "type": "markdown", - "state": { - "file": "Network outage on single hypervisor causes Freenet to shutdown EVERYTHING.md", - "mode": "source", - "backlinks": true, - "source": false - }, - "icon": "lucide-file", - "title": "Network outage on single hypervisor causes Freenet to shutdown EVERYTHING" + "title": "Device mapper" } } ], - "currentTab": 3 + "currentTab": 1 } ], "direction": "vertical" @@ -99,7 +69,7 @@ "state": { "type": "search", "state": { - "query": "OVN upgrade", + "query": "rbd", "matchingCase": false, "explainSearch": false, "collapseAll": false, @@ -219,34 +189,35 @@ "obsidian-excalidraw-plugin:New drawing": false } }, - "active": "96a8cc05fb9592a6", + "active": "a06a83491e7620e5", "lastOpenFiles": [ - "Making notes useful.md", + "daily/11-Apr-2025.md", + "Device mapper.md", + "daily/10-Apr-2025.md", + "List of tags I use in this Vault and their purpose.md", "Managing Ceph on Devstack & OpenStack.md", - "Network outage on single hypervisor causes Freenet to shutdown EVERYTHING.md", + "OpenStack Tempest.md", + "Tempest Plugin.md", + "High Gear Low Gear Testing.md", + "TDD.md", + "The volume type of a Cinder volume can be changed.md", + "How to (RIP) Grep.md", + "I assumed that Mohammed made an oopsie but instead it turned out to be one of us that forgot to clean up.md", + "Writing a test double before the real implementation gives feedback on design.md", + "London-style versus Detroit-style TDD.md", "I'm worried that Freenet will completely go DOWN due to OVN upgrade.md", - "Random notes on Freenet deployment.md", "OVN upgrade.md", + "daily", + "2025-04-10.md", "Installing Ceph on devstack.md", "Overview of Ceph.md", "Gobs Ceph Client Key Access.md", - "Installing a devstack plugin.md", - "Rewriting subselect into just an additional join.md", - "Updating the image os admin user to fix password resets.md", - "Freenet CPU steal graph.md", - "Using Libvirt to manage QEMU image bitmaps.md", - "Build custom modules as profiles to manage various NixOS hosts.md", - "Nix Inherit.md", - "Becoming a Senior.md", - "A reason for becoming senior is that when Rutger is off I take most of his responsibilities.md", - "Acronis Managed Backup provisioning flow.md", - "Ceph.md", - "NixOS - Imperative container management.md", - "NixOS - Declarative container management.md", - "NixOS - Containers.md", - "NixOS - Container networking.md", - "EF02 partition.md", - "EF00 partition.md", + "Detroit-style TDD.md", + "Blank Page Syndrom is when you start with infinite choice and cannot make a choice.md", + "London-style TDD.md", + "Discovery Testing.md", + "YAGNI.md", + "Network outage on single hypervisor causes Freenet to shutdown EVERYTHING.md", "ESP", "Excalidraw" ] diff --git a/.trash/2025-04-10.md b/.trash/2025-04-10.md new file mode 100644 index 0000000..e69de29 diff --git a/.trash/Blank Page Syndrom is when you start with infinite choice and cannot make a choice.md b/.trash/Blank Page Syndrom is when you start with infinite choice and cannot make a choice.md new file mode 100644 index 0000000..417bc2c --- /dev/null +++ b/.trash/Blank Page Syndrom is when you start with infinite choice and cannot make a choice.md @@ -0,0 +1 @@ +A way to counter this is by showing an example instead of a blank page. \ No newline at end of file diff --git a/.trash/I assumed that Mohammed made an oopsie but instead it turned out to be one of us that forgot to clean up.md b/.trash/I assumed that Mohammed made an oopsie but instead it turned out to be one of us that forgot to clean up.md new file mode 100644 index 0000000..9e1cee6 --- /dev/null +++ b/.trash/I assumed that Mohammed made an oopsie but instead it turned out to be one of us that forgot to clean up.md @@ -0,0 +1,6 @@ +--- +tags: + - self + - reflection +--- + diff --git a/.trash/Untitled 2.md b/.trash/Untitled 2.md new file mode 100644 index 0000000..e69de29 diff --git a/Detroit-style TDD.md b/Detroit-style TDD.md new file mode 100644 index 0000000..5726068 --- /dev/null +++ b/Detroit-style TDD.md @@ -0,0 +1,18 @@ +--- +tags: + - software-design + - software-engineering + - tdd +--- +Classic TDD: + +1. Write a failing test for public API (red) +2. Change implementation to make it pass (green) +3. Refactor +4. Go to step 1 + +Benefits: +- Small increments +- When adding features we can spot regressions quickly due to having numerous tests +- Aggressive refactoring possible due to public API being tested +- Complete regression test suite \ No newline at end of file diff --git a/Device mapper.md b/Device mapper.md new file mode 100644 index 0000000..150338b --- /dev/null +++ b/Device mapper.md @@ -0,0 +1,30 @@ +--- +tags: + - linux + - devices + - kernel +--- +Device mapper is a kernel driver to map physical block devices to virtual ones; this means that for example we can have many physical devices be represented by a single virtual one (linear mapping). + +How does it work? +What is its function in relation to multipath? + +Logical devices using device-mapper can be managed using `man 8 dmsetup` +``` +[jasras@n04.compute.vps2-lej1 ~]$ sudo dmsetup info 3600a098038314d736724566a67346538 +Name: 3600a098038314d736724566a67346538 +State: ACTIVE +Read Ahead: 256 +Tables present: LIVE +Open count: 1 +Event number: 3 +Major, minor: 253, 12 +Number of targets: 1 +UUID: mpath-3600a098038314d736724566a67346538 +``` + +If dmsetup cannot remove a device because a process still has it open, but lsof does not show any processes that open it, use `-f` which replaces the device with a fake that rejects all I/O. + +--- +[device-mapper](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/logical_volume_manager_administration/device_mapper#device_mapper) +[dmsetup](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/logical_volume_manager_administration/dmsetup#dmsetup) \ No newline at end of file diff --git a/Discovery Testing.md b/Discovery Testing.md new file mode 100644 index 0000000..9495dfe --- /dev/null +++ b/Discovery Testing.md @@ -0,0 +1,8 @@ +--- +tags: + - tdd + - architecture + - software-design + - software-engineering +--- +Is a form of [[TDD]] \ No newline at end of file diff --git a/High Gear Low Gear Testing.md b/High Gear Low Gear Testing.md new file mode 100644 index 0000000..a33d223 --- /dev/null +++ b/High Gear Low Gear Testing.md @@ -0,0 +1,14 @@ +--- +tags: + - software-design + - software-engineering + - tdd +--- +This phrase came from the https://www.cosmicpython.com/ book.. The core idea here is that during development you write **many** small unit-tests that make it easy and fast to test whether we are doing the correct thing. +Having many such small tests makes it hard to refactor, so at some point we can start dropping some of the very specific tests when we have more general tests and feel confident that everything works as intended. This makes it less brittle, because small very specific tests will start failing the moment you start refactoring a little bit. + +Then when you need to start developing again you start by writing new very specific tests until you again feel confident that everything works as intended and is also tested by more general tests. + +The analog is a bit to cycling: when you start to move you set the bicycle in a low gear and then once you are going you shift the gear lower and lower until you are at speed and comfortable. + +In my mind it is still a bit unclear when that inflection point occurs that you decide to drop specific tests though, but I guess that also kind of comes with practice and experience. \ No newline at end of file diff --git a/How to (RIP) Grep.md b/How to (RIP) Grep.md new file mode 100644 index 0000000..f7122a3 --- /dev/null +++ b/How to (RIP) Grep.md @@ -0,0 +1,17 @@ +--- +tags: + - howto + - grep +--- + +`grep -vxf gobs_servers onecrm_servers` grep all onecrm_servers that **ARE NOT** in gobs_servers +`grep -oxf gobs_servers onecrm_servers` grep all onecrm_servers that **ARE** in gobs_servers +`grep -vxf onecrm_servers gobs_servers` grep all gobs_servers that **ARE NOT** in onecrm_servers +`grep -oxf onecrm_servers gobs_servers` grep all gobs_servers that **ARE** in onecrm_servers + + +# Useful Ripgrep flags + + `-L` to make it follow symlinks: this is handy for example when searching hieradata, because node definitions can be symlinked. + +`-l` to only print filenames that contain matches. \ No newline at end of file diff --git a/List of tags I use in this Vault and their purpose.md b/List of tags I use in this Vault and their purpose.md new file mode 100644 index 0000000..1b514d3 --- /dev/null +++ b/List of tags I use in this Vault and their purpose.md @@ -0,0 +1,17 @@ +--- +tags: + - self + - reflection + - howto +--- +I divide tags into seperate categories, they are as follows: + +# Functional tags +All tags ultimately are there to categorize content, but these tags indicate the note has a specific function or goal. + +- **self**: notes that are about me +- **reflection**: notes that contain some wisdom from looking in the mirror and seeing myself, they often go together with `self` +- **howto**: a note on how to do something, such as this one, how to use tags + +# Category tags +These are just to categorize notes based on for example subject. \ No newline at end of file diff --git a/London-style TDD.md b/London-style TDD.md new file mode 100644 index 0000000..a3c29c2 --- /dev/null +++ b/London-style TDD.md @@ -0,0 +1,8 @@ +--- +tags: + - tdd + - software-design + - software-engineering +--- +Is very similar to [[Detroit-style TDD]] but there are some differences. +[[London-style versus Detroit-style TDD]] \ No newline at end of file diff --git a/London-style versus Detroit-style TDD.md b/London-style versus Detroit-style TDD.md new file mode 100644 index 0000000..7c10d4d --- /dev/null +++ b/London-style versus Detroit-style TDD.md @@ -0,0 +1,12 @@ +--- +tags: + - tdd + - software-design + - software-engineering +--- +Detroit-style TDD focuses mainly on working bottom up, starting with the core of the domain working outward. +London-style TDD works outside in, starting at an API endpoint or controller working inwards. + +The primary con of london style is to build things that aren't needed ([[YAGNI]]) + +https://github.com/testdouble/contributing-tests/wiki/London-school-TDD \ No newline at end of file diff --git a/Managing Ceph on Devstack & OpenStack.md b/Managing Ceph on Devstack & OpenStack.md index cad83e5..6ccb1c5 100644 --- a/Managing Ceph on Devstack & OpenStack.md +++ b/Managing Ceph on Devstack & OpenStack.md @@ -39,4 +39,14 @@ rbd image 'volume-6f5e81dd-40f3-42f9-aad3-afcf5696387c': modify_timestamp: Mon Mar 31 12:56:46 2025 parent: volumes/volume-5b6cc251-8f3e-4572-a8b9-52efa390ebc3@snapshot-3d1f148a-5fe4-4df2-87c3-4272565b82c2 overlap: 10 GiB +``` + +using `rbd du` we can get some actual usage stats: +``` +ubuntu@gobs-devstack:~$ sudo rbd du volumes/volume-d66216fc-0b4a-4480-be0e-5530fb8e6004 +NAME PROVISIONED USED +volume-d66216fc-0b4a-4480-be0e-5530fb8e6004@volume-e65e7bcf-88a8-4afb-ba5c-0dbb27f402d9.clone_snap 1 GiB 44 MiB +volume-d66216fc-0b4a-4480-be0e-5530fb8e6004@snapshot-05b07e40-8cbc-4f2d-896c-bbfc0cd2c801 10 GiB 52 MiB +volume-d66216fc-0b4a-4480-be0e-5530fb8e6004 10 GiB 10 GiB + 10 GiB 10 GiB ``` \ No newline at end of file diff --git a/OpenStack Tempest.md b/OpenStack Tempest.md new file mode 100644 index 0000000..d7d84c4 --- /dev/null +++ b/OpenStack Tempest.md @@ -0,0 +1,7 @@ +--- +tags: + - tempest + - openstack +--- +Test an OpenStack cloud as if it is an end-user. +**Scenario Test** is more complicated than a single APi endpoint, like, booting from an existing volume. \ No newline at end of file diff --git a/TDD.md b/TDD.md new file mode 100644 index 0000000..b555158 --- /dev/null +++ b/TDD.md @@ -0,0 +1,12 @@ +--- +tags: + - tdd + - software-design + - software-engineering +--- +Test Driven Development (or Design) is a software engineering paradigm where one starts off with writing a failing test (suite) and from there tries to implement code to ultimately make the test succeed. + +It is mainly divided into three styles: +- [[Detroit-style TDD]] +- [[London-style TDD]] +- [[Discovery Testing]] \ No newline at end of file diff --git a/Tempest Plugin.md b/Tempest Plugin.md new file mode 100644 index 0000000..58ebaac --- /dev/null +++ b/Tempest Plugin.md @@ -0,0 +1,18 @@ +--- +tags: + - openstack + - tempest +--- +A plugin for [[OpenStack Tempest]] that tests a non-core service such as gobs. + +**Recommendation:** keep tempest plugin for a service in a seperate repository. + +Requires an entrypoint in your app that it can find. +Consists of "service clients", clients that consume specific parts of the service. ie. a backup service client, or snapshot service client. + +1 test suite per service / resource of the API. + +Base class for "API Tests" can handle resource management using an idempotent id it can make sure resources are properly destroyed upon teardown. + +--- +https://www.youtube.com/watch?v=oN4lRR2_S_k \ No newline at end of file diff --git a/The volume type of a Cinder volume can be changed.md b/The volume type of a Cinder volume can be changed.md new file mode 100644 index 0000000..8ee52ad --- /dev/null +++ b/The volume type of a Cinder volume can be changed.md @@ -0,0 +1,7 @@ +--- +tags: + - openstack + - cinder + - howto +--- +Shelve server, set new volume type, unshelve server. \ No newline at end of file diff --git a/Writing a test double before the real implementation gives feedback on design.md b/Writing a test double before the real implementation gives feedback on design.md new file mode 100644 index 0000000..9412676 --- /dev/null +++ b/Writing a test double before the real implementation gives feedback on design.md @@ -0,0 +1,14 @@ +--- +tags: + - tdd + - software-design + - architecture + - software-engineering +--- +https://github.com/testdouble/contributing-tests/wiki/Don't-mock-what-you-don't-own + +The primary value of a test double from a TDD perspective is that it provides **cheap feedback** on the design of an object. Because we can play with its interface before actually having to use it, it is cheap to throw away and start over (vs having implemented all sorts of other stuff around it). + +This is alternatively called "Increased Usage", referring to that a test is an additional usage of a function (which normally sometimes have just a single caller anyway) and that while writing a few test cases you'll quickly get a feel of if the API is good or bad. + +[[London-style versus Detroit-style TDD]] \ No newline at end of file diff --git a/YAGNI.md b/YAGNI.md new file mode 100644 index 0000000..cc43d7a --- /dev/null +++ b/YAGNI.md @@ -0,0 +1,8 @@ +--- +tags: + - tdd + - software-engineering + - software-design + - architecture +--- +It means "You Ain't Gonna Need It". It refers to engineers often building stuff that isn't actually going to be used. \ No newline at end of file diff --git a/daily/10-Apr-2025.md b/daily/10-Apr-2025.md new file mode 100644 index 0000000..8ac3187 --- /dev/null +++ b/daily/10-Apr-2025.md @@ -0,0 +1,18 @@ +--- +tags: + - self + - reflection +--- +### I assumed that Mohammed made an oopsie but instead it turned out to be one of us that forgot to clean up. +Today I found out that on the testpod the user sanoid's ssh keys suddenly belonged to Mohammeds' user. I right away went to his chat with the idea that he did something silly probably and even told him that it made me a bit worried. + +I also went to Rutger, who immediately pointed out that it is probably due to us changing the uid of sanoids' user. Which turned out to be true. + +I ask myself the following: why am I quick to jump to a conclusion like "Ah, mohammed might've caused damage by accidentally chowning to much or something like that." +How can I stop myself from that? +I think one way is to ALWAYS force myself to investigate completely, not right away talking about it with others. Keep it to myself, until I really MUST communicate about it. + + +#### Erik doesn't show up at the office for Carlos even though he said that he would be there on Thursday +This kind of triggers a feeling that Erik has a bit of a lax attitude, which often was associated with ops back in the day as well. +He is the designated mentor but I feel he doesn't prepare it well and just "goes with the flow" too much. \ No newline at end of file diff --git a/daily/11-Apr-2025.md b/daily/11-Apr-2025.md new file mode 100644 index 0000000..8b0a2cf --- /dev/null +++ b/daily/11-Apr-2025.md @@ -0,0 +1,27 @@ +--- +tags: + - weekly +--- +This week: +- [[10-Apr-2025]] + +Today marks another Friday, almost weekend, woohoo. + +This week few notable things happened. + +First of all I finally have restored a Ceph backed volume successfully! The issue was actually kind of silly, I forgot to close the read end of a pipe, thus it kept blocking. Luckily I found it, and fixing it was rather trivial. +After that I refactored a bit, to make it little bit better (still not great), and deployed. +I also fixed the request ID logging that was broken for a while, during refactoring of the agent RPC handler I accidentally moved the ctx.update_store outside of the child thread, so the update was useless. Moving it back into the child thread gave back our precious request ids. +Oh and I also found the cause of some sporadic mysql "object belongs to a different session" issue in the backup service that was haunting me. This happened due the Unit Of Work being instantiated only during application startup, specifically for the RPC handler, and then every RPC request used that same UOW. +Because the UOW creates a new session everytime it wasn't completely broken, but occasionally two RPC calls could come in at the same time and then the latter would override the session of the first. +I fixed this by instantiating a UOW per request, this also happens in the API, and is actually the correct way of using it. + +Then I have kicked off the [[List of tags I use in this Vault and their purpose]] note, which contains a list of tags I use within this vault so I don't forget. +As with the current note i'm writing I added the new "weekly" tag to indicate that this "daily" note is actually a week report which I want to write every friday from now on. + +Just had a little brainfart writing the above.. Is it too long? I plan to use these weekly notes to introspect during self assessments, but of course they shouldn't be too tedious to go through... Hmm, well, I guess we'll have to actually **use** it before deciding that. + +I think it will be good practice to link to the current weeks' notes in this weekly note as well so.. see the top :) I probably should put more stuff into daily notes, and then make this a bit of a TLDR. + +I have also been doing some thinking and note taking about [[TDD]] because I feel kind of bad about the current state of the backup service & agent (no tests). We've been looking at [[OpenStack Tempest]] for a bit which is interesting, but I would also like to just create better and more unit tests, especially during development. [[High Gear Low Gear Testing]] was a phrase from the cosmic python book that particularly seemed to resonate with me, but I haven't yet been able to practice it. + -- cgit v1.2.3 From c8c774ad7c0d98def2e83647d4f635ee275b66de Mon Sep 17 00:00:00 2001 From: Jasper Ras Date: Tue, 15 Apr 2025 19:44:48 +0200 Subject: vault backup: 2025-04-15 19:44:48 --- .obsidian/workspace.json | 65 ++++++++++++++++++---- Examples of proper JQ usage.md | 11 ++++ ...rly do options in a bash script with getopt!.md | 31 +++++++++++ Using Libvirt to manage QEMU image bitmaps.md | 10 +++- ...m discard syscalls on our openstack platform.md | 7 +++ daily/14-Apr-2025.md | 22 ++++++++ daily/15-Apr-2025.md | 28 ++++++++++ 7 files changed, 163 insertions(+), 11 deletions(-) create mode 100644 Examples of proper JQ usage.md create mode 100644 How to properly do options in a bash script with getopt!.md create mode 100644 We don't support filesystem discard syscalls on our openstack platform.md create mode 100644 daily/14-Apr-2025.md create mode 100644 daily/15-Apr-2025.md diff --git a/.obsidian/workspace.json b/.obsidian/workspace.json index 9818d14..d57c196 100644 --- a/.obsidian/workspace.json +++ b/.obsidian/workspace.json @@ -36,9 +36,54 @@ "icon": "lucide-file", "title": "Device mapper" } + }, + { + "id": "30f6cf07a77bd478", + "type": "leaf", + "state": { + "type": "markdown", + "state": { + "file": "Examples of proper JQ usage.md", + "mode": "source", + "backlinks": true, + "source": false + }, + "icon": "lucide-file", + "title": "Examples of proper JQ usage" + } + }, + { + "id": "c550cd183351059f", + "type": "leaf", + "state": { + "type": "markdown", + "state": { + "file": "Using Libvirt to manage QEMU image bitmaps.md", + "mode": "source", + "backlinks": true, + "source": false + }, + "icon": "lucide-file", + "title": "Using Libvirt to manage QEMU image bitmaps" + } + }, + { + "id": "5b6958c83833df4a", + "type": "leaf", + "state": { + "type": "markdown", + "state": { + "file": "daily/15-Apr-2025.md", + "mode": "source", + "backlinks": true, + "source": false + }, + "icon": "lucide-file", + "title": "15-Apr-2025" + } } ], - "currentTab": 1 + "currentTab": 4 } ], "direction": "vertical" @@ -69,7 +114,7 @@ "state": { "type": "search", "state": { - "query": "rbd", + "query": "qemu bitm", "matchingCase": false, "explainSearch": false, "collapseAll": false, @@ -189,10 +234,16 @@ "obsidian-excalidraw-plugin:New drawing": false } }, - "active": "a06a83491e7620e5", + "active": "5b6958c83833df4a", "lastOpenFiles": [ - "daily/11-Apr-2025.md", + "How to properly do options in a bash script with getopt!.md", + "daily/15-Apr-2025.md", + "Using Libvirt to manage QEMU image bitmaps.md", + "daily/14-Apr-2025.md", + "We don't support filesystem discard syscalls on our openstack platform.md", + "Examples of proper JQ usage.md", "Device mapper.md", + "daily/11-Apr-2025.md", "daily/10-Apr-2025.md", "List of tags I use in this Vault and their purpose.md", "Managing Ceph on Devstack & OpenStack.md", @@ -212,12 +263,6 @@ "Installing Ceph on devstack.md", "Overview of Ceph.md", "Gobs Ceph Client Key Access.md", - "Detroit-style TDD.md", - "Blank Page Syndrom is when you start with infinite choice and cannot make a choice.md", - "London-style TDD.md", - "Discovery Testing.md", - "YAGNI.md", - "Network outage on single hypervisor causes Freenet to shutdown EVERYTHING.md", "ESP", "Excalidraw" ] diff --git a/Examples of proper JQ usage.md b/Examples of proper JQ usage.md new file mode 100644 index 0000000..b63ee4e --- /dev/null +++ b/Examples of proper JQ usage.md @@ -0,0 +1,11 @@ +--- +tags: + - linux + - jq + - howto +--- +Double escapes are important! + +``` +jq -r '.fixed_ips[] | select(.ip_address | test("\\.")) | .ip_address' +``` \ No newline at end of file diff --git a/How to properly do options in a bash script with getopt!.md b/How to properly do options in a bash script with getopt!.md new file mode 100644 index 0000000..7d417c5 --- /dev/null +++ b/How to properly do options in a bash script with getopt!.md @@ -0,0 +1,31 @@ +--- +tags: + - bash +--- +``` +#!/bin/bash + +while getopts "ab:c" opt; do + case "$opt" in + a) + echo "Option -a was specified." + ;; + b) + echo "Option -b was specified with argument: $OPTARG" + ;; + c) + echo "Option -c was specified." + ;; + \?) + echo "Invalid option: -$OPTARG" >&2 + exit 1 + ;; + esac +done + +shift $((OPTIND - 1)) + +echo "Remaining arguments: $@" +``` + +man bash // getopt \ No newline at end of file diff --git a/Using Libvirt to manage QEMU image bitmaps.md b/Using Libvirt to manage QEMU image bitmaps.md index 92eef5f..72942b3 100644 --- a/Using Libvirt to manage QEMU image bitmaps.md +++ b/Using Libvirt to manage QEMU image bitmaps.md @@ -18,4 +18,12 @@ There are six primary bitmap-management API commands: And one related query command: - `query-block` -The `node` argument to bitmap cmds can be found as `node-name` in `query-block` results. \ No newline at end of file +The `node` argument to bitmap cmds can be found as `node-name` in `query-block` results. + + +#### Removing a dirty bitmap +Cannot be removed if it is busy. +Arguments: node, nameg + +--- +https://qemu-project.gitlab.io/qemu/interop/bitmaps.html \ No newline at end of file diff --git a/We don't support filesystem discard syscalls on our openstack platform.md b/We don't support filesystem discard syscalls on our openstack platform.md new file mode 100644 index 0000000..a8bb498 --- /dev/null +++ b/We don't support filesystem discard syscalls on our openstack platform.md @@ -0,0 +1,7 @@ +--- +tags: + - openstack + - work + - groupvps +--- +This is due to the NetAPP driver implementing discard as sending NUL-bytes, which increases load significantly. \ No newline at end of file diff --git a/daily/14-Apr-2025.md b/daily/14-Apr-2025.md new file mode 100644 index 0000000..94bc485 --- /dev/null +++ b/daily/14-Apr-2025.md @@ -0,0 +1,22 @@ +Monday! + +# Standup +OnFailure handlers installeren voor rename script ZFS datasets op backup nodes. +Survey invullen +Puppetrun failed nalopen +Uitzoeken waarom de F QEMU GA sporadisch faalt +- Dit had geloof ik iets te maken met QEMU crash ? Ff dubbel checken, heb er ergens een note van denk ik. +# QA with Webglobe team +Q: Virtual buses, volumes, suggested virtio. Can we do iSCSI instead of VirtIO because we do discards? +A: We don't support discard. NetAPP implements by sending nul bytes. -> increase IO +^ i wouldn't know this + +Q: Can we install from CD? +R: yes, possible. create image, props, iso boot, boot vm rescue from image. +J: documented? +R: Will find dgoc for onehome + +Q: IP addresses, do we really need to let OS handle the allocation? +A: OpenStack does this out of the box; used as single source of truth. +J: Finds reason acceptable; will rewrite. + diff --git a/daily/15-Apr-2025.md b/daily/15-Apr-2025.md new file mode 100644 index 0000000..06ab31a --- /dev/null +++ b/daily/15-Apr-2025.md @@ -0,0 +1,28 @@ +**Interview Ali** + +Q: +Waarom ga je weg bij Leaseplan? +Over openstack deployment: hoe geautomatiseerd; welke tools? + +> Implemented virtual staging clusters mirroring the production architecture using KVM, libvirt, Linux bridge/virtual interfaces, and iptables, reducing setup time by 90% while significantly optimizing costs. + +Did you directly integrate to KVM/Libvirt, can you tell a bit about that? + +A: +php/wordpress dev +exp with cpanel and such +then switch devops + + ++1 kolla ansible / openstack ++1 cpanel/hosting +-1 geen puppet ervaring ++1 ovn ervaring; meest voorname probleem is met ovs/ovn + +beetje rare sidetrack config mgmt ansible vs puppet; maakt punt over consistency + +Q: regular day +Q: expectations + +In amsterdam +Avail: 1 june \ No newline at end of file -- cgit v1.2.3 From ab409a3701bf59dd73dc1e0324376bdac8b6d74f Mon Sep 17 00:00:00 2001 From: Jasper Ras Date: Fri, 18 Apr 2025 20:58:54 +0200 Subject: vault backup: 2025-04-18 20:58:54 --- .obsidian/workspace.json | 81 +++++++++------------- .trash/Untitled 3.md | 0 About keyboard layout on Wayland.md | 15 ++++ Benchmarking code in Golang.md | 40 +++++++++++ ...gging issues with updating Puppet dependency.md | 46 ++++++++++++ Exposing development headers on Nixos.md | 15 ++++ Freenet CPU steal graph.md | 2 +- How to (RIP) Grep.md | 17 ----- How to RIP (grep).md | 17 +++++ How to show the mysql galera cluster status.md | 14 ++++ Importing GPG keys from a keyserver.md | 6 ++ ...medium ARC for hypervisors where ZFS is used.md | 10 +++ ...e is done from the apache site configuration.md | 7 ++ Testing Golang programs.md | 9 +++ Working with Nix shells.md | 66 ++++++++++++++++++ daily/16-Apr-2025.md | 15 ++++ daily/17-Apr-2025.md | 2 + daily/18-Apr-2025.md | 6 ++ 18 files changed, 302 insertions(+), 66 deletions(-) create mode 100644 .trash/Untitled 3.md create mode 100644 About keyboard layout on Wayland.md create mode 100644 Benchmarking code in Golang.md create mode 100644 Debugging issues with updating Puppet dependency.md create mode 100644 Exposing development headers on Nixos.md delete mode 100644 How to (RIP) Grep.md create mode 100644 How to RIP (grep).md create mode 100644 How to show the mysql galera cluster status.md create mode 100644 Importing GPG keys from a keyserver.md create mode 100644 It's recommended to use small-medium ARC for hypervisors where ZFS is used.md create mode 100644 Managing the amount of WSGI processes ran by apache is done from the apache site configuration.md create mode 100644 Testing Golang programs.md create mode 100644 Working with Nix shells.md create mode 100644 daily/16-Apr-2025.md create mode 100644 daily/17-Apr-2025.md create mode 100644 daily/18-Apr-2025.md diff --git a/.obsidian/workspace.json b/.obsidian/workspace.json index d57c196..c2973af 100644 --- a/.obsidian/workspace.json +++ b/.obsidian/workspace.json @@ -4,86 +4,71 @@ "type": "split", "children": [ { - "id": "b5921b94f3580090", + "id": "f33c11cf18be28d7", "type": "tabs", "children": [ { - "id": "8533accf5c2515ac", + "id": "3628dba7efa2e99a", "type": "leaf", "state": { "type": "markdown", "state": { - "file": "daily/11-Apr-2025.md", + "file": "Freenet CPU steal graph.md", "mode": "source", "backlinks": true, "source": false }, "icon": "lucide-file", - "title": "11-Apr-2025" + "title": "Freenet CPU steal graph" } }, { - "id": "a06a83491e7620e5", + "id": "b1754e4cb810f3a0", "type": "leaf", "state": { "type": "markdown", "state": { - "file": "Device mapper.md", + "file": "How to show the mysql galera cluster status.md", "mode": "source", "backlinks": true, "source": false }, "icon": "lucide-file", - "title": "Device mapper" + "title": "How to show the mysql galera cluster status" } }, { - "id": "30f6cf07a77bd478", + "id": "5ac5819f83e59327", "type": "leaf", "state": { "type": "markdown", "state": { - "file": "Examples of proper JQ usage.md", + "file": "Managing the amount of WSGI processes ran by apache is done from the apache site configuration.md", "mode": "source", "backlinks": true, "source": false }, "icon": "lucide-file", - "title": "Examples of proper JQ usage" + "title": "Managing the amount of WSGI processes ran by apache is done from the apache site configuration" } }, { - "id": "c550cd183351059f", + "id": "487149710eacf558", "type": "leaf", "state": { "type": "markdown", "state": { - "file": "Using Libvirt to manage QEMU image bitmaps.md", + "file": "daily/18-Apr-2025.md", "mode": "source", "backlinks": true, "source": false }, "icon": "lucide-file", - "title": "Using Libvirt to manage QEMU image bitmaps" - } - }, - { - "id": "5b6958c83833df4a", - "type": "leaf", - "state": { - "type": "markdown", - "state": { - "file": "daily/15-Apr-2025.md", - "mode": "source", - "backlinks": true, - "source": false - }, - "icon": "lucide-file", - "title": "15-Apr-2025" + "title": "18-Apr-2025" } } ], - "currentTab": 4 + "currentTab": 3 } ], "direction": "vertical" @@ -114,7 +99,7 @@ "state": { "type": "search", "state": { - "query": "qemu bitm", + "query": "iptables", "matchingCase": false, "explainSearch": false, "collapseAll": false, @@ -234,12 +219,27 @@ "obsidian-excalidraw-plugin:New drawing": false } }, - "active": "5b6958c83833df4a", + "active": "487149710eacf558", "lastOpenFiles": [ - "How to properly do options in a bash script with getopt!.md", "daily/15-Apr-2025.md", - "Using Libvirt to manage QEMU image bitmaps.md", + "daily/18-Apr-2025.md", + "Working with Nix shells.md", + "Managing the amount of WSGI processes ran by apache is done from the apache site configuration.md", + "How to show the mysql galera cluster status.md", + "Freenet CPU steal graph.md", + "daily/17-Apr-2025.md", "daily/14-Apr-2025.md", + "daily/16-Apr-2025.md", + "Debugging issues with updating Puppet dependency.md", + "It's recommended to use small-medium ARC for hypervisors where ZFS is used.md", + "How to RIP (grep).md", + "Using Libvirt to manage QEMU image bitmaps.md", + "Importing GPG keys from a keyserver.md", + "Exposing development headers on Nixos.md", + "About keyboard layout on Wayland.md", + "Benchmarking code in Golang.md", + "Testing Golang programs.md", + "How to properly do options in a bash script with getopt!.md", "We don't support filesystem discard syscalls on our openstack platform.md", "Examples of proper JQ usage.md", "Device mapper.md", @@ -247,22 +247,7 @@ "daily/10-Apr-2025.md", "List of tags I use in this Vault and their purpose.md", "Managing Ceph on Devstack & OpenStack.md", - "OpenStack Tempest.md", - "Tempest Plugin.md", - "High Gear Low Gear Testing.md", - "TDD.md", - "The volume type of a Cinder volume can be changed.md", - "How to (RIP) Grep.md", - "I assumed that Mohammed made an oopsie but instead it turned out to be one of us that forgot to clean up.md", - "Writing a test double before the real implementation gives feedback on design.md", - "London-style versus Detroit-style TDD.md", - "I'm worried that Freenet will completely go DOWN due to OVN upgrade.md", - "OVN upgrade.md", "daily", - "2025-04-10.md", - "Installing Ceph on devstack.md", - "Overview of Ceph.md", - "Gobs Ceph Client Key Access.md", "ESP", "Excalidraw" ] diff --git a/.trash/Untitled 3.md b/.trash/Untitled 3.md new file mode 100644 index 0000000..e69de29 diff --git a/About keyboard layout on Wayland.md b/About keyboard layout on Wayland.md new file mode 100644 index 0000000..b4e8550 --- /dev/null +++ b/About keyboard layout on Wayland.md @@ -0,0 +1,15 @@ +--- +tags: + - wayland + - hyprland + - keyboard +--- +In Wayland the keyboard layout is managed by the compositor. In my case that is currently Hyprland. + +To find variants: +`localectl list-x11-keymap layouts:` +`localectl list-x11-keymap-variants us` + +^ This doesn't seem to work on NixOS. + +I've set it now via `services.xserver.xkb.variant`, and queried variants from gemini. \ No newline at end of file diff --git a/Benchmarking code in Golang.md b/Benchmarking code in Golang.md new file mode 100644 index 0000000..22d180d --- /dev/null +++ b/Benchmarking code in Golang.md @@ -0,0 +1,40 @@ +--- +tags: + - golang + - howto +--- +Simple timing can be done using the time package. +``` +import time +import fmt + +func main() { + start := time.Now() + elapsed = time.Since(start).Seconds() + fmt.Printf("%.2fs", elapsed) +} +``` + +Writing benchmarks. They look similar to tests: +``` +import testing + +func BenchmarkSome(b *testing.B) { + for i := 0; i < b.N; i++ { + Some() + } +} +``` +b.N is supplied by test driver and is dynamically changed depending on runtime. + +`go test -bench=` + +This just benchmarks a function being called many times, comparative benchmarks can be written using a regular function being called from Benchmarks, something like so : +``` +import testing + +func doBench(b *testing.B, size int) { ... vary inputs size } +func Bench1(b *testing.B) +func Bench10(b *testing.B) +func Bench100(b * testing.B) +``` \ No newline at end of file diff --git a/Debugging issues with updating Puppet dependency.md b/Debugging issues with updating Puppet dependency.md new file mode 100644 index 0000000..da6e2f2 --- /dev/null +++ b/Debugging issues with updating Puppet dependency.md @@ -0,0 +1,46 @@ +--- +tags: + - puppet +--- +Updated the ref in Puppetfile to yoga-eol instead of unmaintained/yoga. + +Stap 1: `librarian-puppet install --verbose`. +Stap 2: `librarian-puppet update openstack-neutron --verbose`. + +Check puppet-neutron `metadata.json` This is where dependency information lives. According to the gitlab readme it is also possible that this lives in `Modulefile` or `Puppetfile`. + +Verbose info gives: +``` +[Librarian] Failed to resolve puppetlabs-stdlib (>= 4.18.0, < 7.0.0) <(no source specified)> (from ) +``` + +Erik determined this is due to `puppet-yum` by inspecting the lock file and checking who sets those requirements. +``` +puppet master*​ ≡ +❯ rg '>= 4.18.0, < 7.0.0' Puppetfile.lock +68: puppetlabs-stdlib (>= 4.18.0, < 7.0.0) +``` +``` + 1 │ puppet-yum (4.3.0) + 1 │ │ puppetlabs-concat (>= 1.2.5, < 7.0.0) + 68 │ │ puppetlabs-stdlib (>= 4.18.0, < 7.0.0) +``` + +It turns out that this version of `puppet-yum` came from `elastic-elastic_stack` which came in turn from `elastic-logstash`. +We found that `elastic-logstash` has been deprecated, so we cannot upgrade this dependency. It is replaced by `puppet-logstash`. + +``` +puppet master ≡5s +❯ rg puppet-yum Puppetfile.lock +8: puppet-yum (>= 0.9.6, < 5.0.0) +66: puppet-yum (4.3.0) + +hx Puppetfile.lock + elastic-elastic_stack (6.3.2) + puppet-yum (>= 0.9.6, < 5.0.0) + +``` + +Changing the dependency from `elastic-logstash` to `puppet-logstash`, also brings issue. + +At the end our hacky resolution is to update the commit hash in `Puppetfile.lock` and push that. Possible only because we didn't change any dependencies in `openstack-neutron`. diff --git a/Exposing development headers on Nixos.md b/Exposing development headers on Nixos.md new file mode 100644 index 0000000..72db811 --- /dev/null +++ b/Exposing development headers on Nixos.md @@ -0,0 +1,15 @@ +--- +tags: + - nixos +--- +On regular distro's such as Ubuntu we would install them system-wide, for example, `apt install libx` + +On NixOS these kind of outputs (on search.nixos.org the pkgs should specify out: dev) are not exposed by default, so installing the package system wide doesn't work. + +Instead they get loaded when the package is installed in a nix shell: +``` +[nix-shell:~/projects/group.one/vps/openstack/puppet-neutron]$ cat shell.nix +{ pkgs ? import {}}: pkgs.mkShell { + packages = [ pkgs.augeas pkgs.pkg-config pkgs.libxml2 ]; +} +``` \ No newline at end of file diff --git a/Freenet CPU steal graph.md b/Freenet CPU steal graph.md index 1172c0b..b18c662 100644 --- a/Freenet CPU steal graph.md +++ b/Freenet CPU steal graph.md @@ -5,4 +5,4 @@ tags: --- `https://prometheus3.fnrz.de/graph?g0.expr=topk(5%2C%20avg_over_time(cpu_usage_steal%7Bcpu%3D%22cpu-total%22%7D%5B10m%5D))&g0.tab=0&g0.stacked=0&g0.show_exemplars=0&g0.range_input=12h` -`sudo ip r a 194.97.212.67 via 172.16.16.1` +`s i` diff --git a/How to (RIP) Grep.md b/How to (RIP) Grep.md deleted file mode 100644 index f7122a3..0000000 --- a/How to (RIP) Grep.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -tags: - - howto - - grep ---- - -`grep -vxf gobs_servers onecrm_servers` grep all onecrm_servers that **ARE NOT** in gobs_servers -`grep -oxf gobs_servers onecrm_servers` grep all onecrm_servers that **ARE** in gobs_servers -`grep -vxf onecrm_servers gobs_servers` grep all gobs_servers that **ARE NOT** in onecrm_servers -`grep -oxf onecrm_servers gobs_servers` grep all gobs_servers that **ARE** in onecrm_servers - - -# Useful Ripgrep flags - - `-L` to make it follow symlinks: this is handy for example when searching hieradata, because node definitions can be symlinked. - -`-l` to only print filenames that contain matches. \ No newline at end of file diff --git a/How to RIP (grep).md b/How to RIP (grep).md new file mode 100644 index 0000000..f7122a3 --- /dev/null +++ b/How to RIP (grep).md @@ -0,0 +1,17 @@ +--- +tags: + - howto + - grep +--- + +`grep -vxf gobs_servers onecrm_servers` grep all onecrm_servers that **ARE NOT** in gobs_servers +`grep -oxf gobs_servers onecrm_servers` grep all onecrm_servers that **ARE** in gobs_servers +`grep -vxf onecrm_servers gobs_servers` grep all gobs_servers that **ARE NOT** in onecrm_servers +`grep -oxf onecrm_servers gobs_servers` grep all gobs_servers that **ARE** in onecrm_servers + + +# Useful Ripgrep flags + + `-L` to make it follow symlinks: this is handy for example when searching hieradata, because node definitions can be symlinked. + +`-l` to only print filenames that contain matches. \ No newline at end of file diff --git a/How to show the mysql galera cluster status.md b/How to show the mysql galera cluster status.md new file mode 100644 index 0000000..27bb50f --- /dev/null +++ b/How to show the mysql galera cluster status.md @@ -0,0 +1,14 @@ +--- +tags: + - mysql + - galera + - howto +--- +Can be queried using `SHOW STATUS`. + +`sudo mysql --exec "SHOW STATUS LIKE 'wsrep%'"` + +**wsrep_cluster_status:** +- Primary; desired state. Primary component (so the cluster is one big component). Quorum of nodes is present. +- Non-Primary; Part of a partition (the non primary one) of nodes, so the cluster is partitioned. Cluster lost quorum. +- Disconnected: The node is not connected to the cluster at all. \ No newline at end of file diff --git a/Importing GPG keys from a keyserver.md b/Importing GPG keys from a keyserver.md new file mode 100644 index 0000000..d8981b2 --- /dev/null +++ b/Importing GPG keys from a keyserver.md @@ -0,0 +1,6 @@ +--- +tags: + - gpg + - howto +--- +`gpg --keyserver --recv-keys FINGERPRINT` \ No newline at end of file diff --git a/It's recommended to use small-medium ARC for hypervisors where ZFS is used.md b/It's recommended to use small-medium ARC for hypervisors where ZFS is used.md new file mode 100644 index 0000000..a92d3d7 --- /dev/null +++ b/It's recommended to use small-medium ARC for hypervisors where ZFS is used.md @@ -0,0 +1,10 @@ +--- +tags: + - architecture + - nova + - zfs + - arc +--- +https://papers.freebsd.org/2019/FOSDEM/jude-eli5_zfs_caching.files/ELI5_ZFS_ARC.pdf + +What is small-medium? TBD. For now we will go with the default of 1/32th of the total memory. diff --git a/Managing the amount of WSGI processes ran by apache is done from the apache site configuration.md b/Managing the amount of WSGI processes ran by apache is done from the apache site configuration.md new file mode 100644 index 0000000..6990073 --- /dev/null +++ b/Managing the amount of WSGI processes ran by apache is done from the apache site configuration.md @@ -0,0 +1,7 @@ +--- +tags: + - openstack + - wsgi + - apache +--- +Instead of configuring for example `octavia.conf` the number of wsgi workers is controller via the httpd config at `/etc/apache2/sites-available/10-octavia_wsgi.conf`.[]() \ No newline at end of file diff --git a/Testing Golang programs.md b/Testing Golang programs.md new file mode 100644 index 0000000..e9d48de --- /dev/null +++ b/Testing Golang programs.md @@ -0,0 +1,9 @@ +--- +tags: + - golang + - howto +--- +`go test` reads files ending with `_test.go`: 3 specialized functions: +- Test functions; names start with Test, 1 arg. \*testing.T +- Benchmarks: names start with Benchmarkt, 1 arg: \*testing.B +- Example functions: names start with Example \ No newline at end of file diff --git a/Working with Nix shells.md b/Working with Nix shells.md new file mode 100644 index 0000000..c042edd --- /dev/null +++ b/Working with Nix shells.md @@ -0,0 +1,66 @@ +Basic invocation: `nix-shell -p [ pkgs ... ]` + +# Shebang +``` +#! /usr/bin/env nix-shell +#! nix-shell -i bash -p bash + +do stuff +``` + +Different interpreters can be set with the `-i` option such as python +``` +#! /usr/bin/env nix-shell +#! nix-shell -i python3 -p python3 + +print("hello world") +``` + +The `-I` option can be used to pin nixpkgs +``` +#! /usr/bin/env nix-shell +#! nix-shell -i bash +#! nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/aed4b19d312525ae7ca9bceb4e1efe3357d0e2eb.tar.gz + +echo hello world +``` +# Shell.nix +Can be used to set up per dir environments, e.g using direnv to automatically activate them. +## Python +https://nixos.org/manual/nixpkgs/stable/#python + +```nix +{ pkgs ? import {}}: + +pkgs.mkShell { + packages = [ pkgs.virtualenv ]; +} +``` + +```nix +with import {}; +let + my_toolz = python311.pkgs.buildPythonPackage rec { + pname = "toolz"; + version = "0.10.0"; + pyproject = true; + src = fetchPypi { + inherit pname version; + hash = "sha256-CP3V73yWSArRHBLUct4hrNMjWZlvaaUlkpm1QP66RWA="; + }; + nativeBuildInputs = [ + python311.pkgs.setuptools + python311.pkgs.wheel + ]; # has no tests + doCheck = false; + meta = { + homepage = "https://github.com/pytoolz/toolz/"; + description = "List processing tools and functional utilities"; + # [...] + }; + }; +in python311.withPackages (ps: with ps; [ numpy my_toolz ]) ).env +``` + + The [[Import]] is required here because imports a nix expression from another source, in this case nixpkgs. +It is not required at line two, because python310Packages itself is already in the local scope. \ No newline at end of file diff --git a/daily/16-Apr-2025.md b/daily/16-Apr-2025.md new file mode 100644 index 0000000..2d22b55 --- /dev/null +++ b/daily/16-Apr-2025.md @@ -0,0 +1,15 @@ +1op1 rutger: niet echt iets besproken. + +***Interview Prep Isabel*** +*Do you live in Amsterdam? If yes, how long? plans to stay?* + +*Very shortly worked for ING (2025 january until now). What happened?* + +*At Civir you mention "deployent & administration of cloud technologies" including openstack, does this mean you deplmoyed an openstack cloud or were you a user of an openstack deployment?* + + +*You mention "24/7 support for troubleshooting issues" on multiple positions. What kind of issues?* + +***Isabel Q to us*** + +**Isabel kwam niet opdagen** \ No newline at end of file diff --git a/daily/17-Apr-2025.md b/daily/17-Apr-2025.md new file mode 100644 index 0000000..7a0a296 --- /dev/null +++ b/daily/17-Apr-2025.md @@ -0,0 +1,2 @@ +`puppet-neutron` gemerged; gezeik met updaten dependency, uiteindelijk commit hash in lock geupdate. Zie [[Debugging issues with updating Puppet dependency]]. + diff --git a/daily/18-Apr-2025.md b/daily/18-Apr-2025.md new file mode 100644 index 0000000..2b9620d --- /dev/null +++ b/daily/18-Apr-2025.md @@ -0,0 +1,6 @@ +Encountered oom on lxchosts. Turned out that octavia wsgi was using huge amounts of ram. +First we disabled the apache on all octavia to prevent more oom kills. +After that Erik limited their allowed memory usage, and we turned them back on. +Found out that we can see what script is ran by apache in the vhost config. Turned out to be some CGI script. +To profile the memory usage I stopped apache and ran a memory profiler directly against the cgi script, I had to stop the LB from using TLS but apart from that it worked smoothly. +We now have a flamegraph of the memory usage, and it looks like it is something to do with ovs. \ No newline at end of file -- cgit v1.2.3