vault backup: 2025-03-20 11:07:48

20 files changed, 471 insertions, 0 deletions

diff --git a/2 Areas/Werk/VPS platform/Add new provider networks.md b/2 Areas/Werk/VPS platform/Add new provider networks.md
new file mode 100644
index 0000000..cd437fd
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/Add new provider networks.md
@@ -0,0 +1,112 @@
+#openstack #network 
+# Schematic on switch network
+ ![[Switch-network]]
+# Procedure
+Kevin configures the switches so that the public network is routed to the correct private network and sets up a VLAN.
+
+By now we should have a VLAN tag and a private subnet that will be used, for example we will use VLAN tag 150 as well as subnet 10.8.4.0/2.
+# Make sure VLAN interface exists on network node
+> For new network nodes this is done with Ansible, however for fear of disrupting live traffic we prefer to add additional ones on existing nodes by hand.
+
+Check whether an interface exists on the bond for the given vlan (e.g `bond0.150` given VLAN tag 150).
+
+If not add an entry in `/etc/network/interfaces` so it survives reboots:
+```
+auto bond0.150
+iface bond0.150 inet manual
+  vlan-raw-device bond0
+```
+And then of course we add this interface with `sudo ifup bond0.150` with 150 being the VLAN tag we've been given.
+# Create switch network on openstack
+Define the switch network and OVN mapping in hieradata. Make sure to run Puppet on relevant controllers and network nodes.
+```YAML
+group/os-onecom-os1.yaml
+profile::openstack::neutron::controller::networks:
+  switch-network-vps4-cph8:
+    provider_network_type: flat
+    provider_physical_network: switch-network-vps4-cph8
+    router_external: true
+    shared: false
+    project_id: bb8fd38613c6464e8c00cbc332e2c67d
+
+domain/network.env.vps4-cph8.one.com.yaml
+profile::openstack::neutron::ovn::controller::bridge_interface_mappings:
+  - 'ext-br150:bond0.150'
+profile::openstack::neutron::ovn::controller::ovn_bridge_mappings:
+  - 'switch-network-vps4-cph8:ext-br150'
+```
+
+>When adding an external or public network openstack will automatically create a RBAC policy that allows any project to access it. Make sure it is removed: `openstack network rbac list --target-project '*'` will contain an entry with **object type network**. Show it, make sure it's the switch network, and delete it.
+
+>The Puppet module used for creating networks assigns the largest possible MTU to a network. We require it to be set to 1500. After changing the MTU to 1500 disable and enable DHCP so that the DHCP server also takes the configuration in effect.
+# Create switch subnets on openstack
+Once that's taken care of we can add the switch subnets to neutron via hieradata, usually a group yaml (e.g `group/os-onecom-os1)
+```yaml
+profile::openstack::neutron::controller::subnets:
+	switch-subnet-vps4-cph8-ipv4:
+	    cidr: 10.8.4.0/24
+	    ip_version: 4
+	    allocation_pools: [ 'start=10.8.4.4,end=10.8.4.254' ]
+	    gateway_ip: 10.8.4.1
+	    network_name: switch-network-vps4-cph8
+	    project_id: bb8fd38613c6464e8c00cbc332e2c67d
+	switch-subnet-vps4-cph8-ipv6:
+	    cidr: 2a02:2350:a:105::/64
+	    ip_version: 6
+	    allocation_pools: [ 'start=2a02:2350:a:105::4,end=2a02:2350:a:105::ffff' ]
+	    gateway_ip: 2a02:2350:a:105::1
+	    network_name: switch-network-vps4-cph8
+	    project_id: bb8fd38613c6464e8c00cbc332e2c67d
+	    ipv6_address_mode: dhcpv6-stateful
+	    ipv6_ra_mode: dhcpv6-stateful
+```
+
+> We want to have AZ reflected in the switch network name as shown in our example above "switch-network-vps4-cph8". Older switch-networks do not yet follow this convention.
+
+> We reserve the first three and last one IP in the pool of a given /24. Hence the allocation pool starts at .4 and ends at .254. These IP's are reserved for routers & switches, for example the .1 is assigned to the gateway.
+
+> When running Puppet on the controller node to create the subnet it can happen that it complains that the subnet overlaps with another. It might be caused because another controller is running Puppet at the same time and it created the subnet before your run.
+
+# Create network and subnet
+In the correct group YAML we define the actual network and subnet that are to be used by tenants.
+Example (`group/os-onecom-os1.yml`)
+```
+profile::openstack::neutron::controller::networks:
+  hostnet_185_95_25:
+    router_external: false
+    shared: false
+    project_id: 5e9dbdce473543e093fb90c3db5cd8f3
+
+profile::openstack::neutron::controller::subnets:
+  hostnet_185_95_25_ipv4:
+    allocation_pools:
+      - start=185.95.25.2,end=185.95.25.254
+    cidr: 185.95.25.0/24
+    dns_nameservers:
+      - 91.184.1.11
+      - 91.184.8.21
+    gateway_ip: 185.95.25.1
+    ip_version: '4'
+    network_name: hostnet_185_95_25
+    project_id: 5e9dbdce473543e093fb90c3db5cd8f3
+```
+
+> Make sure these are added to the correct tenant project
+
+# Create router on openstack
+`openstack router create --external-gateway switch-network-vps4-cph8 --fixed-ip subnet_id=switch-subnet-vps4-cph8-ipv4,ip-address=10.8.4.4 --fixed-ip=switch-subnet-vps4-cph8-ipv6,ip-address='2a02:2350:a:105::4' --disable-snat switch-network-vps4-cph8`
+# Ensure reverse DNS zone
+We should make sure the reverse DNS zone is added to the `service.g1-dns.one` zone to ensure PTR records can be added via SysAPI. 
+```shell
+~
+❯ dig +short 25.95.185.in-addr.arpa DS
+4550 13 4 6BFEE8B7692B15EC8EE01C17CF3F7FDD68F2F4A7581B7606A0CDB44A BDFE7BB171763C66938DFB285D4BF8680EA81B74
+4550 13 2 ADC65456F034323B3F1F3F010E637A04AB78B59D0176BE2B17702626 22B3AA39
+
+~
+❯ dig +short 25.95.185.in-addr.arpa SOA
+auth.g1-dns.one. hostmaster.one.com. 2024011601 1800 900 1209600 300
+```
+Should be via `service.g1-dns.one` rather than `auth`. We can make a ticket in SYSDNS to have it corrected; [example](https://group-one.atlassian.net/browse/SYSDNS-510).
+
+> Do mention in that ticket that we handle RIPE to prevent them from asking :)
diff --git a/2 Areas/Werk/VPS platform/Backing files.md b/2 Areas/Werk/VPS platform/Backing files.md
new file mode 100644
index 0000000..e9c76bd
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/Backing files.md
@@ -0,0 +1,10 @@
+---
+tags:
+  - virsh
+  - kvm
+  - libvirt
+  - linux
+  - "#openstack"
+---
+Backing files can't be disabled. 
+`virsh blockpull domain /path/to/disk` to pull base file into qcow if we have a large snapshot backing file.
\ No newline at end of file
diff --git a/2 Areas/Werk/VPS platform/Backup service/Backup verwijderen faalt.md b/2 Areas/Werk/VPS platform/Backup service/Backup verwijderen faalt.md
new file mode 100644
index 0000000..c7c42a1
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/Backup service/Backup verwijderen faalt.md
@@ -0,0 +1,52 @@
+#groupone #openstack #backup-service #bug
+
+---
+# Summary
+We openen libvirt connection alleen tijdens het starten van het proces. Daarna niet meer. Dus connection was closed door iets -> elk opvolgend request faalt.
+
+**Oplossing**
+Connections e.d openen per request.
+
+# Onderzoek
+```shell
+2025-01-06 11:00:42.760 3562 INFO goba.cmd.agent [None req-3ac13f90-ebe4-482b-82ef-fded0df9be87 - - - - -] action='delete' type='backup' task_uuid='1957362f-44ec-475b-a5a1-96b53aa8be60'
+2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent [-] internal error: client socket is closed: libvirt.libvirtError: internal error: client socket is closed
+2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent Traceback (most recent call last):
+2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent   File "/usr/lib/python3/dist-packages/goba/cmd/agent.py", line 137, in execute
+2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent     execute_fn(ctx)
+2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent   File "/usr/lib/python3/dist-packages/goba/cmd/agent.py", line 192, in execute_fn
+2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent     backup.delete(req, self.storage, self.libvirt)
+2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent   File "/usr/lib/python3/dist-packages/goba/backup.py", line 332, in delete
+2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent     if not libvirt_client.is_running(instance_id):
+2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent   File "/usr/lib/python3/dist-packages/goba/adapters/libvirt.py", line 134, in is_running
+2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent     dom_state = self._get_domain(instance_id).state()[0]
+2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent   File "/usr/lib/python3/dist-packages/libvirt.py", line 3146, in state
+2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent     raise libvirtError('virDomainGetState() failed')
+2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent libvirt.libvirtError: internal error: client socket is closed
+2025-01-06 11:00:42.763 3562 ERROR goba.cmd.agent
+```
+
+```
+[jasras@n14.compute.vps1-lej1 ~]$ systemctl status goba
+● goba.service - Group.one OpenStack Backup Agent
+     Loaded: loaded (/lib/systemd/system/goba.service; enabled; vendor preset: enabled)
+     Active: active (running) since Thu 2024-12-12 06:26:50 UTC; 3 weeks 4 days ago
+       Docs: https://gitlab.group.one/groupvps/group-one-backup-agent
+   Main PID: 3562 (goba)
+      Tasks: 86 (limit: 4915)
+     Memory: 166.9M
+        CPU: 2h 23min 28.981s
+     CGroup: /system.slice/goba.service
+             ├─ 3562 /usr/bin/python3 /usr/bin/goba --config-file /etc/goba/goba.conf
+             └─14368 /usr/bin/python3 /usr/bin/privsep-helper --config-file /etc/goba/goba.conf --privsep_context goba.privsep.file_admin_pctxt --privsep_sock_path /tmp/tmpjfi8jt6b/privsep.sock
+```
+Er draait hier nog een privsep-helper
+
+Na restart:
+```
+     CGroup: /system.slice/goba.service
+             └─30932 /usr/bin/python3 /usr/bin/goba --config-file /etc/goba/goba.conf
+```
+
+Tasks slagen nu wel.
+
diff --git a/2 Areas/Werk/VPS platform/Chef.md b/2 Areas/Werk/VPS platform/Chef.md
new file mode 100644
index 0000000..15f5ab6
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/Chef.md
@@ -0,0 +1,6 @@
+---
+tags:
+  - chef
+  - groupvps
+---
+https://sysdoc.one.com/base/chef/add-ons/chef-secret/
\ No newline at end of file
diff --git a/2 Areas/Werk/VPS platform/Compute VPS2-LEJ1 is mixed.md b/2 Areas/Werk/VPS platform/Compute VPS2-LEJ1 is mixed.md
new file mode 100644
index 0000000..d02702a
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/Compute VPS2-LEJ1 is mixed.md
@@ -0,0 +1,6 @@
+#compute #openstack
+
+---
+VPS2-LEJ1 is a mixed bag of shared and local storage;
+nodes 1-8 are BOTH shared and local storage
+nodes 9-10 are exclusively shared storage
diff --git a/2 Areas/Werk/VPS platform/FS aio.md b/2 Areas/Werk/VPS platform/FS aio.md
new file mode 100644
index 0000000..b47827a
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/FS aio.md
@@ -0,0 +1,22 @@
+---
+tags:
+  - sysctl
+  - aio
+  - linux
+  - filesystem
+references:
+  - https://docs.kernel.org/admin-guide/sysctl/fs.html
+  - https://man7.org/linux/man-pages/man2/io_setup.2.html
+  - https://sort.veritas.com/public/documents/HSO/2.0/linux/productguides/html/hfo_admin_ubuntu/ch04s03.htm
+---
+aio-nr & aio-max-nr:
+
+```
+aio-nr is the running total of the number of events specified on the
+io_setup system call for all currently active aio contexts.  If aio-nr
+reaches aio-max-nr then io_setup will fail with EAGAIN.  Note that
+raising aio-max-nr does not result in the pre-allocation or re-sizing
+of any kernel data structures.
+```
+
+Openstack compute fs.aio-max-nr naar `1048576` gezet zoals beschreven op het web.
\ No newline at end of file
diff --git a/2 Areas/Werk/VPS platform/Freenet network.md b/2 Areas/Werk/VPS platform/Freenet network.md
new file mode 100644
index 0000000..5e51f74
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/Freenet network.md
@@ -0,0 +1,6 @@
+Packet loss & retransmits between compute nodes.
+ Comp -> netw -> top of rack ->  netw -> compute
+ 
+ ![[FreenetNetwork.png]]
+
+Hope that OVN can direct route between 2 VMs. Otherwise an interleave will probably be the next best option.
\ No newline at end of file
diff --git a/2 Areas/Werk/VPS platform/FreenetNetwork.png b/2 Areas/Werk/VPS platform/FreenetNetwork.png
new file mode 100644
index 0000000..48bac63
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/FreenetNetwork.png
diff --git a/2 Areas/Werk/VPS platform/Get server uuid by dm.md b/2 Areas/Werk/VPS platform/Get server uuid by dm.md
new file mode 100644
index 0000000..6b47970
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/Get server uuid by dm.md
@@ -0,0 +1,8 @@
+---
+tags:
+  - groupvps
+  - device-mapper
+---
+```
+for dm in dm-3 dm-50; do ps axf | grep "\"\/dev\/${dm}\"" | egrep -o [a-z0-9\-]{36} | head -1; done
+```
diff --git a/2 Areas/Werk/VPS platform/Issues/High storage load 05-12-2024.md b/2 Areas/Werk/VPS platform/Issues/High storage load 05-12-2024.md
new file mode 100644
index 0000000..7d26c6a
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/Issues/High storage load 05-12-2024.md
@@ -0,0 +1,15 @@
+#issue #groupone 
+
+---
+Vraag uitgezet bij Allan voor meer informatie.
+Allan: het komt al 1.5 maand voor; https://group-onecom.slack.com/archives/C02FT9KEFNH/p1729863978525299
+
+Uit dat bericht blijkt dat Jerry zou kijken of het Acronis is.
+
+Jeroen gecontacteerd om ff te vragen of ze dat al gedaan hebben; jeroen denkt dat het niet door Acronis komt omdat dat vooral read ops zijn en verspreid over de nacht.
+
+Allan stuurt graph met kantteking: "seems to involve alot of LUNs": `https://prometheus2.env.vps1-cph8.one.com/graph?g0.expr=count(lun_write_ops%20%3E%20(lun_write_ops%20offset%2020m%20%2B%20100))&g0.tab=0&g0.display_mode=lines&g0.show_exemplars=0&g0.range_input=30m&g0.end_input=2024-12-05%2005%3A47%3A35&g0.moment_input=2024-12-05%2005%3A47%3A35`
+
+Jeroen dacht mogelijk mysql dumps van shared, maar zou ook verspreid moeten zijn: ging kijken wanneer die gdraaid hadden, niks meer over gehoord.
+
+O.b.v LUN graph heb ik wat LUNs gepakt en bleek allemaal managed vps te zijn; Jeroen op onderzoek naar wat daar binnen rond die tijd gebeurd. Verdenkt PSA crons.
\ No newline at end of file
diff --git a/2 Areas/Werk/VPS platform/Logging.md b/2 Areas/Werk/VPS platform/Logging.md
new file mode 100644
index 0000000..a2f0a73
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/Logging.md
@@ -0,0 +1,11 @@
+---
+tags:
+  - logging
+  - groupvps
+  - "#openstack"
+---
+The logging context follows the default.
+`... [global-req-id req-id user project domain system_scope user_domain project_domain] ...`
+
+https://docs.openstack.org/oslo.log/latest/configuration/index.html#DEFAULT.logging_context_format_string
+https://docs.openstack.org/oslo.log/latest/configuration/index.html#DEFAULT.logging_context_format_string
\ No newline at end of file
diff --git a/2 Areas/Werk/VPS platform/Maintenance/10-12-2024.md b/2 Areas/Werk/VPS platform/Maintenance/10-12-2024.md
new file mode 100644
index 0000000..c7f9824
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/Maintenance/10-12-2024.md
@@ -0,0 +1,5 @@
+#maintenance
+
+---
+manually updated instance record host/node for instance `10924c62-7f0f-4df1-9dd8-9108e3cb0764`
+suspended guest: virsh dompmwakeup and try again
diff --git a/2 Areas/Werk/VPS platform/OVN.md b/2 Areas/Werk/VPS platform/OVN.md
new file mode 100644
index 0000000..369dc95
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/OVN.md
@@ -0,0 +1,18 @@
+---
+tags:
+  - openstack
+  - ovn
+  - groupvps
+references:
+  - https://docs.redhat.com/en/documentation/red_hat_openstack_platform/12/html/networking_guide/sec-ovn#sec-ovn
+  - https://docs.ovn.org/en/latest/intro/install/ovn-upgrades.html
+  - https://discuss.linuxcontainers.org/t/ovn-cluster-init-issue/11080
+---
+https://dani.foroselectronica.es/ovn-where-is-my-packet-665/g
+
+OVN NB & SB DB's:
+```
+systemctl status ovn-ovsdb-server-sb
+systemctl status ovn-ovsdb-server-nb
+```
+They were stopped after the last upgrade to OVN 24.09.
\ No newline at end of file
diff --git a/2 Areas/Werk/VPS platform/Our image updater.md b/2 Areas/Werk/VPS platform/Our image updater.md
new file mode 100644
index 0000000..73eff81
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/Our image updater.md
@@ -0,0 +1,9 @@
+#groupone #openstack 
+
+---
+https://gitlab.group.one/groupvps/openstack-image-updater
+
+Updater runs in CI.
+
+`images.yaml`; contains which images
+`cloud_images.yaml`; can override params for all images in a specific cloud, sadly cannot override per specific image.
diff --git a/2 Areas/Werk/VPS platform/Prometheus scraping.md b/2 Areas/Werk/VPS platform/Prometheus scraping.md
new file mode 100644
index 0000000..3b87bfd
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/Prometheus scraping.md
@@ -0,0 +1,67 @@
+---
+tags:
+  - groupvps
+  - firewall
+  - "#prometheus"
+---
+
+# Create scrape config
+`roles/prometheus/hostnet/onecom-prometheus-scrape-config-vps-gobs.json`
+
+# Testing the endpoint
+https://prometheus1.env.vps1-lej1.one.com/targets?search=&scrapePool=gobs
+
+# Metrics only accessible over HTTPS
+```
+commit e1e36bd073c8faf777577a41eb2eb67035e917b2 (HEAD -> master, upstream/master, upstream/HEAD, origin/master)
+Author: Ihor Piddubnyak <ihp@one.com>
+Date:   Mon Jan 27 13:24:30 2025 +0100
+
+    onecom-prometheus-relabel-config-vps-gobs role to scrape https from gobs
+
+diff --git a/roles/prometheus/hostnet/onecom-prometheus-relabel-config-vps-gobs.json b/roles/prometheus/hostnet/onecom-prometheus-relabel-config-vps-gobs.json
+new file mode 100644
+index 0000000000..a18adccf7c
+--- /dev/null
++++ b/roles/prometheus/hostnet/onecom-prometheus-relabel-config-vps-gobs.json
+@@ -0,0 +1,21 @@
++{
++    "name": "onecom-prometheus-relabel-config-vps-gobs",
++    "description": "Role for configuration of Prometheus gabs scrape to get metrics with https",
++    "json_class": "Chef::Role",
++    "default_attributes": {
++        "prometheus": {
++            "server": {
++                "scrape_configs": {
++                    "gobs": {
++                        "scheme": "https",
++                        "tls_config": {
++                            "insecure_skip_verify": true
++                        }
++                    }
++                }
++            }
++        }
++    },
++    "chef_type": "role"
++}
++
+diff --git a/roles/prometheus/onecom-prometheus-server.json b/roles/prometheus/onecom-prometheus-server.json
+index 6b430e4171..1089f6bb73 100644
+--- a/roles/prometheus/onecom-prometheus-server.json
++++ b/roles/prometheus/onecom-prometheus-server.json
+@@ -53,6 +53,7 @@
+         "role[pl-onecom-prometheus]",
+         "role[onecom-prometheus-scrape-config-thanos-sidecar]",
+         "role[onecom-prometheus-relabel-config-thanos]",
++        "role[onecom-prometheus-relabel-config-vps-gobs]",
+         "role[onecom-prometheus-pdns-passwd]",
+
+
+```
+
+# Zeh Firewall
+https://gitlab.group.one/systems/chef-repo/-/merge_requests/2838
+
+base servers need to have a fw rule added to allow traffic out from prom to service/port.
+if in same vlan no need for a rule on the gateways
diff --git a/2 Areas/Werk/VPS platform/Query duplicate volume connectors.md b/2 Areas/Werk/VPS platform/Query duplicate volume connectors.md
new file mode 100644
index 0000000..1c976ac
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/Query duplicate volume connectors.md
@@ -0,0 +1,16 @@
+---
+tags:
+  - groupvps
+  - datacorruption
+  - volume-connector
+---
+```mysql
+select volume_id, count(*) as con from volume_attachment where deleted_at is null group by volume_id having con > 1 ;
+```
+
+and to find the ones in use
+```mysql
+select * from volume_attachment where deleted_at is NULL and volume_id = "e9ee5691-a3cd-4696-9db9-bda17c2cf664" \G
+```
+
+Take care that the server is not **MIGRATING** because it will have 2 active attachments then.
\ No newline at end of file
diff --git a/2 Areas/Werk/VPS platform/Rankmath mongodb.md b/2 Areas/Werk/VPS platform/Rankmath mongodb.md
new file mode 100644
index 0000000..84f877e
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/Rankmath mongodb.md
@@ -0,0 +1,7 @@
+---
+tags:
+  - groupvps
+  - rankmath
+---
+Rankmath runs a MongoDB server on our platform; `mongodb-stg1.g1i.rankmath.com`.
+They have made requests for IP whitelisting, which is just adding it to a sec grp for that vm.
\ No newline at end of file
diff --git a/2 Areas/Werk/VPS platform/SSL certificates.md b/2 Areas/Werk/VPS platform/SSL certificates.md
new file mode 100644
index 0000000..2696415
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/SSL certificates.md
@@ -0,0 +1,7 @@
+---
+tags:
+  - groupvps
+references:
+  - https://group-one.atlassian.net/jira/software/c/projects/SYSWEB/boards/331/backlog
+---
+Create a ticket at the SYSWEB JIRA board requesting a LetsEncrypt certificate for the given hostname.
\ No newline at end of file
diff --git a/2 Areas/Werk/VPS platform/Snapshot restore procedure.md b/2 Areas/Werk/VPS platform/Snapshot restore procedure.md
new file mode 100644
index 0000000..6a63949
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/Snapshot restore procedure.md
@@ -0,0 +1,17 @@
+---
+tags:
+  - goba
+  - snapshot
+---
+stop server if active
+
+clone snapshot -> out comes a filesystem -> `{fs-name}-clone`
+- `sudo zfs clone <snapshot> <fs-name(-clone)>`
+promote said filesystem
+- `sudo zfs promote <fs-name(-clone)>`
+rename original fs
+- `sudo zfs rename <fs-name> <fs-name-datetime>`
+rename cloned snapshot to original fs
+- `sudo zfs rename <fs-name(-clone)> <fs-name>`
+
+start server if it was active
diff --git a/2 Areas/Werk/VPS platform/VM cannot be rescued.md b/2 Areas/Werk/VPS platform/VM cannot be rescued.md
new file mode 100644
index 0000000..934da0f
--- /dev/null
+++ b/2 Areas/Werk/VPS platform/VM cannot be rescued.md
@@ -0,0 +1,77 @@
+---
+tags:
+  - rescue
+  - groupvps
+---
+## Error cause
+`{'code': 400, 'created': '2025-02-10T10:38:39Z', 'message': 'Instance 44240d84-52ca-4474-b4ac-163bd1ba2def cannot be rescued: Driver Error: Image 9115b741-6eb3-4574-894d-9f2e28600ff8 could not be found.'}`
+
+Likely a deleted snapshot!
+## Show BDM in DB
+```
+MariaDB [nova]> select image_id from block_device_mapping where instance_uuid = "44240d84-52ca-4474-b4ac-163bd1ba2def";
++--------------------------------------+
+| image_id                             |
++--------------------------------------+
+| 9115b741-6eb3-4574-894d-9f2e28600ff8 |
++--------------------------------------+
+1 row in set (0.000 sec)
+```
+
+## Find base image
+```
+MariaDB [glance]> select * from image_properties where image_id = "9115b741-6eb3-4574-894d-9f2e28600ff8" and name = "base_image_ref" \G
+*************************** 1. row ***************************
+        id: 24353
+  image_id: 9115b741-6eb3-4574-894d-9f2e28600ff8
+      name: base_image_ref
+     value: a689d899-903d-49be-96ab-6ac638bbc5fd
+created_at: 2024-07-20 11:09:34
+updated_at: 2024-08-04 14:45:12
+deleted_at: 2024-08-04 14:45:12
+   deleted: 1
+1 row in set (0.000 sec)
+```
+... repeat until its base image is not snapshot.
+
+## Update references
+```
+MariaDB [nova]> select image_ref from instances where uuid = "44240d84-52ca-4474-b4ac-163bd1ba2def";
++--------------------------------------+
+| image_ref                            |
++--------------------------------------+
+| a689d899-903d-49be-96ab-6ac638bbc5fd |
++--------------------------------------+
+1 row in set (0.000 sec)
+
+MariaDB [nova]> select image_id from block_device_mapping where instance_uuid = "44240d84-52ca-4474-b4ac-163bd1ba2def";
++--------------------------------------+
+| image_id                             |
++--------------------------------------+
+| a689d899-903d-49be-96ab-6ac638bbc5fd |
++--------------------------------------+
+1 row in set (0.000 sec)
+
+update instance_system_metadata set value = "a689d899-903d-49be-96ab-6ac638bbc5fd" where instance_uuid = "44240d84-52ca-4474-b4ac-163bd1ba2def" and `key` = "image_base_image_ref";
+```
+
+## Notes on specific case
+44240d84-52ca-4474-b4ac-163bd1ba2def instance_system_metadata geupdate maar alleen het image_base_image_ref geupdatet.
+De rest van de image metadata is nog alsof het het oude snapshot.
+Misschien kunnen we nova-manage gebruiken om het te updaten:
+```
+[jasras@n07.compute.prv.vps1-testpod-cph3.one.com ~]$ sudo nova-manage image_property set --help
+Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
+usage: nova-manage image_property set [-h] [--property <image_property>] <instance_uuid>
+
+Set the values of instance image properties stored in the database. This is only allowed for instances with a STOPPED, SHELVED or SHELVED_OFFLOADED vm_state.
+
+positional arguments:
+  <instance_uuid>       UUID of the instance
+
+options:
+  -h, --help            show this help message and exit
+  --property <image_property>
+                        Image property to set using the format name=value. For example: --property hw_disk_bus=virtio --property hw_cdrom_bus=sata
+
+```
\ No newline at end of file