This is being done on FreeBSD 14.2-RELEASE-p3.

First I had to install certbot, pip and the appropriate Python module:

pkg install -y py311-certbot py311-pip
pip install certbot-dns-desec

pip will warn against installing system-wide packages and I would generally agree, but this time I will go against the warning. At some point I will look into making my own custom port for the deSEC module, but today is not that day.

Next step is to log in to deSEC and create a new token. No extra permissions are required. Create a place for storing the token:

mkdir /usr/local/etc/letsencrypt/secrets
chmod 600 /usr/local/etc/letsencrypt/secrets

Save the token in /usr/local/etc/letsencrypt/secrets/domain.tld.ini:

dns_desec_token = <token>

And fix permissions:

chmod 600 /usr/local/etc/letsencrypt/secrets/domain.tld.ini

Now try and request a certificate:

certbot certonly --authenticator dns-desec --dns-desec-credentials /usr/local/etc/letsencrypt/secrets/domain.tld.ini -d "domain.tld" -d "*.domain.tld"

If you check in the deSEC interface at the same time you'll see new _acme-challenge records have been published.

The first time I did this I got an error about the new records not existing, but after re-trying the command again the certificate was created. I don't have an explanation for this, but it felt like the Let's Encrypt servers couldn't find the records the first time around.

Finally, to renew the certificate automatically this can be added to /etc/periodic.conf:

weekly_certbot_enable="YES"
weekly_certbot_post_hook="service nginx restart"     # if you're running nginx
# or
weekly_certbot_post_hook="service apache24 restart"  # if you're running apache

See /usr/local/etc/periodic/weekly/500.certbot-3.11 for some more configuration options.

This will for instance fetch all active devices which has the tag network_edge:

plugin: netbox.netbox.nb_inventory
api_endpoint: https://netbox.example.net
token: token_from_netbox

query_filters:
  - status: active
  - tag: network_edge

Let's check what data we get back (very much truncated):

$ ansible-inventory -v --list -i inventory/netbox.yml

{
    "_meta": {
        "hostvars": {
            "router1": {
                "ansible_host": "2001:db8::1",
                "platforms": [
                    "cisco-ios-xr"
                ],
                "primary_ip4": "192.0.2.1",
                "primary_ip6": "2001:db8::1",
                "status": {
                    "label": "Active",
                    "value": "active"
                },
                "tags": [
                    "network_edge"
                ]
            },
            "router2": {
                "ansible_host": "2001:db8::2",
                "platforms": [
                    "arista-eos"
                ],
                "primary_ip4": "192.0.2.2",
                "primary_ip6": "2001:db8::2",
                "status": {
                    "label": "Active",
                    "value": "active"
                },
                "tags": [
                    "network_edge"
                ]
            },
        }
    },
    "all": {
        "children": [
            "ungrouped"
        ]
    },
    "ungrouped": {
        "hosts": [
            "router1",
            "router2"
        ]
    }
}

Unfortunately we can't pass this inventory to Ansible directly, since Ansible needs the ansible_network_os and ansible_connection variables.

The method I ended up with takes the platform from the output above, groups the devices together and then uses group_vars to set the connection settings.

Grouping the devices

We can use the keyed_groups option in the plugin to group devices together, based on something (in this case the platform). Something like this:

keyed_groups:
  - key: platform.slug
    prefix: platform

This would give the following output:

{
    "_meta": {
        "hostvars": {
            "router1": {
                "ansible_host": "2001:db8::1",
                "platforms": [
                    "cisco-ios-xr"
                ],
                "primary_ip4": "192.0.2.1",
                "primary_ip6": "2001:db8::1",
                "status": {
                    "label": "Active",
                    "value": "active"
                },
                "tags": [
                    "network_edge"
                ]
            },
            "router2": {
                "ansible_host": "2001:db8::2",
                "platforms": [
                    "arista-eos"
                ],
                "primary_ip4": "192.0.2.2",
                "primary_ip6": "2001:db8::2",
                "status": {
                    "label": "Active",
                    "value": "active"
                },
                "tags": [
                    "network_edge"
            }
        }
    },
    "all": {
        "children": [
            "ungrouped",
            "platform_arista_eos",
            "platform_cisco_ios_xr"
        ]
    },
    "platform_arista_eos": {
        "hosts": [
            "router2"
        ]
    },
    "platform_cisco_ios_xr": {
        "hosts": [
            "router1"
        ]
    }
}

Ansible group variables

There's already documentation available in the Ansible documentation about group variables, what they are and how they work.

But for our purpose we can use the groups platform_arista_eos and platform_cisco_ios_xr coming from the inventory to set connection settings, per platform.

Let's create the directories inventory/group_vars/platform_arista_eos and inventory/group_vars/platform_cisco_ios_xr, then create the file settings.yml inside both of them. I decided to put the following content into the files:

Arista:

ansible_network_os: arista.eos.eos
ansible_connection: ansible.netcommon.httpapi
proxy_env:
  http_proxy: http://proxy.example.com:8080

IOS-XR:

ansible_network_os: cisco.iosxr.iosxr
ansible_connection: ansible.netcommon.network_cli
ansible_iosxr_commit_comment: Committed by Ansible

Settings for the supported platforms can be found here.

Final execution

Let's run the ansible-inventory command one last time and check the output:

{
    "_meta": {
        "hostvars": {
            "router1": {
                "ansible_connection": "ansible.netcommon.network_cli",
                "ansible_host": "2001:db8::1",
                "ansible_iosxr_commit_comment": "Committed by Ansible",
                "ansible_network_os": "cisco.iosxr.iosxr",
                "platforms": [
                    "cisco-ios-xr"
                ],
                "primary_ip4": "192.0.2.1",
                "primary_ip6": "2001:db8::1",
                "status": {
                    "label": "Active",
                    "value": "active"
                },
            },
            "router2": {
                "ansible_connection": "ansible.netcommon.httpapi",
                "ansible_host": "2001:db8::2",
                "ansible_network_os": "arista.eos.eos",
                "platforms": [
                    "arista-eos"
                ],
                "primary_ip4": "192.0.2.2",
                "primary_ip6": "2001:db8::2",
                "proxy_env": {
                    "http_proxy": "http://proxy.example.com:8080"
                },
                "status": {
                    "label": "Active",
                    "value": "active"
                },
                "tags": [
                    "network_edge"
                ]
            }
        }
    }
}

Now the correct connection variables will be set for the devices, and the inventory can be passed to for instance ansible-playbook using the -i flag.

New platforms can be added easily by creating a new directory and a settings file.

AWX can use several of these environment containers, so depending on needs custom containers can be created that contain different Python packages, Ansible collections and so on.

I wanted to create a custom container so ensure a newer version of Ansible was available, ansible-pylibssh was installed and the collections I used was already installed when a playbook was started.

To do this some files are required, and the ansible-builder tool will be used to create the container.

The files

execution-environment.yml

This file defines the environment:

---
version: 3
images:
  base_image:
    name: quay.io/centos/centos:stream10
dependencies:
  ansible_core:
    package_pip: ansible-core
  ansible_runner:
    package_pip: ansible-runner
  galaxy: requirements.yml
  system: bindep.txt
  python: requirements.txt
additional_build_steps:
  append_base:
    - RUN $PYCMD -m pip install -U pip
  append_final:
    - RUN git lfs install --system

This will install the latest available version of ansible-core and ansible-runner, look for other requirements in the files specified and execute some commands.

bindep.txt

Contains the software that should be installed in the environment, from the OS package manager.

git-core [platform:rpm]
git-lfs [platform:rpm]
epel-release [platform:rpm]

requirements.yml

A list of Ansible collections to be installed from Galaxy.

---
collections:
  - name: ansible.netcommon
  - name: ansible.posix
  - name: ansible.utils
  - name: awx.awx
  - name: cisco.iosxr
  - name: containers.podman
  - name: netbox.netbox

requirements.txt

Python modules to be install via pip.

ansible-pylibssh

Create the environment

Once the files are created ansible-builder can be used to create the container:

ansible-builder build -v3 -t custom-environment

Depending on the number of dependencies this will take a while but once it's done you should have a container which can be tagged and used. I push it to a local container registry:

docker tag custom-environment ee:latest
docker push ee:latest

Using the environment

If your container registry requires a credential you have to create a "Container Registry" credential.

To add the environment to AWX, log in as an admin user and to go Administration and Execution Environments. Click the add button and fill in the fields as necessary.

Then ensure you pick the correct execution environment in your job templates.

AWX is required to run on a Kubernetes cluster, like k3s. Each execution of a playbook takes place inside of a pod which gets created and then destroyed once the playbook is done.

To ensure AWX can connect via a ssh jumphost we need to create a ConfigMap containing the relevant ssh configuration and a Secret containing the secret part of the ssh key used.

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: awx-ssh-config
  namespace: awx
data:
  default: |
    Host * !jumphost
      UserKnownHostsFile /dev/null
      StrictHostKeyChecking no
      HostKeyAlgorithms=+ssh-rsa
      KexAlgorithms=+ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
      Ciphers=+aes128-cbc
    Host jumphost
      Hostname jumphost.example.net
      User awx
      UserKnownHostsFile /dev/null
      StrictHostKeyChecking no
      IdentityFile /runner/.ssh/id_ed25519
---
kind: Secret
apiVersion: v1
metadata:
  name: awx-ssh-key
  namespace: awx
type: Opaque
data:
  default: <base64 encoded data of id_ed25519>

Apply the changes (kubectl apply -f .) and log in to AWX as an admin user.

Go to Administration, Instance Groups and modify the default group:

apiVersion: v1
kind: Pod
metadata:
  namespace: awx
spec:
  serviceAccountName: default
  automountServiceAccountToken: false
  containers:
    - image: quay.io/ansible/awx-ee:latest
      name: worker
      args:
        - ansible-runner
        - worker
        - '--private-data-dir=/runner'
      resources:
        requests:
          cpu: 250m
          memory: 100Mi
      volumeMounts:
        - name: ssh-config
          mountPath: /runner/.ssh/config
          subPath: default
        - name: ssh-key
          mountPath: /runner/.ssh/id_ed25519
          subPath: default
      securityContext:
        runAsUser: 1000
        runAsGroup: 0
  volumes:
    - name: ssh-config
      configMap:
        name: awx-ssh-config
        defaultMode: 0400
    - name: ssh-key
      secret:
        secretName: awx-ssh-key
        defaultMode: 0400
  securityContext:
    runAsUser: 1000
    runAsGroup: 0
    fsGroup: 0

Kubernetes is not my area of expertise but the ConfigMap and Secret gets mounted as files into the pod filesystem. The securityContext ensures the user inside of the pod can read the files, not sure how but it works.

Important to note is that the default execution environment (at least when I'm writing this) doesn't contain ansible-pylibssh so as noted in my [last post]({{% ref "posts/connect-to-network-devices-through-a-jumphost-with-ansible/" %}}) the relevant ssh settings might not work anyway.

In theory this is should be pretty easy but I spent so much time getting this to work. There's plenty of information about this subject out there but for some reason I couldn't get it to work. Hopefully my findings can help someone else, because this was not a fun experience.

This is my playbook:

---
- name: Jumphost testing
  hosts: all
  gather_facts: false

  tasks:
    - name: Gather facts
      cisco.iosxr.iosxr_facts:
      register: device_output

    - name: Print facts
      ansible.builtin.debug:
        msg:
          - "{{ device_output }}"

I execute it with this command:

ansible-playbook -i inventory/hosts.yml -u user -k playbooks/jumphost_test.yml

After I enter my password for the routers the playbook will fail with a connection refused error message. This is because Ansible is trying to connect directly, without jumping through a jumphost.

Ideally I want Ansible to use the settings in my ~/.ssh/config file automatically (like described here) but no matter what I couldn't get that to work.

I suspect using the ansible.netcommon.network_cli connection setting messes a bit with how the ssh settings are read, but I have no source for that. In the end I had to combine the two methods.

In ansible.cfg I had to make the following modification:

[ssh_connection]
ssh_common_args="-o ProxyCommand='ssh -W %h:%p -q jumphost'"

And in ~/.ssh/config:

Host * !jumphost
  HostKeyAlgorithms=+ssh-rsa
  KexAlgorithms=+diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1
  Ciphers=+aes128-cbc

Host jumphost
  Hostname jumphost.example.net
  User user
  IdentityFile ~/.ssh/id_ed25519

Some caveats I found along the way:

• I couldn't specify another ssh_config using -F.
• I couldn't specify a jumphost using -J, instead of using ProxyCommand.
• Specifying a User under Host * doesn't work, not sure what's happning but I have to use -u to ansible-playbook.
• Can't set ProxyJump under Host *, nothing seems to happen when I try to connect.

With these changes I'm now able to execute the playbook, and it will happily use the jumphost specified.

Update 2025-03-06

I did another test after committing this post.

My ansible.cfg looks like this:

[defaults]
transport=ssh

From what I can tell this means that Ansible should use ansible-pylibssh (which is backed by libssh) for ssh connections and if that Python package isn't available Ansible will fallback to Paramiko.

When I set transport=paramiko in the config Ansible didn't connect via my jumphost, so make sure ansible-pylibssh is installed.

• In order for the VM to receive a IPv4 address from UTMs internal DHCP server I had to change the network setting from Shared Network to Emulated VLAN.

  • This option doesn't exist when using Apple Virtualization.
  • The VM does receive a IPv6 address, but no default route. Not sure if there's any internal NAT64 magic going on, but if the rest of the network isn't IPv6 capable it's just easier to change the network mode.

• There's no option for port forwarding when using Apple Virtualization.

• When creating a port forward the order of the fields are guest address, guest port, host address, host port.

  • To create a local forward for SSH the guest address can be left empty, guest port is 22, host address set to 127.0.0.1 and host port to 2222. SSH to 127.0.0.1:2222 and you're connected to the VM.

• When sharing a host folder with the VM it can be mounted using mount -t 9p -o trans=virtio share /mnt/.

  • Or in /etc/fstab:
    • share /opt/dev 9p trans=virtio 0 0
  • share is a mount tag defined by UTM.

I couldn't get this to work with a already existing installation of Thunderbird so I had to remove all settings by removing (on MacOS) ~/Library/Thunderbird.

Go to Settings and scroll down to Config Editor. Search for mailnews.default.

Ensure these settings are set like this:

mailnews.default_sort_order -> 1
mailnews.default_sort_type  -> 18
mailnews.default_view_flags -> 1

Close the config editor and add an account. Don't close Thunderbird in between since any changed settings will be reverted.

If you're using Ansible and want to achieve this, here's a way to do it:

- name: Reset SSH connection
  ansible.builtin.meta:
    reset_connection

The post explains the concept of anycast, server configuration and how to setup BGP for this so I don't really have anything to add.

I do however wanted to expand on the ExaBGP configuration. What Mr Yeti does is querying the local DNS resolver for yetiops.net and checks the result. If the exit code is anything but 0 (meaning some kind of error happened) then the BGP route for the DNS resolver will be withdrawn from the network. Otherwise, announce the route and receive traffic from users.

This isn't a bad thing to do, but we're putting a lot of trust into the servers that host the zone being checked. If those servers are experiencing issues and the zone being checked isn't resolving the routes will be withdrawn, cutting off DNS for all the users even though there's no real problem with the resolver.

One option could be to check some other zone, but as we saw with Facebook back in 2021 the big giants can also fail, and the way I see it it's really just a matter of time. Make sure to read the link, it describes the problem quite well.

The option I went with instead is to always announce the IP address of the DNS resolver, but in case of a failure increase the MED to move the traffic away from the server. The server will still accept traffic (and throw it away in case of an actual error) but the network will choose not to send any.

What this means is that if a server encounters a local error (for instance crashing software) it will be taken out of rotation, but if there's a major fault with the zone being checked all servers will increase their MED and nothing will really change for the users.

Of course, if there's both a local fault and a fault upstream the server in question will cause issues for the users but hopefully this is something that can be caught using monitoring.

This is the ExaBGP configuration I'm using, there's more processes for secondary IPv4 addresses and IPv6 but in the end all checks copies from this:

process resolver_v4-check {
        run /usr/local/bin/python3.9 -m exabgp healthcheck \
                --cmd "/usr/local/bin/dig +timeout=1 +tries=1 -4 -t SOA google.com @192.0.2.53" \
                --interval 1 \
                --rise 30 \
                --fall 3 \
                --disable /usr/local/etc/exabgp/disable_resolver \
                --ip 192.0.2.53 \
                --next-hop 198.51.100.2 \
                --up-metric 0 \
                --down-metric 10000 \
                --disabled-metric 10000 \
                --up-execute "logger DNS Resolver 192.0.2.53 going into state UP" \
                --down-execute "logger DNS Resolver 192.0.2.53 going into state DOWN" \
                --disabled-execute "logger DNS Resolver 192.0.2.53 going into state DISABLED";
        encoder text;
}

neighbor 198.51.100.1 {
        router-id 198.51.100.2;
        local-address 198.51.100.2;
        local-as 65053;
        peer-as 64496;
        hold-time 31;

        family {
                ipv4 unicast;
        }

        api anycast_v4 {
                processes [ resolver_v4-check ];
        }
}

192.0.2.53 is the local IP address bound to the loopback interface. There's a check every second and the --fall parameter tells ExaBGP to increase the MED after three failed checks. Once check per second might be to exaggerate a bit but I want to react quickly on faults and the result from the upstream DNS server is cached locally so I don't feel bad for putting any extra load on them.

The --rise parameter instead tells ExaBGP to only announce the route after 30 seconds of successful checks (meaning 30 checks). I choose this number to ensure there's no intermittent failures, causing traffic flapping back and forth.

I've also added the --disable parameter. If the file specified exists then ExaBGP will set the --disabled-metric. I touch the file before any maintenance on the server, like upgrades and reboots, and remove it afterwards. This will drain the server of queries and the users will not experience any loss of service.

Other than that there's some logging which will send events to the local syslog daemon which sends if off to a remote syslog server.

This configuration serves me well and it's been running in production for a few years. Maybe someone else will find it useful.

if __name__ == "__main__":
    errors = do_thing("file.yml")
    if errors:
        print("The following errors were found:")
        for error in errors:
            print(f"- {error}")
        os._exit(1)
    else:
        print("File is looking good!")
        os._exit(0)

And the gitlab-ci.yml:

stages:
  - build

validate file:
  stage: build
  image: python:3.11
  script:
    - python3 ./scripts/check.py

If there are any errors they won't get shown before the pipeline fails.

This can be fixed by setting the environment variable PYTHONUNBUFFERED to true or using the -u flag:

stages:
  - build

validate file:
  stage: build
  image: python:3.11
  script:
    - python3 -u ./scripts/check.py

The explanation is that it "forces the stdout and stderr streams to be unbuffered".