Firewall Groups

For a long time it has been a wish to utilize the firewall of the router more that it already is. The specific feature I waned to add was block lists. That is, from a dynamically created list, it should be possible to generate a list of IPs or IP ranges which will be blocked by the firewall in either direction. Previously I created static lists of known IPs. While this works rather okay, the problem is that Ansible ¹ along with VyOS config system is really slow when handling loads of data. Adding thousands of lines of configs take minutes to finish. Even worse, having those lists in the VyOS configs results in additional processing that increases the editing time of the firewall config. As previously mentioned, this was not a desired approach and another solution had to be found.

It is worth mentioning that modifying the file system on VyOS is rather difficult, because the vyos.vyos collection overrides the connection. To go around this, one needs change connection, delegate the config to VyOS host, change user connected with, and escalate privileges. Once done, uploading scripts to be executed or other files is fairly easy.

1
2
3
4

  connection: ssh
  remote_user: "{{ hostvars[inventory_hostname].ansible_user }}" # Avoid recursive lookup
  delegate_to: "{{ ansible_host }}"
  become: yes

1. Add the block list as a group to the firewall, and add its rules in the state machine. Make sure to leave the group empty.
2. Upload the block list file with online IPs to /config/user-data folder.
3. Add the script to the task scheduler, preferably once a day, but at least follow the requested guidelines for dynamic lists.
4. Add the script to be executed as a post-boot script (/config/scripts/vyos-postconfig-bootup.script), to make sure the firewall works instantly after reboot.

The following script only works for one list, but it basically takes that list and parses it (this might differ depending on the type of list being used, better conversion might be needed for other lists). It then creates an nftables (only works for VyOS 1.5 and beyond) config file, which flushes the affected list and then creates a complete list of all the previously parsed values. If this was one at a time, it would take considerate amount of time (trust me, I tried it), but using one file is both easier and a lot faster. For IPv4, having 500k IPs/ranges would not take that long time to add, though, it would be counterproductive. Do note that I do not minimize the amount of IPs, nor do I put them together into ranges. The reason for this is because it is rather computation heavy doing so, and nft can handle duplicates with not problem, and VyOS turn that one on automatically for us anyway.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

#!/usr/bin/env python3

import requests
import sys, subprocess

def main(args: list[str]):
    with open('/config/user-data/block-lists', 'r') as f:
        urls = f.read().splitlines()
    ls = []
    session = requests.session()
    for url in urls:
        res = session.get(url)
        ls.extend((line if '/' in line else f"{line}/32") for line in res.text.splitlines() if line and line[0] != '#')

    # Done all in one swell swoop, both for performance and atomicity
    block_file = '/config/user-data/block-list.sh'
    with open(block_file, "w") as f:
        f.write("#!/usr/sbin/nft -f\n")
        f.write("flush set ip vyos_filter N_BLOCK-LIST\n")
        f.write("add element ip vyos_filter N_BLOCK-LIST {\n")
        f.write(",\n".join(ls)+"\n")
        f.write("}\n")
    res = subprocess.run(['nft', '-f', block_file])
    res.check_returncode()
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv))

Executing the script, considering it needs to fetch a bunch of files, and also add them to another file after some parsing, with just 20k IP ranges is executed in under half a second, while still being transactional. And the best part is that it does not touch the VyOS config when updating the group, meaning that it will not pollute the parsing time on VyOS config. I consider making this practice more of a standard and included as a local action instead. That way it is more clean, and can be incorporated in its own locked system. The best part is that we now have support for dynamic lists too, making it possible to keep up to date with Open Dynamic Block Lists or similar service. Though, there is a bit more to be done to make sure that all lists are properly handled.