Blocking AI

Usually I am quite open to new ideas and techniques, but there are those rare occasions where I need to take a standpoint due to my ethical perspective. In this case, it is the use of AI. I have never used it, at least not from how I see it. I am not going to go into why I do this, but rather keep to the subject at hand: How to make sure that bots, crawlers and AI are blocked from your websites.

It started as a joke at first, someone mentioning how to confuse AI with an HTTP header. I did in fact add that header to my proxy. Then it got me thinking if there was a way to block, or at least reduce, the amount of bots that visit my sites. Well, sometimes you want them, so people can search for your site. But there are some that use this scraping for their own good. We cannot have that, can we? Therefore I first looked up robots.txt, which all bots should read and follow accordingly, but at most it is used for proper etiquette. I found a site mentioning how to do this ¹, but the list was quite minimal, so I found a repository with a thousands of them ². However, as it is not guaranteed that the bots will follow this, I continued to block User-agent as well inside the reverse proxy. After reading more about how Nginx did this, I found a site giving me the initial thought for my integration ³.

Next was to combine the findings into the infrastructure I have built up with Ansible, starting with parsing an existing robots.txt file which is updated daily ²⁴.

1
2
3
4
5
6
7
8

{{ ansible_managed | comment(decoration="# ") }}
{% set agents = lookup('url', 'https://files.ichi.do/recommended-robots-block-ai-bots.conf', split_lines=false) | robots2list %}
map $http_user_agent $aiagent {
	default 0;
	{% for agent in agents %}
	~*{{ agent | replace(' ', '\\s') }} 1;
	{% endfor %}
}

As the existing files with the blocking of AI generates these lists on the fly, and I would rather not copy their code, I instead download their generated robots.txt file and parse it. Due to this parsing, a filter was created to parse robots.txt and return a list of bots.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

class FilterModule:
	def filters(self):
		return {
			'robots2list': self.robots2list,
		}
	def robots2list(self, config: str, prefix: str="User-agent: "):
		agents: list[str] = []
		for line in config.splitlines():
			if not line or line[0] == "#":
				continue
			if line.startswith(prefix):
				agents.append(line.removeprefix(prefix))
		return agents

Then add $aiagent as a check to block the bots from access, to all the running hosts.

1
2
3

    if ($aiagent) {
        return 403;
    }

Finally create two Ansible tasks which adds this to the reverse proxy.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

- name: Push agents
  diff: no # Too large file
  template:
    src: aiagents.j2
    dest: /etc/nginx/aiagents.map
    lstrip_blocks: yes
    trim_blocks: yes
    mode: '0644'
    owner: root
    group: root
  notify: restart nginx

- name: Include agents
  lineinfile:
    path: /etc/nginx/nginx.conf
    insertafter: gzip_vary on;
    line: include /etc/nginx/aiagents.map;
    state: present
  notify: restart nginx

I then tried this with curl to see if it worked curl -A 008 https://blog.aposoc.net/, which returned 403, as expected. Next is to make this feature optional for some sites which does not need it. Is this the best way to solve this? Not really, as this could also be ignored, so I dived deeper, found a couple of lists with IP-ranges and added them to an AIBots group in my firewall ¹.

Was this the nail in the coffin for the bots? Not at all. The only real way to deal with them is to block scrapers at all, which is close to impossible if done right. I do not think web scraping is bad, it is more what you use it for that makes me act, which I now have a wall to fend against companies feeding their AI. Web scraping in itself is another subject, and I might write about that in a later post.