~~SearXNG Set Up~~
The process and journey of setting up and configuring a SearXNG instance
The other day I was trying to search Google for a funny image I remembered seeing a while back.
Figure 1: The image in
question, I found it in my DOWNLOADS folder
Unfortunately, I only had the text content of the image to go off. As I iteratively modified my search query to try and hone on what exactly I was searching for, I found myself becoming increasingly exasperated at the in-your-face and exceedingly unhelpful “AI” result that would “think” its way to the top of my results every time.
To think, every time Google missed the mark, I was likely burning enough energy to power Palau for a year.
The reasonable and sensible action (and the action I’d recommend) is using an alternative meta search engine such as startpage; however, I wanted to have a little more “fun” with finally de-Googling this part of my life. Which meant going through the process of setting up a SearXNG instance on my server.
Note: a “meta search engine”, for the unawares, is simply a search engine that itself aggregates the results of other search engines together.
To be fair to myself, I'm not a complete masochist, there are some practical benefits to setting up a private SearXNG instance. Specifically, being able to personalize engine weights and result hostname priorities (which I’ll detail later in this article).
Regardless, this was a project I’d had on the backburner for a while; however, I’ve always found the SearXNG documentation to be somewhat “difficult” to parse. Perhaps “anti-helpful” would be a good word.
My specific use case/requirements were as follows:
1) Run SearXNG in a Docker container to provide easy set up and a minor degree of isolation from the rest of my server.
a. Since this instance would be facing the public web, it should have some form of rate limiting.
2) Have it play nice with apache2, and most importantly my LetsEncrypt SSL certificates for HTTPS.
Anyways I figured I’d write some notes in case I needed to do this again, or if someone else in the same boat stumbles upon this.
Assuming Docker is already installed, and the installing user is added to the docker group.
1) Install the container as described on the SearXNG guide
1. # Create the environment and configuration directories
2. mkdir -p ./searxng/core-config/
3. cd ./searxng/
4.
5. # Fetch the latest compose template
6. curl -fsSLO \
7. https://raw.githubusercontent.com/searxng/searxng/master/container/docker-compose.yml
\
8. https://raw.githubusercontent.com/searxng/searxng/master/container/.env.example
9.
1) Rename .env.example to .env and set SEARXNG_HOSTNAME (to site URL) and LETSENCRYPT_EMAIL (to your lets encrypt email)
2) Run docker composer up -d and docker compose down to initialize the containers.
3) Create settings.yml in the core-config directory
The older SearXNG Docker container github guide puts the installation folder in /usr/local/. Even though the new guide doesn’t mention this, I would recommend doing it as well, as I found when I tried installing to my home directory that Docker couldn’t access the files in core-config.
For core-config/settings.yml start with the following:
1. # see
https://docs.searxng.org/admin/settings/settings.html#settings-use-default-settings
2. use_default_settings: true
3. server:
4. # base_url is defined in the SEARXNG_BASE_URL environment
variable, see .env and docker-compose.yml
5. secret_key: "ultrasecretkey"
# change this!
6. limiter: true # enable this
when running the instance for a public usage on the internet
7. image_proxy: true
8. valkey:
9. url: valkey://valkey:6379/0
10.
The reference to valkey is required for connecting to the valkey container and enabling rate limiting.
Use core-config/limiter.toml to configure the rate limiter, with "limiter" set to "true" in core-config/settings.yml:
1. [botdetection.ip_limit]
2. link_token = true
To generate a new secret key, run the following command:
1. sed -i
"s|ultrasecretkey|$(openssl rand -hex 32)|g" core-config/settings.yml #Linux
2. sed -i
'' "s|ultrasecretkey|$(openssl rand -hex 32)|g" core-config/settings.yml #macOS/BSD
With this you should be able to run “docker compose up -d” and access the container from https://127.0.0.1:8080
To make the container available to the outside world, I set up a reverse proxy to the container by adding something like the following lines to the apache2 sites-available configs (since puyoman.net supports both http and https I added these lines to both the standard and the “le-ssl” configs). Note: The Location block should go inside the VirtualHost block.
1. <Location /searxng>
2. Require all granted
3. Order deny,allow
4. Deny from all
5. # Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1
6. Allow from all
7.
8. # add the trailing slash
9. RedirectMatch 308 /searxng$
/searxng/
10.
11.
ProxyPreserveHost
On
12.
#ProxyPass unix:/usr/local/searxng/run/socket|uwsgi://uwsgi-uds-searxng
13.
ProxyPass
http://127.0.0.1:8080
14.
ProxyPassReverse
http://127.0.0.1:8080
15.
16.
# see flaskfix.py
17.
RequestHeader
set X-Forwarded-Proto %{REQUEST_SCHEME}s
18.
RequestHeader
set X-Script-Name /searxng
19.
20.
# see
botdetection/trusted_proxies.py
21.
RequestHeader
set X-Real-IP %{REMOTE_ADDR}s
22.
RequestHeader
append X-Forwarded-For %{REMOTE_ADDR}s
23.
24.
</Location>
25.
With that your SearXNG instance should be available at <url>/searxng
(To do: I also want to make this accessible at search.<url>, but haven’t figured out how to do that yet).
Now that that’s completed, we can start making changes to settings.yml to customize the instance. Whenever you make changes, you’ll need to restart the session by running “docker compose down” and “docker compose up -d” in /usr/local/searxng/.
To modify search engine availability and weight add the “engines” tag to settings.yml, here are a couple examples from my current file:
1. engines:
2. -name: bing
3. disabled: true
4. -name: wikipedia
5. weight: 1.5
I think Bing might be disabled by default, and I enabled it manually at some point, but I noticed it was giving absolutely garbage unrelated results in Russian and Chinese so I re-disabled it.
Then I simply wanted to give Wikipedia a higher priority in my results. Speaking of…
With the “hostnames” tag you can assign priorities to specific websites in your results. For example, I wanted to lower the priority of fandom wikis and up the priority of any alternatives I knew of:
1. hostnames:
2. low_priority:
3. - '
(.*\.)?fandom.com$'
4. high_priority:
5. - '(.*\.)?touhouwiki.net$'
Finally, management. I found after about a week of usage my instance was no longer returning Google results, and the solution was to update the container. In order to automate the process, I created a script and set up a cron job for it to execute weekly:
The script:
1. $ cat /usr/local/bin/update-searxng
2. #!/bin/bash
3.
4. echo Updating searxng docker: $(date)
5.
6. cd /usr/local/searxng/
7.
8. docker compose
down
9. docker compose
pull
10.
docker compose up -d
Edit your crontab:
1. sudo crontab -u {user} -e
And add a line like the following:
1. 0 2 * * 1 /usr/local/bin/update-searxng
(which will run the update script at 2am every Monday).
Remember that the output of cron jobs is sent to your user’s mailbox.
Lastly, I wouldn’t consider this essential, but a good idea to keep this in memory:
https://docs.searxng.org/admin/answer-captcha.html
Occasionally, search engines will complain and want you to answer a CAPTCHA to continue using them. When using a headless server, this means setting up an ssh tunnel to the server with your local machine and accessing the website through a browser using that tunnel. With my enabled set of engines, I found this to most often be the case with duckduckgo and startpage. I tried to keep up for a time, but I eventually found it was too much of a pain. If it happens, it happens. As they say.
Anyways, this was an interesting project. And with recent news that came out as I was slowly writing the blog entry, a pressing one as well. If anyone who isn’t me uses this as reference I hope it helps. The other day I had someone comment on one of my Youtube videos asking a question regarding my post about booting PowerPC MacOS off a USB flash drive, noting that they had no other way to contact me. Which got me thinking, I have been wanting to self-host an email server…
Maybe next time.