I've got a second server!

Hi, I'm Steffo and - as you might already know - I have a root server hosted by netcup. Now, I've been thinking about getting another server for a long time. Not because I need it - my root server still has enough CPU, RAM, and storage to host more stuff - but more because I want to do things right. Let me explain...

I have had my root server for more than three years now. Initially, I was quite a noob with Linux. I was primarily able to do stuff thanks to my friend @Kitsune_Yasu, who helped me set up the initial stuff (ssh, nginx, ...) and then I learned, service by service, how to do Linux stuff. This means, of course, that my server still has a lot of old and weird things that probably shouldn't be there anymore. That's why I wanted to get a new server - to do "everything" correct now. (Of course, doing everything correct is impossible, but I want to try at least...)

But there's another reason why I wanted another server... I'm hosting a few services for some friends, like a few Minecraft servers and a Matrix instance. If I have to update my server - and thus, take these services down for a few minutes - I want them to have something where they can look whether the downtime is scheduled or not. Luckily, I can simply self-host Uptime Kuma, create a few status pages and everything's good, right? Well, yes. Until you have to restart the whole root server - because when you're restarting the root server, people can't look at the status page, that is hosted on the root server.

So the decision was made: I'm getting another server - specifically for monitoring. Since I thought that a full-blown root server would be too much, I decided on a smaller vServer. Luckily, netcup had a few deals while I was thinking of getting a server, so I took the opportunity and got a nice and small vServer that will be way more than enough for me to use. Now I had to set it up... eh, I'm gonna do that later...

Setting up the vServer

After an embarrassing amount of time, I finally decided to set the new server up. I asked @Kitsune_Yasu if he wanted to help me set it up, and he said yes!

After doing the initial setup for a VLAN between my two servers, I gave netcup my SSH public key and installed Ubuntu using netcup's official images, which also installed some important settings like the above-mentioned SSH public key or creating my main user.

Once Ubuntu was installed, I SSH'd into my new server and saw: IT WORKS! Now onto setting the correct SSH settings (like disabling unsafe options, changing the port, disallowing password auth, ... stuff like that).

One of the last things we still had to set up was the VLAN. For this, we used Network Manager (and nmtui) to set up the correct network settings, like which ethX is the public connection and which is the VLAN, and so on. Also, we've set the IP for the vServer to use for the VLAN, which is just a private IPv4 address. (192.168.xxx.xxx)

To finalize the VLAN setup, I added my IPs to the hosts file, so I simply could use server-1234.lan instead of the IP addresses to talk to the other server.

After this was done, I was able to install docker and nginx and move all the monitoring stuff I had on my root server over to my vServer.

Configuration files can be hidden well at times

While moving my Prometheus settings, I noticed that some metrics on my root server were publicly accessible! Of course, that's not how it should be, so I searched for how to change it. Note, even after more than 3 years using Linux as a server operating system, I still don't know everything about it. (Who would have thought?)
I'm quite familiar with using nginx and docker, but when there's something running fully on the system itself, then it's still somewhat new for me.

So, when I wanted to configure the Node Exporter by Prometheus, I looked for a folder like /etc/prometheus/..., but I didn't find one. There also was no file like /etc/prometheus-node-exporter.conf, so I looked further. After checking the --help command of the Prometheus Node Exporter, I saw starting arguments. Aha! I guess I'll have to edit the systemctl service file. Sigh...

Looking into /etc/systemd/system/... showed me... no Prometheus Node Exporter service file?! But... how was that possible? I mean, I know that I started the exporter using systemctl, so it HAD to be there, right?

Well... after a lot of searching I found: Nope, there are more folders. The file I was searching for was inside /lib/systemd/system/, which... of course everyone knows it's inside there, eh? It doesn't matter now... I found the file I was searching for, and I finally could make the exporter not run on 0.0.0.0, whee!

But then I looked at that file a bit... longer...

EnvironmentFile=/etc/default/prometheus-node-exporter
ExecStart=/usr/bin/prometheus-node-exporter $ARGS

There's a config file!! But it's in /etc/default... I would've never found that without looking inside the service file, which I also didn't find at first. I love it when there are so many different standards... reminds me a little bit of this xkcd:

Anyway, after finding the environment file, I updated it to have the correct listening address:

ARGS="--web.listen-address=127.0.0.1:9100"

Firewalls can be useful sometimes

After I published my thoughts about /etc/defaults on Mastodon, a friend of mine told me that I might want to install a firewall. At first, I didn't like that idea. Why installing something that might slow me down when hosting new stuff? Also, I already gave my services the correct address to use (127.0.0.1), so I really didn't see the reason behind a firewall.

This was while I hadn't yet fully transferred all metrics to the new server. While I actually had to change all the metrics hosts (because I needed to change the IP from 127.0.0.1 to the VLAN address), I saw how many metric hosts were still using 0.0.0.0. Seeing this, I thought that a firewall might actually be useful.

So I asked my friend @Kitsune_Yasu, if he wanted to help me set one up. Since I'm using docker and iptables can be a bit complicated and overwhelming for beginners, he recommended the more beginner-friendly firewallD.

After a few hours, firewalld was properly set up and now everything should work.

Simple explained: How does firewalld work?

Firewalld uses "zones" to allow / disallow certain things for certain interfaces. It comes with many pre-defined zones - like public, DMZ (demilitarized zone) and internal - that have some good preset behaviors.

In my case, I'm allowing the public internet to use the ports 80 and 443 (so HTTP and HTTPS) and the SSH, so I can still connect to my server. I could (but currently don't need to) allow my VLAN connection to use some services, without having to make them public to the world.

And instead of having to manually enter the port numbers by yourself, you can define and use "services", of which there are already a lot by default. Using services instead of manually allowing port numbers mainly has the advantage of seeing which services are allowed instead of having to know each port number.

I definitely still don't know everything about it, but it seems to work, and I'm happy that it does.

Interesting things I noticed

While setting up firewalld, I noticed a few weird things when I looked at the list of open ports. There were two open ports that I didn't know and that were created by a user called node. This was really suspicious, but eventually, we found out that these were from two docker containers that were running in network-mode host. (Yes, sadly I have to do that...)

But thanks to the firewall, this shouldn't be an issue for now.

Adding IPv6 because I can

While setting up the firewall, we noticed that my servers didn't use IPv6 yet. I didn't know that I got an IPv6 address, but apparently I've got not just one, but a whole subnet! (That's apparently normal? Huh!)

Anyway... after noticing that all my servers already had IPv6 addresses set up, I added their IPv6 addresses to my DNS settings and... now you can read this blog through IPv6, I guess.

I think there might be some services that don't yet fully work with IPv6, but I haven't tested everything yet. I'm just a dragon, I can't do everything everywhere all at once...