Building a Graylog server to run on an Amazon Lightsail instance

This post reviews how to build a Graylog server on Amazon Lightsail, the easiest way to get started on AWS. Graylog is an open source log management system that allows textual logging data created by network devices, applications, and servers to be centrally stored, searched, and reported on.

This blog is relevant to those working from home with various pieces of network equipment and a need to centralize log data for these devices. My personal networking equipment includes a pfSense gateway managing a couple of broadband lines, routers, and Wi-Fi access points. With Graylog, you can centralize the log data collection for these devices and automate looking for issues raised by them in their log messages.

In this post, I walk you through how I built a Graylog server on a Lightsail instance running Ubuntu 18.04 LTS with the pre-requite packages, mainly Elasticsearch, and MongoDB. This server receives log messages from my pfSense server, routers and access points. Also, taking into account that the devices being used are inside a private network NATing out to the internet but that must be uniquely identified in Graylog.

Network design

The following diagram shows where the various parts of the network fit and provides details of the TCP and UDP ports involved at different points in the network. You can see, the internal Wi-Fi AP and router behind the pfSense server with its own firewall, outbound NAT (Network Address Translation) and outbound load balancing (over two broadband lines, not shown). Traffic flowing over the internet to the Lightsail edge firewall and on into the Lightsail instance running Graylog and the Elasticsearch and MongoDB services.

The following image is a simple diagram of the network.

Network access to the Ubuntu instance is restricted by the Lightsail firewall which allows TCP/UDP ports (and PING) to be allowed or blocked. Ports TCP:22 (SSH) and UDP (syslog from pfSense), UDP:51401 (syslog from the Wi-Fi AP) and UDP:51402 (Syslog from the router). These separate UDP ports are used so that Graylog can have a listener on each of the separate ports and can tag a source on them for the individual devices. This is needed as the Source IP is one of two IPs of the two broadband lines that pfSense routes traffic through (outbound load balancing). The pfSense and other devices are configured to use the Public IP of the Ubuntu Lightsail instance as their remote Syslog server with the relevant destination UDP Port. Recent changes to the Lightsail firewall now allow for the source IP address of inbound traffic to be used to limit where the Syslog data comes from. This is useful to prevent when whole internet trying to send Syslog data to the Graylog server.

Lightsail instance setup

Now that you have an idea of the network architecture, I can walk through how to set up Graylog on Amazon Lightsail.

The following section details the setup and configuration of the Lightsail instance to be used to run Graylog under the Ubuntu operating system (OS). This gets the instance ready to connect to and to start the process of installing Graylog.

The Lightsail Ubuntu 18.04 LTS instance is a 4-GB RAM instance, based on the minimum server specification given in the Graylog installation guide.

1. From the Lightsail console, click Create instance.
2. From Select a platform, choose Linux/Unix.
3. From Select a blueprint, choose OS Only and then Ubuntu 18.04 LTS.

4. From Choose your instance plan, choose the $20 bundle, with 4 GB, 2 vCPUs and 80 GB SSD.
5. In Identify your instance, enter a unique name for your instance.

6. Then click Create instance.

You are then taken back to the main Lightsail home page with your new instance showing grayed out and in a state of “Pending” until it has been created. Once it is running, the state changes to “Running.”

7. Click on the three dots at the top right of the new instance’s panel and select Manage.
8. Then select Networking.
9. Click Attach static IP in the “IP addresses” box.

10. If you already have a static IP available, select it from the dropdown list and click the green tick icon to the right of the “Select static IP” dropdown list.
11. If not, click Create static IP, select your new instance, give the IP a unique name, and click Create.
12. Under the firewall remove (click) the TCP:80 rule.
    As a best practice you should restrict any incoming traffic to your Graylog server to the IP addresses to the specific IP address (or addresses) that will need to access your instance.  
    
13. Click the SSH (TCP:22) rule and click the edit icon, then check the Restrict to IP address box,  enter the IP address of the system you will use to SSH into the instance in the Source IP address box, and click Save.
14. Click on Add rule, set Application as Custom, Protocol as TCP and Range as 9000 (this is later used for web access to Graylog), specify the IP you will use to access the system as you did in the previous step, and click Create.
15. Click on Add rule, use Application as Custom, Protocol as UDP and Range as 51400-51402 (one port of each of the devices sending syslog data), specify the IP you will use to access the system as you did in the previous step, and click Create.
    

The static IP address used preceding should  be assigned to a DNS name (“A” record) on your domain’s DNS server. The exact mechanism for doing depends on where and how your DNS is hosted. This forms the Fully Qualified Domain Name (FQDN) used to connect to the Lightsail instance. But, you can also use the public IP address  toconnect via SSH, the Graylog web interface and for device to send logging data.

Access the Lightsail instance to configure and install the software.

Having set up the Lightsail instance, the next step is to connect to the Ubuntu operating system to be able to run commands to configure Ubuntu and install Graylog. The remote command-line connection utility “SSH” is used. This secure (encrypted) connection method requires the security to be set up before use.

The Lightsail browser-based SSH client can also be used to connect and enter the command to install and configure the system without the need to manage the SSH authentication key file. However, I prefer to use a standalone SSH client for two main reasons. Firstly, I have a number of servers in different hosting environments and I prefer to use the same method to connect to them all. Secondly, I automate the installation and configuration using ansible, which connects via SSH and needs access to the authentication key file.

An SSH connection is used to enter commands into the Lightsail instance. Lightsail protects SSH connections using an authentication key (pem). The preceding procedure assumes you are using the default pem for SSH connections to the new Lightsail instance. The pem must be downloaded and saved for SSH use.

1. From the Lightsail console, click Account, and select Account from the menu.

2. Click SSH keys and Download to the right of the “Default” key.

3. Download () the pem file as “aws.pem” for later use by SSH.
4. On UNIX systems from the command line chmod 0600 aws.pem.

Test the SSH connection to the Lightsail instance. Use the directory where you saved the “aws.pem” file to, use the command “SSH -l ubuntu -i aws.pem <FQDN>” where “<FQDN>” is the Full Qualified Domain Name of the Lightsail instance. Your SSH client may ask for the initial connection to be confirmed or may reject it if the name or IP of the Lightsail instance already exists in the local SSH “.ssh/known_hosts” file, if so, edit the file and delete the record.

Configuring Ubuntu from the Command Line (SSH)

Now that you created the Lightsail instance, you are ready to connect to your instance using your SSH client of choice. After you connect, there is a small amount of Ubuntu operating system configuration required to make certain the software that is pre-installed on the Lightsail instance is up to date, to set the hostname/timezone and create a swap file (which allows more memory to be used than actually exists by swapping out unused parts until needed again).

Update the operating system to the latest level and reboot:

apt –y update

apt –y upgrade

reboot

Set the hostname (e.g. mygraylog):

hostname mygraylog

Edit “/etc/hosts” and add the new host name to the “127.0.0.1” record

127.0.0.1 localhost mygraylog

Set your local timezone (mine is “Europe/London”):

timedatectl set-timezone Europe/London

Create a swap file, activate, and make available at boot time:

dd if=/dev/zero of=/swap count=8192 bs=1MiB

chmod 600 /swap

mkswap /swap

swapon /swap

Edit “/etc/fstab” add the following at the end of the file

/swap swap swap 0 0

Install Graylog and pre-requisites from the Command Line (SSH)

Finally, Graylog itself (and pre-requisite software packages that Graylog uses) can be installed.

Generate secrets to be used by Graylog:

This is required to create an encrypted version of the Graylog login password.

apt –y install pwgen

Save the string create by the next command to be used as <secret> later

pwgen -N 1 -s 96

Save the string create by the next command to be used as <password-sha2> later

<yourpassword> will be the password for the user “admin” for the Graylog web interface

echo –n “<yourpassword>” | sha256sum

The quotes around <yourpassword> are needed.

Install pre-requisite software packages:

These packages are required for the Graylog server to operate.

apt –y install apt-transport-https openjdk-8-jre-headless

apt –y install uuid-runtime curl dirmngr

Set up install for Elasticsearch:

Elasticsearch is used by Graylog to store all the received messages and for searching the stored messages in a flexible way. First, the location to install Elasticsearch from must be configured.

(the following is a single-line command)

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

(the following is a single-line command)

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

apt –y update

Install Elasticsearch, enable it to start at boot and start it:

apt –y install elasticsearch

Edit “/etc/elasticsearch/elasticsearch.yml” and change cluster.name: my-application to cluster.name: graylog

systemctl enable elasticsearch

systemctl start elasticsearch

Set up install for MongoDB:

MongoDB is used by Graylog to store its configuration. First, the location to install MongoDB from must be configured.

(the following is a single-line command)

wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc | sudo apt-key add -

(the following is a single-line command)

echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list

apt –y update

Install MongoDB, enable it to start at boot and start it:

apt –y install mongodb-org

systemctl enable mongod

systemctl start mongod

Set up install for Graylog:

(the following is a single-line command)

wget https://packages.graylog2.org/repo/packages/graylog-3.2-repository_latest.deb

(the following is a single-line command)

dpkg -i graylog-3.2-repository_latest.deb

apt –y update

Install Graylog:

apt –y install graylog-server

Update the Graylog configuration:

Before starting the Graylog server, a few file updates are required for the network and security environment in which it runs.

Edit “/etc/graylog/server/server.conf” and make the following changes

• Change “password_secret =” to “password_secret = <password-sha2>” (see preceding)
• Change “elasticsearch_shards = 4” to “elasticsearch_shards = 1”
• Change “http_bind_address = 127.0.0.1:9000” to “http_bind_address = 0.0.0.0:9000”
• Change “http_publish_uri = …” to “http_publish_uri = http://<FQDN>:9000” (see preceding)
• Uncomment “#root_email = ….” and enter your email address
• Uncomment “#root_timezone = ….” And change to “root_timezone = UTC”

Edit “/etc/default/graylog-server” and the make the following change.

• Add “-Djava.net.preferIPv4Stack=true” at the start of the “GRAYLOG_SERVER_JAVA_OPTS”

Enable Graylog to start at boot and start it:

systemctl enable graylog-server

systemctl start graylog-server

Connect and log in to Graylog

The Graylog server is now ready to be connected to via its Web interface so that final configuration to be completed.

Assuming all the preceding ran without error, you can now log in to Graylog via;

http://<FQDN>:9000

<FQDN> is the Fully Qualified Domain Name of your Lightsail instance. Logon as the user “admin” with the password that you used to generate the <password_sha2> preceding.

Graylog basic configuration.

Assuming that the devices that send their syslog records to Graylog have been configured to forward to <FQDN>:51400 (51401 and 51402), Graylog listeners must be set up to receive the syslog records. Repeat the following for each of the ports;

• From the top menu bar, go to System then Inputs.
• From the Select input dropdown list, select Syslog UDP.
• Click Launch new input.

• On the Launch new input pop-up, tick Global, fill in the Title, Port, Override source (the source name that shows on messages received via this Listener) and click Save.

Having completed the creation and configuration of a Lightsail instance, configuring Ubuntu, installing the Graylog server and additional services, with a small amount of Graylog configuration, you start to see messages from the devices appearing in Graylog. Additional devices can be added and the numerous other features of Graylog can be tried out.

Graylog provides an excellent way of bringing all the logging data from various devices into one central management server, allowing you to see the effects of issues within a network in a single time line, making problem determination a much simpler process.

Author

Richard Gate, CommuniG8 Ltd

Email: richard.gate@communig8.com

Twitter: @communig8