Hot coffee, black. Grafana to-go. Protect Grafana without IP whitelisting.

I know I got you wondering how to secure Grafana so that you can check your dashboards on public 4G LTE. And not need to worry about IP-whitelisting. I am assuming by now you are thinking,

Yes Alex! I know that Fail2ban, and a CDN is exactly what I need. I just need that pesky fail2ban filter and jail configuration, and I will be golden.

Here ‘ya go.

Fail2ban Filter and Jail to Protect Grafana Without IP-Whitelisting

# filter.d/grafana-apache.local

[Includes]
before = apache-common.conf

[Init]
datepattern = \[%%d/%%b/%%Y:%%H:%%M:%%S %%z\]

[Definition]
failregex = ^.*:*.* <HOST> .* "POST /login .*" 401\b.*
ignoreregex =


# action.d/grafana-apache.local

[Definition]
# Parameters:
#  - banlist: path to the Apache include file
#  - apache_reload_cmd: how to reload Apache

actionstart = touch <banlist>
actionstop  = /bin/true

# Add "Require not ip <ip>" if not already present, then graceful reload.
actionban = set -euo pipefail; \
    BL="<banlist>"; IP="<ip>"; \
    LINE="Require not ip ${IP}"; \
  if ! /usr/bin/grep -qxF "$LINE" "$BL"; then echo "$LINE" >> "$BL"; fi; \
    <apache_reload_cmd>

# Remove the line on unban, then reload.
actionunban =  set -euo pipefail; \
    BL="<banlist>"; IP="<ip>"; \
  LINE="Require not ip ${IP}"; \
    ESC="$(printf "%%s\n" "$LINE" | /usr/bin/sed -e "s/[.[\\*^$(){}+?|\\\\]/\\\\&/g")"; \
  /usr/bin/sed -i -e "\|^$ESC$|d" "$BL"; \
    <apache_reload_cmd>

[Init]
banlist = /opt/banlist/grafana
apache_reload_cmd = /usr/sbin/apachectl graceful


# jail.local

[grafana]
enabled = true
filter  = grafana

logpath = /var/log/apache2/path_to.log
backend = auto
maxretry  = 4
findtime  = 10m
bantime   = 1h

action    = grafana-apache[name=grafana, banlist=/opt/banlist/grafana]

The filter checks for 401 responses recorded in the Apache access log (assuming you are using Apache HTTP server as a reverse proxy). The action appends the recorded IP into an Apache include file. And the jail configuration makes sure that fail2ban invokes the action on every event detected by the filter.

Let us set this up in Apache

And for good measure, add a custom header key in CloudFront. Every time CloudFront pulls from your origin to generate its cache, it will pass along a secret key. Anybody who tries to circumvent your CloudFront distribution will probably not be able to do the same, so they cannot hit your origin directly.

CloudFront will include the Verify: 12345 header with every origin pull.

Next, add the RewriteCond to your Apache config to enforce the secret key.

# sites-available/grafana.conf

<VirtualHost *:3443>
        ServerName  5.5.5.5 # your public IP
        ServerAlias my.sites.com

        SSLEngine on
        SSLCertificateFile    /etc/ssl/certs/my.crt
        SSLCertificateKeyFile /etc/ssl/private/my.key

        RewriteEngine On
        # Check if the custom header is NOT present or does NOT match the secret value
        RewriteCond %{HTTP:X-Origin-Verify} !^12345$
        # Deny the request if the condition is met
        RewriteRule .* - [F,L]

        ProxyPreserveHost on
        ProxyPass / http://127.0.0.1:3000/
        ProxyPassReverse / http://127.0.0.1:3000/
        <Location "/">
                <RequireAll>
                        Require all granted
                        # Pull dynamic bans
                        IncludeOptional /opt/banlist/grafana
                </RequireAll>
        </Location>
</VirtualHost>

Next, add a CloudFront behavior to cache static content at /public/. You should already know by now there is no reason not to.

I chose to pass just the Host header, and cache everything at /public. For everything else, I disabled caching entirely.

Now, fail2ban protects your Grafana app from bruteforce attempts and we did not bother to protect Grafana with IP whitelisting. With CloudFront caching your static content, visits to the Grafana login page do not drain your origin server.