How to [un]block your CDN and detect your real visitors

CloudFront IPs

When putting a webserver behind eg an Application Load Balancer or a CDN such as CloudFront, the server will start to perceive all requests as coming from the Load Balancer or CloudFront itself.  If not correctly configured, the server security software (eg mod_security, csf) will start to block the addresses of CloudFront itself.  Maybe not immediately – if AWS WAF rules are on the volume of attacks reaching the webserver itself may be relatively low – always leave CSF alerts on so you can spot this happening.

AWS Application Load Balancer adds the origin ip address to the HTTP X-Forwarded-For header, and when using CloudFront together with a Load Balancer the HTTP X-Forwarded-For will show multiple addresses:

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request-custom-headers-behavior 

Elastic Load Balancing.. append the IP address of the CloudFront edge server that forwarded the request onto the end of the X-Forwarded-For header. For example, if CloudFront includes X-Forwarded-For: 192.0.2.2 in a request that it forwards to ELB and if the IP address of the CloudFront edge server is 192.0.2.199, the request that your EC2 instance receives contains the following header:

X-Forwarded-For: 192.0.2.2,192.0.2.199

To associate traffic correctly with the origin IP addresses the webserver needs to know which are the trusted IPs of our CDN, Load Balancer or other proxies, and how to find the real client IP address.

Amazon IP CIDR address ranges are found here: https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html

On OSX pull the list of addresses for CloudFront like this:

brew install jq
curl 'https://ip-ranges.amazonaws.com/ip-ranges.json' | jq -r '.prefixes[] | select(.service=="CLOUDFRONT") | .ip_prefix'

Nginx

To enable in Nginx use the realip module, not included in a default Nginx install but included in distributions such as Engintron. http://nginx.org/en/docs/http/ngx_http_realip_module.html

The realip module will look for last non-trusted ip address in the header: this might seem counter-intertuitive because the original ip address should be the first in the list, each proxy is appending to the list in turn.  However this technique is a defence against spoofing: the last non-trusted ip address is the genuine address of the machine which connected to our trusted proxy: any http header existing before that could be fake/spoofed.

Engintron’s nginx.conf includes set_real_ip_from settings for CloudFlare IPs but not AWS CloudFront, so add them in, the relevant config section will look like this:

 # CloudFront IPs
# List from: https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html
set_real_ip_from 120.52.22.96/27;
set_real_ip_from 205.251.249.0/24;
set_real_ip_from 180.163.57.128/26;
set_real_ip_from 204.246.168.0/22;
set_real_ip_from 205.251.252.0/23;
set_real_ip_from 54.192.0.0/16;
set_real_ip_from 204.246.173.0/24;
set_real_ip_from 120.253.240.192/26;
set_real_ip_from 116.129.226.128/26;
set_real_ip_from 99.86.0.0/16;
set_real_ip_from 205.251.200.0/21;
set_real_ip_from 223.71.71.128/25;
set_real_ip_from 13.32.0.0/15;
set_real_ip_from 120.253.245.128/26;
set_real_ip_from 13.224.0.0/14;
set_real_ip_from 70.132.0.0/18;
set_real_ip_from 210.51.40.0/24;
set_real_ip_from 13.249.0.0/16;
set_real_ip_from 205.251.208.0/20;
set_real_ip_from 58.254.138.0/25;
set_real_ip_from 116.129.226.0/25;
set_real_ip_from 52.222.128.0/17;
set_real_ip_from 64.252.128.0/18;
set_real_ip_from 205.251.254.0/24;
set_real_ip_from 71.152.0.0/17;
set_real_ip_from 216.137.32.0/19;
set_real_ip_from 204.246.172.0/24;
set_real_ip_from 120.52.39.128/27;
set_real_ip_from 118.193.97.64/26;
set_real_ip_from 223.71.71.96/27;
set_real_ip_from 130.176.0.0/16;
set_real_ip_from 54.240.128.0/18;
set_real_ip_from 205.251.250.0/23;
set_real_ip_from 180.163.57.0/25;
set_real_ip_from 52.46.0.0/18;
set_real_ip_from 223.71.11.0/27;
set_real_ip_from 52.82.128.0/19;
set_real_ip_from 54.239.128.0/18;
set_real_ip_from 36.103.232.128/26;
set_real_ip_from 52.84.0.0/15;
set_real_ip_from 111.51.66.0/24;
set_real_ip_from 143.204.0.0/16;
set_real_ip_from 144.220.0.0/16;
set_real_ip_from 120.52.153.192/26;
set_real_ip_from 119.147.182.0/25;
set_real_ip_from 120.232.236.0/25;
set_real_ip_from 54.182.0.0/16;
set_real_ip_from 58.254.138.128/26;
set_real_ip_from 120.253.245.192/27;
set_real_ip_from 54.239.192.0/19;
set_real_ip_from 120.52.12.64/26;
set_real_ip_from 99.84.0.0/16;
set_real_ip_from 54.230.0.0/16;
set_real_ip_from 52.124.128.0/17;
set_real_ip_from 204.246.164.0/22;
set_real_ip_from 13.35.0.0/16;
set_real_ip_from 204.246.174.0/23;
set_real_ip_from 36.103.232.0/25;
set_real_ip_from 119.147.182.128/26;
set_real_ip_from 118.193.97.128/25;
set_real_ip_from 120.232.236.128/26;
set_real_ip_from 204.246.176.0/20;
set_real_ip_from 120.253.241.160/27;
set_real_ip_from 13.124.199.0/24;
set_real_ip_from 35.167.191.128/26;
set_real_ip_from 18.200.212.0/23;
set_real_ip_from 99.79.169.0/24;
set_real_ip_from 52.15.127.128/26;
set_real_ip_from 34.223.12.224/27;
set_real_ip_from 54.233.255.128/26;
set_real_ip_from 13.54.63.128/26;
set_real_ip_from 13.59.250.0/26;
set_real_ip_from 3.234.232.224/27;
set_real_ip_from 52.66.194.128/26;
set_real_ip_from 13.228.69.0/24;
set_real_ip_from 64.252.64.0/18;
set_real_ip_from 18.216.170.128/25;
set_real_ip_from 3.231.2.0/25;
set_real_ip_from 52.220.191.0/26;
set_real_ip_from 34.232.163.208/29;
set_real_ip_from 35.162.63.192/26;
set_real_ip_from 34.223.80.192/26;
set_real_ip_from 34.226.14.0/24;
set_real_ip_from 13.113.203.0/24;
set_real_ip_from 34.195.252.0/24;
set_real_ip_from 52.52.191.128/26;
set_real_ip_from 52.56.127.0/25;
set_real_ip_from 34.216.51.0/25;
set_real_ip_from 52.199.127.192/26;
set_real_ip_from 52.212.248.0/26;
set_real_ip_from 13.210.67.128/26;
set_real_ip_from 35.158.136.0/24;
set_real_ip_from 52.57.254.0/24;
set_real_ip_from 52.78.247.128/26;
set_real_ip_from 52.47.139.0/24;

 # Replace with correct visitor IP
real_ip_header X-Forwarded-For;
real_ip_recursive on;

Don’t forget to also add the local VPC subnet!  Traffic from an Application Load Balancer will show up as coming from a local subnet address.

before the change phpinfo accessed via CloudFront was showing:

$_SERVER['HTTP_X_FORWARDED_FOR'] VisitorIP, CloudFrontIP
$_SERVER['REMOTE_ADDR'] LoadBalancerIP
$_SERVER['HTTP_X_REAL_IP'] LoadBalancerIP

after the change phpinfo shows:

$_SERVER['HTTP_X_FORWARDED_FOR'] VisitorIP, CloudFrontIP
$_SERVER['REMOTE_ADDR'] VisitorIP
$_SERVER['HTTP_X_REAL_IP'] VisitorIP

Apache

in broad terms:

  1. Enable mod_remoteip: https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html
    a2enmod remoteip or Cpanel EasyApache select modules
  2. Add RemoteIP configuration  RemoteIPProxyProtocol On
    1. Add the CloudFront CIDR ranges to the RemoteIPTrustedProxy directive.
    2. Add the local subnet to the RemoteIPInternalProxy directive.
  3. change the Apache log format original header field position (%h) with the field %a

On a Cpanel server the httpd.conf is generated by Cpanel in /usr/local/apache/conf so any customisation needs to be made to the appropriate include files which Cpanel will include each time the main httpd.conf needs to be updated.

Firstly, in WHM-Apache Configuration-Include Editor-Pre Main Include add the directives and the trusted CIDR ranges:

<IfModule !mod_remoteip.c>
LoadModule remoteip_module modules/mod_remoteip.so
</IfModule>

<IfModule mod_remoteip.c>
# If CloudFlare use CloudFlare special CF Header
# RemoteIPHeader CF-Connecting-IP
RemoteIPHeader X-Forwarded-For

# example VPC Subnet address range for Load Balancer
RemoteIPInternalProxy 172.31.0.0/20

# CloudFront CIDR IP Address Ranges
RemoteIPTrustedProxy 120.52.22.96/27
RemoteIPTrustedProxy 205.251.249.0/24
RemoteIPTrustedProxy 180.163.57.128/26
RemoteIPTrustedProxy 204.246.168.0/22
RemoteIPTrustedProxy 205.251.252.0/23
RemoteIPTrustedProxy 54.192.0.0/16
RemoteIPTrustedProxy 204.246.173.0/24
RemoteIPTrustedProxy 120.253.240.192/26
RemoteIPTrustedProxy 116.129.226.128/26
RemoteIPTrustedProxy 99.86.0.0/16
RemoteIPTrustedProxy 205.251.200.0/21
RemoteIPTrustedProxy 223.71.71.128/25
RemoteIPTrustedProxy 13.32.0.0/15
RemoteIPTrustedProxy 120.253.245.128/26
RemoteIPTrustedProxy 13.224.0.0/14
RemoteIPTrustedProxy 70.132.0.0/18
RemoteIPTrustedProxy 210.51.40.0/24
RemoteIPTrustedProxy 13.249.0.0/16
RemoteIPTrustedProxy 205.251.208.0/20
RemoteIPTrustedProxy 58.254.138.0/25
RemoteIPTrustedProxy 116.129.226.0/25
RemoteIPTrustedProxy 52.222.128.0/17
RemoteIPTrustedProxy 64.252.128.0/18
RemoteIPTrustedProxy 205.251.254.0/24
RemoteIPTrustedProxy 71.152.0.0/17
RemoteIPTrustedProxy 216.137.32.0/19
RemoteIPTrustedProxy 204.246.172.0/24
RemoteIPTrustedProxy 120.52.39.128/27
RemoteIPTrustedProxy 118.193.97.64/26
RemoteIPTrustedProxy 223.71.71.96/27
RemoteIPTrustedProxy 130.176.0.0/16
RemoteIPTrustedProxy 54.240.128.0/18
RemoteIPTrustedProxy 205.251.250.0/23
RemoteIPTrustedProxy 180.163.57.0/25
RemoteIPTrustedProxy 52.46.0.0/18
RemoteIPTrustedProxy 223.71.11.0/27
RemoteIPTrustedProxy 52.82.128.0/19
RemoteIPTrustedProxy 54.239.128.0/18
RemoteIPTrustedProxy 36.103.232.128/26
RemoteIPTrustedProxy 52.84.0.0/15
RemoteIPTrustedProxy 111.51.66.0/24
RemoteIPTrustedProxy 143.204.0.0/16
RemoteIPTrustedProxy 144.220.0.0/16
RemoteIPTrustedProxy 120.52.153.192/26
RemoteIPTrustedProxy 119.147.182.0/25
RemoteIPTrustedProxy 120.232.236.0/25
RemoteIPTrustedProxy 54.182.0.0/16
RemoteIPTrustedProxy 58.254.138.128/26
RemoteIPTrustedProxy 120.253.245.192/27
RemoteIPTrustedProxy 54.239.192.0/19
RemoteIPTrustedProxy 120.52.12.64/26
RemoteIPTrustedProxy 99.84.0.0/16
RemoteIPTrustedProxy 54.230.0.0/16
RemoteIPTrustedProxy 52.124.128.0/17
RemoteIPTrustedProxy 204.246.164.0/22
RemoteIPTrustedProxy 13.35.0.0/16
RemoteIPTrustedProxy 204.246.174.0/23
RemoteIPTrustedProxy 36.103.232.0/25
RemoteIPTrustedProxy 119.147.182.128/26
RemoteIPTrustedProxy 118.193.97.128/25
RemoteIPTrustedProxy 120.232.236.128/26
RemoteIPTrustedProxy 204.246.176.0/20
RemoteIPTrustedProxy 120.253.241.160/27
RemoteIPTrustedProxy 13.124.199.0/24
RemoteIPTrustedProxy 35.167.191.128/26
RemoteIPTrustedProxy 18.200.212.0/23
RemoteIPTrustedProxy 99.79.169.0/24
RemoteIPTrustedProxy 52.15.127.128/26
RemoteIPTrustedProxy 34.223.12.224/27
RemoteIPTrustedProxy 54.233.255.128/26
RemoteIPTrustedProxy 13.54.63.128/26
RemoteIPTrustedProxy 13.59.250.0/26
RemoteIPTrustedProxy 3.234.232.224/27
RemoteIPTrustedProxy 52.66.194.128/26
RemoteIPTrustedProxy 13.228.69.0/24
RemoteIPTrustedProxy 64.252.64.0/18
RemoteIPTrustedProxy 18.216.170.128/25
RemoteIPTrustedProxy 3.231.2.0/25
RemoteIPTrustedProxy 52.220.191.0/26
RemoteIPTrustedProxy 34.232.163.208/29
RemoteIPTrustedProxy 35.162.63.192/26
RemoteIPTrustedProxy 34.223.80.192/26
RemoteIPTrustedProxy 34.226.14.0/24
RemoteIPTrustedProxy 13.113.203.0/24
RemoteIPTrustedProxy 34.195.252.0/24
RemoteIPTrustedProxy 52.52.191.128/26
RemoteIPTrustedProxy 52.56.127.0/25
RemoteIPTrustedProxy 34.216.51.0/25
RemoteIPTrustedProxy 52.199.127.192/26
RemoteIPTrustedProxy 52.212.248.0/26
RemoteIPTrustedProxy 13.210.67.128/26
RemoteIPTrustedProxy 35.158.136.0/24
RemoteIPTrustedProxy 52.57.254.0/24
RemoteIPTrustedProxy 52.78.247.128/26
RemoteIPTrustedProxy 52.47.139.0/24
</IfModule>

then, in WHM-Apache Configuration-Global Configuration change substitute %h with %a in LogFormat (combined) value.  tail logs in /usr/local/apache/domlogs/ to check this is working.

WordPress – WordFence

Additionally, in a WordPress scenario, if WordFence is using X-Forwarded-For then it also needs to know the list of CloudFront CIDR ranges, otherwise with the Nginx solution it can happily use the X-Real-IP or REMOTE_ADDR option.

CloudFlare

CloudFlare also uses a Cloudflare “CF-Connecting-IP” HTTP header and there’s also a mod_cloudfront Apache module available to process this (though Cloudflare themselves have now deprecated it in favour of mod_remoteip).

It’s also a shorter list, included by default with Engintron but latest versions available at: https://www.cloudflare.com/ips-v4 .

ConfigServer Security & Firewall (csf)

Consider settings to update when putting behind proxy:

  • consider whitelisting the proxy addresses: if these addresses trigger an IP block rule the site will go down – eg CloudFront HTTP 502 Status Code (Bad Gateway).
  • or, disable CT_LIMIT since a high number of connections from a small number of ip addresses is now expected

Lastly, suspicious behaviour coming from within the AWS network, can be reported to AWS Abuse reporting: https://support.aws.amazon.com/#/contacts/report-abuse.

 

part of a series on cPanel migration to AWS

One thought on “How to [un]block your CDN and detect your real visitors

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s