RTBH with BIRD and JunOS

Remote Triggered Black Hole is a very useful Security Tool that can be extremely helpful during DDoS Attacks, it leverages on BGP Communities and help us drop tarffic before it reaches our Infrastructure.

Many could (understandably) argue that by blackholing it mens that the DDoS was successful and you lose, but thats not entirely true, there are many situations where its worth sacrificing one of your services in order to save all of the others, or perhaps the target of the attack is an IP that is not even in use, nonetheless, traffic still reaches your network and exhaust your resources, this would be another perfect candidate to be Blackholed.

Ideally you would implement RTBH with your ISP, most of the big players Support Blackhole Communities, if that is not an option, you could still implement it at your own Edge. Implementing it with your ISP means that traffic will be dropped all together before reaching your network, applying it at your Edge would protect the Backend Infrastructure, but could still Impact your Edge (eg: Volumetric attack).

On this example we will be spinning up 4 VMs: RTBH, Edge, ISP and Internet. The RTBH Instance will act as a BGP OOB Speaker, it will be running Ubuntu and BIRD as the BGP Daemon. The EDGE and ISP will be running JunOS, the “Internet” instance will be used to test the result of the Blackhole. More detail on each one to come.

VAGRANT

On this example I will be using Vagrant to spin up JunOS Routers and Ubuntu Instances. Vagrant is an interesting tool that helps you manage, build, destroy VMs, its is extremely powerful, you can spin up entirely Labs in minutes, its worth a look. I won’t be getting in details on how it works, but here’s a quick and very simple guide of how this Lab was setup:

vagrant plugin install vagrant-host-shell vagrant-junos
vagrant box add juniper/ffp-12.1X47-D15.4-packetmode
vagrant init

Considering that you already have Vagrant installed, this will add the Plugins needed to run the JunOS Instances, it will then download a vSRX in packet-mode (Meaning that Stateful is disabled, making it pretty much a pure Router). With that done, vagrant init should have added a Vagrantfile to your currently directory, by editing it and replacing the code by the following we should be good to go:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  config.vm.define "edge" do |edge|
    edge.vm.box = "juniper/ffp-12.1X47-D15.4-packetmode"
    edge.vm.network "private_network", virtualbox__intnet: "edge_isp", auto_config: false
    edge.vm.network "private_network", virtualbox__intnet: "edge_rtbh", auto_config: false
  end

  config.vm.define "isp" do |isp|
    isp.vm.box = "juniper/ffp-12.1X47-D15.4-packetmode"
    isp.vm.network "private_network", virtualbox__intnet: "edge_isp", auto_config: false
    isp.vm.network "private_network", virtualbox__intnet: "isp_internet", auto_config: false
  end

  config.vm.define "rtbh" do |rtbh|
    rtbh.vm.box = "ubuntu/trusty64"
    rtbh.vm.network "private_network", virtualbox__intnet: "edge_rtbh", auto_config: false
  end

  config.vm.define "internet" do |internet|
    internet.vm.box = "ubuntu/trusty64"
    internet.vm.network "private_network", virtualbox__intnet: "isp_internet", auto_config: false
  end

end

All right, all we need to do now is run vagrant up and this will bring the 4 VMs up with the proper defined Network Interfaces.

BIRD
BIRD is a Linux Routing Daemon, as a RTBH Station, we will set it up as a OOB BGP Speaker, it will peer with our Edge Router, and its goal will be to Inject /32 Routes that we’d like to Blackhole.

Modern versions of the Linux Kernel let us create Multiple Routing Tables, we will use that feature and will create a RT dedicated to the Blackholed Routes, BIRD will then Sync its BGP Table with the RT that we created, whenever it see a new route it will Inject into BGP and advertise to Edge (Given that it match our Filter). Here’s a configuration example with comments. Using a dedicated RT for the Blackholed IPs helps with Management, Operation, Automation, etc. Checking what IPs you have blackholed is a matter of listing the RTBH Route Table.

# Setup eth1 IP address on same segment as our Edge Router
ifconfig eth1 10.0.0.100 netmask 255.255.255.0
#Create a new Route Table, call it RTBH with ID 100
echo "100   rtbh" >> /etc/iproute2/rt_tables
# Install BIRD
apt-get update
apt-get install bird

#Apply the BGP Configurations
cd /etc/bird
cp -p bird.conf bird.conf.bkp

BIRD will read the configuration from file /etc/bird/bird.conf, here’s the full configurations used with comments:

Here’s with will happen. we define a function called check_prefix(), we add to the function a List of the prefixes which we own and that contains IP that we may want to Blackhole at some point, we then make sure that only /32 IPs are advertised. This function will be called on our filter named EXPORT_BLACKHOLED_32. This filter will advertise only what is True from check_prefix(), and it will add a Community AS:666.

EDGE

Here at the Edge of our Network we will be running JunOS. This router will iBGP peer with our RTBH Instance, and will eBGP with our ISP. Here’s a simple configuration to illustrate our example:

#Interface configuration
set interfaces ge-0/0/1 unit 0 family inet address 203.0.113.1/25
set interfaces ge-0/0/2 unit 0 family inet address 10.0.0.1/24

#Each one of these IP will be used to illustrate the Blackhole in operation
set interfaces lo0 unit 0 family inet address 203.0.113.130/32
set interfaces lo0 unit 0 family inet address 203.0.113.131/32
set interfaces lo0 unit 0 family inet address 203.0.113.132/32

#BGP Configuration
set protocols bgp advertise-inactive
set routing-options router-id 203.0.113.1
set routing-options autonomous-system 65001

#iBGP to BIRD (RTBH)
set protocols bgp group IBGP type internal
set protocols bgp group IBGP local-address 10.0.0.1
set protocols bgp group IBGP neighbor 10.0.0.100
set protocols bgp group IBGP import IMPORT-FROM-RTBH

# This import filter rule is to make sure that we don't receive anything other than /32 + blackhole community from the RTBH Station
set policy-options policy-statement IMPORT-FROM-RTBH term 1 from route-filter 0.0.0.0/0 prefix-length-range /32-/32
set policy-options policy-statement IMPORT-FROM-RTBH term 1 from community BLACKHOLE
set policy-options policy-statement IMPORT-FROM-RTBH term 1 then accept
set policy-options policy-statement IMPORT-FROM-RTBH term 20 then reject

#EBGP to ISP
set protocols bgp group EBGP type external
set protocols bgp group EBGP local-address 203.0.113.1
set protocols bgp group EBGP neighbor 203.0.113.2 peer-as 65002
set protocols bgp group EBGP export EXPORT-TO-ISP

set policy-options community BLACKHOLE members 65001:666

#Define the Export Filter. Only our /25 prefix and the prefixes announced from BIRD, with the proper Community, will
# be advertised to our ISP.
set policy-options policy-statement EXPORT-TO-ISP term 1 from protocol static
set policy-options policy-statement EXPORT-TO-ISP term 1 from route-filter 203.0.113.128/25 exact
set policy-options policy-statement EXPORT-TO-ISP term 1 then accept
set policy-options policy-statement EXPORT-TO-ISP term 5 from community BLACKHOLE
set policy-options policy-statement EXPORT-TO-ISP term 5 then accept
set policy-options policy-statement EXPORT-TO-ISP term 20 then reject

set routing-options static route 203.0.113.128/25 reject

This configuration should be pretty straight forward. We should receive the routes advertised from the RTHB Station via BGP, the Import Filter will make sure that we only receive /32s that have the proper defined Blackhole Community set, this is to avoid a mistake on the BIRD Configuration and end-up announcing what it shouldn’t, which could be really bad :ˆ) The Edge Router will Advertise to the Internet our Service Prefix Range (203.0.113.128/25 in this example) + The Blackholed prefixes, anything else that we try to advertise to ISP will be dropped.

Note: 
In this example we are setting up RTBH with an Upstream ISP, and letting them drop the traffic. If your ISP does not support RTBH you can still use it on your own Edge Router, saving resources of your Backend Infrastructure, to do so it should be a matter of adapting the below ISP Configuration Section and apply it to your Edge Router.

ISP

Here we will simulate what the ISP would do with your traffic, each ISP will implement it on a different way, but the idea is the same, based on the Blackhole Community they should Drop traffic to the specified prefix/ip. Here’s an example of how I did it on our lab:

set interfaces ge-0/0/1 unit 0 family inet address 203.0.113.2/25
set routing-options router-id 203.0.113.2
set routing-options autonomous-system 65002

set protocols bgp group EBGP type external
set protocols bgp group EBGP local-address 203.0.113.2
set protocols bgp group EBGP neighbor 203.0.113.1 peer-as 65001
set protocols bgp group EBGP import IMPORT-FROM-AS65001

set policy-options community BLACKHOLE members 65001:666
set policy-options community NO-EXPORT members no-export
set policy-options community NO-ADVERTISE members no-advertise

set policy-options policy-statement IMPORT-FROM-AS65001 term 1 from route-filter 203.0.113.128/25 exact
set policy-options policy-statement IMPORT-FROM-AS65001 term 1 then accept
set policy-options policy-statement IMPORT-FROM-AS65001 term 5 from community BLACKHOLE
set policy-options policy-statement IMPORT-FROM-AS65001 term 5 then community add NO-EXPORT
set policy-options policy-statement IMPORT-FROM-AS65001 term 5 then community add NO-ADVERTISE
set policy-options policy-statement IMPORT-FROM-AS65001 term 5 then next-hop reject
set policy-options policy-statement IMPORT-FROM-AS65001 term 5 then accept
set policy-options policy-statement IMPORT-FROM-AS65001 term 20 then reject

It should only accept our /25 prefix or the /32 with Blackhole Community, if the community 65001:666 is set, the Router will add the /32 with a next-hop Reject, meaning traffic to this IP will be dropped at the ISP Level, outside of your network, saving your network resources. The NO-EXPORT and NO-ADVERTISE communities is important to make sure that it won’t export the prefixes to be blackholed to any peer/upstream.

RESULT

Lets check it in practice. We will first show that we can reach all of our Service IPs (The ones on our Loopback interface). For that I will use the “Internet” instance, which is just an instance connected behind the ISP Router.
Screen Shot 2018-05-13 at 13.52.49.png

Now, lets go ahead and Trigger a Blackhole:
Screen Shot 2018-05-13 at 13.08.58.png

As you can see, 203.0.113.130/32 was added to the rtbh table, and to show as an example, I also added 203.0.113.131/32 into the Main Route Table, which should not be advertised Upstream as our configuration plan is to keep all blackholed IPs into a single rtbh namespace.
Screen Shot 2018-05-13 at 13.16.56.png

On the EDGE we can see that, as expected, we are learning a single /32 from BIRD, and it has the community 65001:666 set. So far so good. Now to the ISP Router:
Screen Shot 2018-05-13 at 13.28.58.png

We have our ISP Receiving our Service Prefix (/25), but also our Blackholed /32, which in turn it add a Next-Hop Type Reject, which will drop traffic to that IP before it reach us. Back to the Test Instance, lets check the results:
Screen Shot 2018-05-13 at 13.33.11.png

And Voilà. It works. Traceroute show traffic getting dropped at our ISP Router (198.51.100.1).

SUMMARY

I hope this can help someone looking to Implement a RTBH Solution in their Network, or even someone that’s trying to start with BIRD, which is a very powerful Solution. Bear in mind that this Post can be used as an example/idea, but each ISP implements RTBH in a different way, different Community Numbers, Single or Split BGP Sessions, etc..
RFC7999 tries to Standardise the Blackhole Community but not all ISP follows it yet, so Make sure to contact your ISP and see if they Support Blackhole and what kind of Setup they have.

Até a Proxima!

Advertisements

Cisco ZBF and ICMP Inspect Drops

I stumbled into a interesting issue the other day with icmp inspect breaking MTR. After cutting over traffic to an Cisco ASR1001HX running IOX-XE Zone Bases Firewall, mtr running from behind the ZBF was showing 99.9% packet-loss for all the hops between the ZBF and the Last.

There was no real packet loss and the last hop was always showing 0% loss as expected, however Monitoring Systems that relies on MTR to monitor hop-by-hop went completely crazy, thinking that we were having real packet loss.

Screen Shot 2018-02-12 at 12.07.44

We noticed messages similar to the following on the Logs:

%IOSXE-6-PLATFORM: R0/0: cpp_cp: QFP:0.0 Thread:122 TS:00002775229457146791 %FW-6-DROP_PKT: Dropping icmp pkt from internal0/0/recycle:0 X.X.X.X:11 => 10.152.10.139:0(target:class)-(ZP-TRUST-TO-UNTRUST:CM-TRUST-TO-UNTRUST-ALLOW) due to ICMP ERR Pkt:exceed burst lmt with ip ident 0
%IOSXE-6-PLATFORM: R0/0: cpp_cp: QFP:0.0 Thread:025 TS:00002773939123637161 %FW-6-DROP_PKT: Dropping icmp pkt from Port-channel1 X.X.X.X:11 => 10.152.10.139:0(target:class)-(ZP-TRUST-TO-UNTRUST:CM-TRUST-TO-UNTRUST-ALLOW) due to ICMP ERR Pkt:exceed burst lmt with ip ident 4427

First thought seeing ICMP ERR Pkt:exceed burst lmt is that it was some sort of icmp rate-limit on the box, so we disabled ip icmp rate-limit unreachable that’s enabled by default and is used to limit the amount of unreachable icmp packets in a X amount of time.

Well, that didn’t work. Paying more attention to the logs we noticed that they were all icmp type 11 code 0, that is ttl-exceeded in transit, ZBF was dropping the icmp 11/0 responses, and because of that the source running MTR would never receive them and would that packet loss was happening.

Solution? Add an ICMP Pass rule on your Policy-Map, making ICMP Stateless for the Firewall.

class-map type inspect match-any CM-UNTRUST-TO-TRUST-ICMP-ALLOW
 match protocol icmp

policy-map type inspect PM-UNTRUST-TO-TRUST
 class type inspect CM-UNTRUST-TO-TRUST-ICMP-ALLOW
  pass
 class type inspect CM-UNTRUST-TO-TRUST-ALLOW
  inspect
 class class-default

Pass means that the ZBF will allow the traffic according to your class-map, and won’t keep any state of the connection, making it stateless for icmp traffic, you can apply it to your Untrust to Trust zone-pair, and MTR should work fine again.

You can and probably should be more specific and match on ttl-exceeded instead of all icmp messages, but this should be a good enough example to illustrate the issue.

I hope this can help someone having the same issue, I honestly couldn’t find much about this with my Google-Fu, and I still haven’t heard from Cisco with an official answer, if its a bug or expected for some reason.. If you know more about it, please write a comment 🙂

Cisco IOS-XE Guest Shell

Cisco IOS-XE now comes with a neat feature called Guest Shellit give us the power of spinning up a Linux Container on the router, giving us many new Network Programmability options, the main one being the option of running custom Python Scripts. With that we can write things like custom Metrics, Integrations, Features and tools not natively available.

Enabling the Guest Shell

I will be using a Cisco CSR1000v on AWS as an example (its easier to spin up and lab), but the feature is also available on ASR and ISR, and the way to enable is is fairly similar.
When you bring the CSR up, you should notice that it has a VirtualPortGroup Interface enabled, this Interface will be responsible for the Interface between the GuestShell Container and the router IOS-XE/External World. On platforms with managements ports, the mgmt interface/vrf would be used for the Interface with the GuestShell.

app-hosting appid guestshell
 vnic gateway1 virtualportgroup 0 guest-interface 0 guest-ipaddress 192.168.35.2 netmask 255.255.255.0 gateway 192.168.35.1
 resource profile custom cpu 1500 memory 512

guestshell enable VirtualPortGroup 0 guest-ip 192.168.35.2

That’s it, we can run show app-hosting list to confirm that GuestShell is enabled:

#show app-hosting list
App id                           State
------------------------------------------------------
guestshell                       RUNNING

To gain shell access, run  guestshell run bash:

#guestshell run bash
[guestshell@guestshell ~]$ uname -a
Linux guestshell 4.4.51 #1 SMP Sun Apr 23 01:42:33 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux
[guestshell@guestshell ~]$

We are now able to start playing with Python. Your main Python package will be the cli package, this let you run IOS commands from the shell:

[guestshell@guestshell ~]$ python
Python 2.7.5 (default, Jun 17 2014, 18:11:42)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from cli import cli
>>> from cli import configure
>>>
>>> configure('hostname IT-WORKS')
[ConfigResult(success=True, command='hostname IT-WORKS', line=1, output='', notes=None)]
>>> cli('show running-config | in hostname')
'\nhostname IT-WORKS\n'
>>>

It also comes with pip installed, so you are able to install pretty much any package that you need to write your code.

I’ve been implementing a bunch of Cisco routers with ZBF lately, its a simple and nice Firewall solution for those that don’t need all that fancy features that vendors as CheckPoint or Palo Alto offer you, and one thing that happened a couple of times is ACL out o Sync between the devices, before we wrote a NAPALM script to update both devices at same time (more of this on a future post), I would sometimes update the ACL in one Router and forget to update the pair device, I would then test a failover and notice that traffic was dropped on the secondary router. (Oops!)

Verify ACL Sync with IOS-XE GuestShell and Python

This gave the Idea to write this example using GuestShell and Python for this blog post, the following Script runs after X specified seconds, compare the Local ACLs with the Remote ACLs, and generate a Syslog Message in case they are out of sync. Simple and helpful. Here’s the code, which is also available on my github:

Copy this to a .py on the GuestShell, make it an executable with chmod +x file_name.py, and execute it! If the ACLs on both devices are not the same, you will see this log msg on the Router:

*Feb  8 11:57:16.924: %SYS-4-USERLOG_WARNING: Message from tty3(user id: ): ATTENTION. ACLs are not in Sync, local ACL has more entries than Peer.

If you have a syslog Collector that Integrates with Slack or some other Chat Platform that your team uses, it should be easy to see when the ACLs are not in-sync, and now anyone can go to their favourite Version Control system where they manage their ACLs and re-apply the ACL to the device with missing entries. I am sure you store you ACLs in a repo, right? 🙂

With imagination and some available time this Script could be much improved, and many other cool tools/features could be written to level-up your Cisco IO-XE Routers!

I hope this helps someone, thanks for reading.

FHRP On AWS with Ansible, Keepalived and Python

AWS Does not support Broadcast or Multicast, so implementing a FHRP Solution as we are used to do on-premise won’t work. Fortunately, keepalived support Unicast Peers, so implementing it on AWS is no problem!

The use-case for this post continues where we left on the previous one with the difference that we will be using 2 VPN instances per vpc-region running Ubuntu instead of CentOS. The full configuration is found on my github. Here’s a quick illustration:

Screen Shot 2018-02-06 at 10.20.10

We have LibreSwan running on the vpn instance, and tunnels between
vpn01a.euw1<>vpn01a.euc1 and another between vpn02a.euw1<>vpn02a.euc1.

Each subnet on AWS has a Route Table attached to it, you could think of it as being similar to a VRF when we think about external routes, but internal to the VPC, all subnets have Layer2 access between them, so unless you apply SG and NACL, everything can reach everything inside the same VPC.

Here’s how my vpn01a Route Table instances looks like right now:

Both have a default Route via the First IP of the Subnet, which on AWS will always be the local subnet AWS Route Table. They both know how to reach the remote VPC via the IPSEC Tunnel (vti0), and they can indeed reach the other side. However, the AWS RT’s still does not have the proper routes, they don’t know how to reach the remote side, meaning that all the other Instances that relies on the AWS RT (all of them do!) won’t be able to reach the remote vpc. Heres how my private-subnet RT looks on EUW1:
Screen Shot 2018-02-06 at 10.37.03

As we can see, EUW1 does not know how to reach EUC1 (10.240.0.0/24)! A easy fix would be adding a Manual Route to 10.240.0.0/24, set next-hop to the instance vpn01a, do the same on the EUC1 side with the inverse route, done! It works, but what if vpn01a fails? Also, adding manual routes is a nightmare when it starts to grow. AWS Does not (yet) supports any type of Dynamic Routing Protocols, so to workaround that we will be using KeepAlived + a crafted Python Script for Dynamic Route Injection, and we’re going to use Ansible to Automate the KeepAlived deployment.

KeepAlived

VRRP Is a old friend of us, Network Engineers, we’ve been doing it on our Routers forever, so why not using this powerful FHRP solution also on the Public Cloud? To deploy the keepalived configuration into our VPN Instance, we will be using Ansible. I assume Keepalived is already installed, if not, please install it. Here’s how the playbook and the Jinja2 Template looks like:

---
- hosts: vpn0*.euc1.*
  #gather_facts: no
  vars:
    left_side: 'vpn01.euc1.netoops.net'
    right_side: 'vpn02.euc1.netoops.net'
    host1: "{{ hostvars['vpn01.euc1.netoops.net']['ansible_ens3']['ipv4']['address'] }}"
    host2: "{{ hostvars['vpn02.euc1.netoops.net']['ansible_ens3']['ipv4']['address'] }}"

  tasks:
  - name: write the keepalived config file
    template: src=keepalived.j2 dest=/etc/keepalived/keepalived.conf
    become: true
    notify:
      - restart keepalived

  - name: ensure keepalived is running
    service: name=ipsec state=started
    become: true

  handlers:
    - name: restart keepalived
      service: name=keepalived state=restarted
      become: true
{% if ( left_side in inventory_hostname) %}
vrrp_instance VPN {
    interface ens3
    state MASTER
    priority 200

    virtual_router_id 33
    unicast_src_ip {{ host1 }}
    unicast_peer {
        {{ host2 }}
    }

  notify_master "/usr/local/bin/master.sh"

}
{% endif %}

{% if ( right_side in inventory_hostname) %}
vrrp_instance VPN {
    interface ens3
    state BACKUP
    priority 100

    virtual_router_id 33
    unicast_src_ip {{ host2 }}
    unicast_peer {
        {{ host1 }}
    }

    notify_master "/usr/local/bin/master.sh"
}
{% endif %}

The trick to make VRRP works on AWS is the following lines of the KeepAlived configuration:

    unicast_src_ip {{ host1 }}
    unicast_peer {
        {{ host2 }}
    }

We change its behaviour from Multicast to Unicast, and with Ansible we are able to get facts of any Inventory Instance, including its IP address, making the automation more dynamic by not having to worry what IP the Instance is using, these are the lines of our Playbook that takes care of the IP address of the Instance:

    host1: "{{ hostvars['vpn01.euc1.netoops.net']['ansible_ens3']['ipv4']['address'] }}"
    host2: "{{ hostvars['vpn02.euc1.netoops.net']['ansible_ens3']['ipv4']['address'] }}"

Note: Don’t forget to quote 🙂

KepAlived also has a neat feature called notify, where we are able to run a shell script as the Node change its state. Here in our example, whenever the Node gets into MASTER state, we are going to run a shell script called master.sh. This script in turn will be responsible to call the aws_inject_routes.py Script that takes care of Injecting the routes Into the AWS Route Tables.

#!/bin/bash
/usr/local/bin/aws_route_inject.py
echo "Route Injection Done" | cat >/var/tmp/test.log

Note: I installed the scripts into the Instances during boot time using user-data on Terraform.

That’s it! With that, whenever a Instance become Master, it will Inject the VPN Routes into the AWS Route Table with the Next-Hop as itself, traffic from other Instances on the VPC should then be able to access the remote VPN locations via the vpn instances. Lets try it.

First we define on our keepalived-playbook.yml in which instances we want to deploy the config. In our example, we want the VRRP Cluster between vpn01.euw1 and vpn02.euw1, we then save and run the Playbook:

Screen Shot 2018-02-06 at 11.17.31Screen Shot 2018-02-06 at 11.17.48

Done! We have a VRRP Cluster UP and Running on AWS. Now, if our setups works, when vpn01a became Master, notify_master should have called the Python Script, and the routes should be available on the AWS RT with a next-hop as vpn01a-instance-id. Lets check:
Screen Shot 2018-02-06 at 11.23.12

Hooray! Now, lets force KeepAlived to fail on vpn01a by stopping its service and see what happens while we tail vpn02a logs:
Screen Shot 2018-02-06 at 11.24.44Screen Shot 2018-02-06 at 11.25.38

Sweet! Routes converged to vpn02a as expected! But not enough to failover the traffic on both side, so far we were able to failover the Tunnel in only one side, to make keepalived trigger the failover/routes change on both VPCs we need to make use of the feature called vrrp_script. Its a small change to our Keepalived template, here’s how it looks:

vrrp_script vpn_check {
  script       "/usr/local/bin/vpn_check.sh"
  interval 2   # check every 2 seconds
  fall 2       # require 2 failures for KO
  rise 2       # require 2 successes for OK
}
vrrp_instance VPN {

......

    track_script {
    vpn_check
  }    
#!/bin/bash
VTI=$(ifconfig | grep vti)
if [ -z "$VTI" ]
then
        exit 1
else
        exit 0
fi

That’s it, keepalived will run the vpn_check.sh script every 2 seconds, and the script has the simple task of checking if the VPN is up, by checking if a VTI interface exists. If when return is 1 VRRP will enter in FAULT state and failover, when resturn is 0 health check will pass.

To test it, I’ve spin up 1 test instance in each VPC so we can ping from one to the other and here’s the Result:

Screen Shot 2018-02-06 at 14.23.06

From EUW1(10.250/24) I have a MTR to 10.240.0.84, our test instance in EUC1. Traffic is flowing symmetrical via vpn01a.euw1 (10.250.0.18) and vpn01a.euc1 (10.240.0.9). Now I will go ahead and Terminate vpn01a.euw1, to simulate a Instance Failure (and also because I need to shutdown the lab to save money).

Screen Shot 2018-02-06 at 14.23.59

6 Packet Loss, which is pretty much due the health check timers that we have, or it could be much quicker! Here’s the prove that traffic shifted:

Screen Shot 2018-02-06 at 14.24.20

Hooray!

The goal here is to help people with Ideas of networking tricks that can be done on Public Clouds, there’s much more we could (and should) do here to improve this use-case, like dynamic routing on the VTIs and better Health Check Scripts so we improve failover/recovery and reduce errors, also this setup works only as Active/Standby, but it should be possible to improve and make works as Active/Active, etc..
As the network grows it will start to get challenging to manage static p2p Tunnels, so might be time to start Thinking about AWS Transit if the hub-spoke Latency is not a issue, or a automated DMVPN Solution as shown on the end of this presentation from re:Invent 2017 🙂

Hope this helps someone. Adios.

LibreSwan IPSEC Tunnel with Ansible

On the previous post I wrote about deploying a AWS Network Stack with Terraform and how to use Terraform to deploy a Linux Instance with LibreSwan installed.

I’ve been wanting to learn Ansible for a while now, so on this post we are going to use it to make it easy to deploy VPN Configuration into new VPN Instances. I won’t get into details of how Ansible works because they have a great Documentation, its easy to understand the basics and start playing with it!

On this setup we will be using the two VPCs in eu-central-1 and eu-west-1 that we built on the previous posts, with the difference that a second vpn-instance was added per vpc, so we can have 2 vpc-instance for redundancy, and also the Security Groups were changed because I notice I had left it allowing everything (Oops!).

The full thing can be found here on my Git Repo.

Playbook Tree

This is the File structure used by our vpn-instances Playbook:

└── vpn-instances
    ├── hosts
    ├── libreswan.j2
    ├── libreswan_secrets.j2
    ├── vpn-playbook.yml
    └── vpn_vars
        ├── vault.yml
        ├── vpn01-euc1-euw1.yml
        └── vpn02-euc1-euw1.yml

Hosts is our Ansible Inventory File, it should list all of our VPN Instances:

[euc1]
vpn01.euc1.netoops.net ansible_host=18.196.67.47
vpn02.euc1.netoops.net ansible_host=52.58.236.243

[euw1]
vpn01.euw1.netoops.net ansible_host=34.243.28.111
vpn02.euw1.netoops.net ansible_host=52.211.207.206

Libreswan.j2 and libreswan_secrets.j2 are our Jinja2 Template Files, these holds the LibreSwan Template, and the Playbook will pass our defined variables to it and deploy it to our vpn instances.

{% if ( left_side in inventory_hostname) %}
conn {{ conn_name }}
  left=%defaultroute
  leftid={{ left_id }}
  right={{ right_peer }}
  authby=secret
  leftsubnet=0.0.0.0/0
  rightsubnet=0.0.0.0/0
  auto=start
  # route-based VPN requires marking and an interface
  mark=5/0xffffffff
  vti-interface={{ vti_left }}
  # do not setup routing because we don't want to send 0.0.0.0/0 over the tunnel
  vti-routing=no

{% endif %}

{% if ( right_side in inventory_hostname) %}
conn {{ conn_name }}
  right=%defaultroute
  rightid={{ right_id }}
  left={{left_peer}}
  authby=secret
  leftsubnet=0.0.0.0/0
  rightsubnet=0.0.0.0/0
  auto=start
  # route-based VPN requires marking and an interface
  mark=5/0xffffffff
  vti-interface={{ vti_right }}
  # do not setup routing because we don't want to send 0.0.0.0/0 over the tunnel
  vti-routing=no
{% endif %}%
{{ left_peer }} {{right_peer}} : PSK "{{ vpn_psk }}"%

vpn-playbook.yml Is the Ansible Playbook. Here’s where we tell Ansible what to do and how to do it. The Playbook has a YAML format, where you define Tasks and defines hosts where the Tasks needs to be applied.

---
- hosts: vpn0x.xxx1.*,vpn0y.other-yyy1.*
  gather_facts: no
  vars_files:
    - ./vpn_vars/vault.yml
    - ./vpn_vars/vpn0x-xxx1-yyy1.yml

  tasks:
  - name: write the vpn config file
    template: src=libreswan.j2 dest=/etc/ipsec.d/{{ conn_name }}.conf
    become: true
    register: tunnel

  - name: write the vpn secrets file
    template: src=libreswan_secrets.j2 dest=/etc/ipsec.d/{{ conn_name }}.secrets
    become: true


  - name: Activate the tunnel
    shell: "{{ item }}"
    with_items:
      - ipsec auto --rereadsecrets
      - ipsec auto --add {{ conn_name }}
      - ipsec auto --up {{ conn_name }}
    become: true
    when: tunnel.changed


  - name: Install Routes left
    shell: ip route add {{ right_subnet }} dev {{ vti_left }}
    when: (left_side in inventory_hostname) and tunnel.changed
    become: true

  - name: Install Routes Right
    shell: ip route add {{ left_subnet }} dev {{ vti_right }}
    when: (right_side in inventory_hostname) and tunnel.changed
    become: true

  - name: ensure ipsec is running
    service: name=ipsec state=started
    become: true

The vault.yml is an Ansible Vault, here is where we are going to Store our PSKs as we don’t want them to be available in Clear-Text in our configuration Files, and specially not in clear-text on a Git Repo, so do remember to also add the vault.yml to your .gitignore 🙂

Finally we have the vpn01-euc1-euw1.yml and vpn02-euc1-euw1.yml file. These files holds our Variables, with all the Parameters that we need to setup each one of our IPSEC VPN Pairs. Here is the example of one of them:

---
#Name for the VPN Connetion
conn_name: euc1-euw1
#EUW1, Left Side
left_id: 34.243.28.111
left_peer: 34.243.28.111
left_side: vpn01.euw1.netoops.net
vti_left: vti0
left_subnet: 10.250.0.0/24

#EUC1, right Side
right_peer: 18.196.67.47
right_id: 18.196.67.47
right_side: vpn01.euc1.netoops.net
vti_right: vti0
right_subnet: 10.240.0.0/24

#PSK to be used. Note: PSK is stored on vault-psk
vpn_psk: "{{ vault_vpn01_psk }}"%

For each new P2P IPSEC That we want to setup, we will create a new file with a suggestive name, we are going to set the variables with the common VPN params as Local Peer IP and ID, Remote Peer IP and ID, Subnet of Each Side, the VTI Tunnel Number (one VTI per Tunnel), and last but not least, our Pre-Shared Key. If you pay attention on the vpn_psk variable, you will see that it points to another variable with a prefix of vault_, this variable is stored on our Ansible Vault, and for each new Tunnel we should define a new PSK on the Vault. Try not to use the same PSK everywhere!

With your variables done, all we need to do now is go back to our Playbook and change the Variables File, pointing to the var_files we defined above and filter the hosts to apply the configuration to:

- hosts: vpn01.euc1.*,vpn01.euw1.*
  gather_facts: no
  vars_files:
    - ./vpn_vars/vault.yml
    - ./vpn_vars/vpn01-euc1-euw1.yml

That’s it. The playbook will run against the inventory file, will match the host-entries vpn01.euc1* and vpn01.euw1* (wildcards allowed, take a look on Ansible Patterns), we will source our encrypted-variables from vault.yml and our vpn variables from vpn01-euc1-euw1.yml. We only need now to execute the Playbook and the 2 VPN Instances in EUC1 and EUW1 should have their IPSEC Tunnels established and proper routes set to talk to each-other.

AWS Cross-Region Talk in no time 🙂 So far we got the 2 VPN Instances to establish the Tunnel and setup the Static Routes, the next step is to make the AWS Route Tables itself to route traffic from the AWS Subnets into the proper VPN Instances. Lets do that in a different post.

I hope this post helps to see that a Network Engineer doesn’t need to be a Programmer or a Systems Wizard to be able to start writing some Network Automation tools that will help us on our day-by-day tasks, we just need to go a little bit out of our comfort zone and start learning new and interesting skills, its fun, try it!

AWS Network and Terraform – Part two

On Part One we saw how to create the base aws network stack using Terraform, on this post we are gonna deploy a Linux instance that will be used to establish Inter-Region IPSEC Tunnels using LibreSwan.

AWS Inter-Region Traffic

In November 2017 AWS Announced the Support for Inter-Region Peering, this allow VPCs in different regions to communicate between each other using AWS’s own Backbone, however, not all Regions currently support the Inter-Region Peering Feature, and to work around that the solution is the good old IPSEC VPN Tunnels.
The reason we use VPN Instances instead of the native AWS VPN Gateway is because aws-vgw works on passive mode only, so its not possible to initiate a Tunnel between 2 aws-vgws.

VPN Instance Module

We start by defining a new Module called vpn-instance, this module will hold the template to create our Linux Instance on the Public Subnet, and we will use user-data to run a bootstrap script on the Instance, so it defines a few settings and install LibreSwan on the first boot. LibreSwan is a nice and simple Linux IPSEC implementation that should do a good job to demonstrate our use-case. Lets get started.

“../modules/vpn-instance”

We can then create our new project and make use of our previous created network-stack and the vpn-instance Modules.

“../projects/eu-central-1/main.tf”

Now we need to define our user-data script, I call it init_config.sh, and we will define what we want to run on our Instance at Lunch time.

“../projects/eu-central-1/init_config.sh”

Done. All wee need to do now is run terraform init / terraform get / terraform apply, wait a few minutes and our eu-central-1 vpc will be up and running with a Linux Instance Ready to be used as a VPN Endpoint.

Screen Shot 2018-02-02 at 20.57.26Screen Shot 2018-02-02 at 20.45.08

Now we go back to our eu-west-1 project, and all we need to do is add the module vpn-instance snippet on the main.tf, terraform get / terraform apply and it should deploy the instance also on eu-west-1. We only need to change the ami-id for one that is available in euw1.

“../projects/eu-west-1/main.tf”

Screen Shot 2018-02-02 at 23.17.33Note that only 3 Resources were added? In the previous post we had already deployed all the network-stack, now we are only adding the Resources defined on the module vpn-instance.

Terraform and Configuration Management

Terraform is a great tool for creating Infrastructure, but it is not a configuration management tool. If we go back to our user-data file and try to add or remove something, Terraform will detect the change and when you apply it the instance will be destroyed and re-created. We don’t want that every time we add a new VPN Peer, so we should use a ConfigManager such as Ansible to define our LibreSwan Configurations.

I haven’t played much with Ansible yet, but as the main goal of this blog is to help-me learn and document what I am learning, on the Next Blog post I should have an Ansible environment ready to deploy those VPN Configs, and use Terraform to define and control the AWS Route Tables.

The complete Lab can be found on my GitHub.

That’s all folks.

AWS Network and Terraform – Part one

The word out there is that Public Cloud is the solution for all the problems..a bit too strong, right? As Network Engineers we tend to have a feeling for our on-premise Datacenter where we have control over all network matters, but I strong believe that we should embrace the Cloud, it is indeed a very interesting, good, reliable and fun solution to learn and use in MANY different use-cases. AWS is the leader IaaS in the market by far, and that’s why I decide to learn my way into the Public Cloud with them. This post won’t cover AWS Networking Fundamentals, for that I recommend the 2 part blog posts by Nick Matthews “Amazon VPC for On-Premises Network Engineers – Part 1 and Part 2

Terraform is a Simple and Powerful Infrastructure as a Code tool that can be used to Template, Plan and Create Infrastructure in a multitude of different Providers, including but not limited to AWS. I suggest a look on Terraform Getting Started guide for a better idea of how it works, they have an amazing Documentation, and there’s no need to be a Programmer to understand it.

Lets play around with it so we can learn more.

Defining your Modules

Modules can be thought as Functions, you define a Module that will be reused as many times as you want, this avoid repetitive code and also give us a cleaner and easier to manage configuration. On our example, I am gonna have 2 main folders, one called Modules and another one called Projects, which is where each new Project will be placed and the Resources will all be sourced from the Modules Folder. Let start by defining the Modules. Under “/modules/networking-stack” we will create the terraform files (.tf) that will contain our Templated Code to spin-up the aws networking stack. 

“../modules/network-stack/vpc.tf”

“../modules/network-stack/network.tf”

“../modules/network-stack/subnet.tf”

“../modules/network-stack/security.tf”

With that we have a complete aws-network-stack Module, with all the pieces necessary to bring up a AWS VPC, with 2 Public & Private subnets in 2 different AZs, with the proper IGW for Public Internet on the Public Subnets, NAT Gateway providing Internet Access for Private Instances and Security Groups adding Layer3/4 Security to the Instances, so we are ready to use this Module as many time as we want in our Projects.

To demonstrate that, lets create a folder called eu-west-1 on our Projects folder, and there we are going to call the Module Network-Stack that we created above, we are going to pass variable values and run Terraform over this Project, which should create our AWS Network Stack in a matter of minutes.

“../projects/eu-west-1/main.tf”

That’s all. All we need to do now is run  terraform initterraform plan / terraform apply and the Base AWS Network Stack will be ready for use in a matter of minutes!

Screen Shot 2018-02-02 at 12.33.34

On a Next posts I want to show how to use Terraform to Manage our Route Tables, VPNs, Cross-Region (X)Swan IPSEC Tunnels, VPC-Peering, etc. Stay Tuned.

Git Repo with all the configs can be find here.