• Chris Dixon

Automating ASAv deployment in AWS

Updated: Nov 27, 2019

Introduction


ASAv is a great tool to add VPN and firewall features to your cloud environment, but to be able to launch it from an ASG (Auto Scaling Group) in AWS there's a few little caveats to work around. In this blog post we'll be running through step by step the use case of having an ASAv to terminate a Remote Access VPN, which is launched from an ASG to allow a new instance to be spawned in the event that the running instance is terminated. The script to carry out the tweaks on instance launch will be a Lambda Python script.


We will run through this in 4 Phases as follows:


1. Preparing the VPC

2. Build and customise the ASAv image

3. Creating the Launch Configuration and ASG

4. Automating the things required when ASA relaunches


Phase 1 - Preparing the VPC


Firstly, we need to create the environment for our ASAv to run in.

If they don't already exist, create a VPC, Subnets and Route tables, in your AWS environment to host the ASAv. Make sure you have 3 subnets prepared for the ASAv, as it will need 3 separate interfaces - one Management (which needs to be the primary interface) one inside, and one outside. Prepare a Key Pair which you will use to access the instance Create 2 Elastic IPs - one for Mgmt and one for Outside Create a security group with the access rules you would like for access to the mgmt. I recommend allowing ssh and https in from just your own public IP initially, as this will be required to allow asdm access to the box. In this example we are using the ASAv as an RAVPN concentrator, so the Security group for the outside interface will need to allow https/443 from everywhere.

Phase 2: Build the Image, based on a customised version of AWS Marketplace ASAv AMI


In this section we will find the official Cisco ASAv AMI in the AWS Marketplace, launch this, customise it so it is configured how we want our ASA to run, including uploading any Anyconnect image to it, and we will then rebake a new image, which will become the image that is launched from our ASG.

Go to the AWS Marketplace here: https://aws.amazon.com/marketplace and find the ASAv AMI (Select Security, Cisco, BYOL, AMI) as shown: [Screenshot1 - find_ami]


The current stable ASA software version at the time of writing is 9.12.1 Click “Continue to Subscribe” in the top right hand corner Infrastructure hosting pricing estimates are shown for various possible image sizes (specs). I'm using ASAv10 and the Cisco recommended size for this is c4.Large at a cost (hosted in EU in London) of $72 USD per month Click “Continue to Configuration” You should be able to leave the version and fulfilment option as they are, and select whichever region you would like to run it in Click “Continue to Launch” Change to “Launch from EC2” and Click Launch Make sure you specify the instance size that Cisco recommends for your instance type (e.g. mine is ASAv10 and hence I'm using c4.large). Be sure to launch it in the correct VPC and into the subnet you created for its mgmt interface Everything else can keep the defaults - Check here also that the security groups are correct [screenshot2 - ASv_select_ AMI_settings].

Click Add storage It automatically selects a 10GB HDD. Click “Review and Launch” and then click Launch It will prompt you for the Key Pair to use - choose the one you created in Phase 1, or create a new Key Pair Tick the box and click Launch Instances Go to the Instances page and make sure your new instance is booting The ASAv will hang and not boot properly until you give it 3 NICs, so now is the time to go to Network Interfaces, create an interface in each of the Outside and Inside Subnets (it should have already created the one in your mgmt subnet), and then attach both of these to the ASAv Instance. Also within the Network Interfaces tab, find the interface created for your mgmt NIC (it should be in the MGMT subnet) and associate it with your first Elastic IP. This should allow you to be able to access the ASAv via SSH. We will need to connect via SSH first using the Key-Pair, then enable the http server and http access . SSH to it and set the enable password when prompted, and configure an administrator user Then configure http server and authorised IPs to access it from as follows: Conf t Http server enable Http 0.0.0.0 0.0.0.0 management You should change the above to just your IP however I have left as 0.0.0.0 (every IP) for this example (also the sec grp in AWS is limiting access to it to a single IP). Add a static/32 route to your mgmt workstation via the mgmt interface with next hop of the gateway for your mgmt subnet (if you used a /24 as in this example then the AWS next hop to use will be .1) as follows: route [interface-name] [mgmt-workstation-IP] 255.255.255.255 [AWS-next-hop-IP]

For example route management 100.100.100.100 255.255.255.255 172.16.1.1 Enable and configure the inside and outside interfaces, set them all to dhcp, and set ONLY the outside interface to get default route from dhcp Save the config, and reboot the ASAv - you will need to do this so that it picks up the interfaces you assigned If you have a config script you want to apply, or if you prefer configuring via CLI now is the time to do that. Alternatively you can configure it via the ASDM. Configure the ASA as you need to, in this example it is acting as a Remote Access VPN concentrator so I will upload the image for that and configure it appropriately. The ASAv doesn’t ship with an Anyconnect image, so upload one now. Upload a certificate and select it for RAVPN Enable RAVPN, Create a profile, configure user auth, configure it to use an address pool for clients. Disable the src/destination check within AWS from either the instances page or from each individual nic (outside and mgmt interfaces need this disabling). Go to Running instances, click on the ASAv Image and from the actions menu, under Image, select Create Image [Screenshot 3 - Creating Image]


This takes a good few minutes so be patient, make a cup of tea, and go back and check the AMIs screen. Phase 3: Creating the Launchconfiguration and ASG

In this section we will create a Launchconfiguration based on our new customised image, and an ASG to automate the launching of our instance.

Go to Launch Configurations, New Launch Configuration [Screenshot 4 - Creating LaunchConfig]


Choose your saved AMI Choose the image size as before (c4.large in my case) Click Next: Configure Details Click Next Add storage, Check storage is as expected Click Next: Configure Security group and ensure to apply sec grp for mgmt access Click Review Click Create Launch Configuration Select your Key Pair, and tick the box Click Create Launch Configuration Click Create an AutoScaling Group using this LC Give the ASG a Name Under Network, choose your VPC Click in the Subnet Box and choose the mgmt subnet Click Review Check details and Click Create ASG Should get a green tick saying ASG created

Phase 4: Automating the things required when ASA relaunches (Attaching additional NICs, Associating EIPs, Disabling src/dest check) If you just let the ASG you just created boot the ASAv as it is, you will notice it again hangs in the initialising step, because the AMI can only have one NIC and hence the additional NICs have been taken off the instance, and it wont get past initialisation without them. So this is where the lifecycle hook comes in, which I have adapted from the following AWS article: https://aws.amazon.com/premiumsupport/knowledge-center/attach-second-eni-auto-scaling/

Firstly kill the stuck instance by going back into ASG and setting the min and desired number of instances to 0 In order for your function to be able to carry out all the jobs in this post, it will need to have a role with permissions for the following: "ec2:CreateNetworkInterface", "ec2:DescribeInstances", "ec2:DetachNetworkInterface", "ec2:DescribeNetworkInterfaces", "autoscaling:CompleteLifecycleAction", "ec2:ModifyNetworkInterfaceAttribute", "ec2:DeleteNetworkInterface", "ec2:AttachNetworkInterface", "ec2:ModifyInstanceAttribute", "ec2:AssociateAddress" If you already have a role defined that has these permissions then you can assign that role to the function, or you can create a new role from the IAM console. [screenshot of creating role]

If you want to create a custom role, go to your IAM console and create a new role with the following policy:


{
   
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateNetworkInterface",
                "ec2:DescribeInstances",
                "ec2:DetachNetworkInterface",
                "ec2:DescribeNetworkInterfaces",
                "autoscaling:CompleteLifecycleAction",
                "ec2:ModifyNetworkInterfaceAttribute",
                "ec2:DeleteNetworkInterface",
                "ec2:AttachNetworkInterface",
                "ec2:ModifyInstanceAttribute",
                "ec2:AssociateAddress"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:*:*:*"
        }
    ]
}


From the Amazon Lambda console, create a Lambda function that automatically does all the things we need (attach 2 new ENIs, disable src/dst check, associate EIP) to an instance when Auto Scaling launches it: 1. Choose Create function. 2. Add a relevant name to the Name field, and choose Python 2.7 for Runtime. 3. For Role, either choose an IAM role that gives Lambda the permissions mentioned above or choose Create a custom role if you haven’t already created one.

[Screenshot of creating function]



Choose Create function. Add the following to the Function code field, and then choose Save: In the below code, I have hardcoded where necessary the security Group IDs, EIP Association ID and subnet IDs relevant to the environment because I wanted to specify which subnet the NICs were created in, the AWS example used a method to grab the subnet ID of the instance being launched. To do the same, you will need to enter your own subnet ID, sec group ID and EIP Allocation ID in the sections where I have put '_here', for example 'subnet_outside_id_here' should be substituted for your own outside subnet ID. The instance ID is garnered from the instance being launched, so this code executes on that Instance ID only.


import boto3
import botocore
from datetime import datetime

ec2_client = boto3.client('ec2')
asg_client = boto3.client('autoscaling')

ec2_res = boto3.resource('ec2')

def lambda_handler(event, context):
    if event["detail-type"] == "EC2 Instance-launch Lifecycle Action":
        instance_id = event['detail']['EC2InstanceId']
        LifecycleHookName=event['detail']['LifecycleHookName']
        AutoScalingGroupName=event['detail']['AutoScalingGroupName']
        interface_id = create_interface('subnet_outside_id_here')
        interface_inside_id = create_interface_inside('subnet_inside_id_here')
        attachment = attach_interface(interface_id, instance_id)
        attachment2 = attach_interface_2(interface_inside_id, instance_id)
        eni_outside = ec2_res.NetworkInterface(interface_id)
        secgrpadd = eni_outside.modify_attribute(Groups=['outside_sec_group_ID_here'])
        ec2_id = ec2_client.describe_instances(InstanceIds=[instance_id])
        mgmt_id = ec2_id['Reservations'][0]['Instances'][0]['NetworkInterfaces'][0]['NetworkInterfaceId']
        mgmt_eip = ec2_client.associate_address(AllocationId='mgmt_EIP_allocationID_here',NetworkInterfaceId=mgmt_id)
        eni_mgmt = ec2_res.NetworkInterface(mgmt_id)
         outside_eip = ec2_client.associate_address(AllocationId='outside_eip_allocationID_here',NetworkInterfaceId=interface_id)                  
        srcdest = eni_mgmt.modify_attribute(SourceDestCheck={'Value': False})
        if not interface_id:
            complete_lifecycle_action_failure(LifecycleHookName,AutoScalingGroupName,instance_id)
        elif not attachment:
            complete_lifecycle_action_failure(LifecycleHookName,AutoScalingGroupName,instance_id)
            delete_interface(interface_id)
        elif not interface_inside_id:
            complete_lifecycle_action_failure(LifecycleHookName,AutoScalingGroupName,instance_id)    
        elif not attachment2:
            complete_lifecycle_action_failure(LifecycleHookName,AutoScalingGroupName,instance_id)
            delete_interface(interface_inside_id)
        else:
            complete_lifecycle_action_success(LifecycleHookName,AutoScalingGroupName,instance_id)
       
        
def get_subnet_id(instance_id):
    try:
        result = ec2_client.describe_instances(InstanceIds=[instance_id])
        vpc_subnet_id = result['Reservations'][0]['Instances'][0]['SubnetId']
        log("Subnet id: {}".format(vpc_subnet_id))

    except botocore.exceptions.ClientError as e:
        log("Error describing the instance {}: {}".format(instance_id, e.response['Error']))
        vpc_subnet_id = None

    return vpc_subnet_id


def create_interface(subnet_id):
    network_interface_id = None

    if subnet_id:
        try:
            network_interface = ec2_client.create_network_interface(SubnetId=subnet_id)
            network_interface_id = network_interface['NetworkInterface']['NetworkInterfaceId']
            log("Created network interface: {}".format(network_interface_id))
        except botocore.exceptions.ClientError as e:
            log("Error creating network interface: {}".format(e.response['Error']))

    return network_interface_id
    
def create_interface_inside(subnet_id):
    network_interface_id = None

    if subnet_id:
        try:
            network_interface_2 = ec2_client.create_network_interface(SubnetId=subnet_id)
            network_interface_id_2 = network_interface_2['NetworkInterface']['NetworkInterfaceId']
            log("Created network interface: {}".format(network_interface_id_2))
        except botocore.exceptions.ClientError as e:
            log("Error creating network interface: {}".format(e.response['Error']))

    return network_interface_id_2


def attach_interface(network_interface_id, instance_id):
    attachment = None

    if network_interface_id and instance_id:
        try:
            attach_interface = ec2_client.attach_network_interface(
                NetworkInterfaceId=network_interface_id,
                InstanceId=instance_id,
                DeviceIndex=1
            )
            attachment = attach_interface['AttachmentId']
            log("Created network attachment: {}".format(attachment))
        except botocore.exceptions.ClientError as e:
            log("Error attaching network interface: {}".format(e.response['Error']))

    return attachment
    
def attach_interface_2(network_interface_id, instance_id):
    attachment = None

    if network_interface_id and instance_id:
        try:
            attach_interface = ec2_client.attach_network_interface(
                NetworkInterfaceId=network_interface_id,
                InstanceId=instance_id,
                DeviceIndex=2
            )
            attachment = attach_interface['AttachmentId']
            log("Created network attachment: {}".format(attachment))
        except botocore.exceptions.ClientError as e:
            log("Error attaching network interface: {}".format(e.response['Error']))

    return attachment


def delete_interface(network_interface_id):
    try:
        ec2_client.delete_network_interface(
            NetworkInterfaceId=network_interface_id
        )
        log("Deleted network interface: {}".format(network_interface_id))
        return True

    except botocore.exceptions.ClientError as e:
        log("Error deleting interface {}: {}".format(network_interface_id, e.response['Error']))
        
        
def complete_lifecycle_action_success(hookname,groupname,instance_id):
    try:
        asg_client.complete_lifecycle_action(
                LifecycleHookName=hookname,
                AutoScalingGroupName=groupname,
                InstanceId=instance_id,
                LifecycleActionResult='CONTINUE'
            )
        log("Lifecycle hook CONTINUEd for: {}".format(instance_id))
    except botocore.exceptions.ClientError as e:
            log("Error completing life cycle hook for instance {}: {}".format(instance_id, e.response['Error']))
            log('{"Error": "1"}')    
            
def complete_lifecycle_action_failure(hookname,groupname,instance_id):
    try:
        asg_client.complete_lifecycle_action(
                LifecycleHookName=hookname,
                AutoScalingGroupName=groupname,
                InstanceId=instance_id,
                LifecycleActionResult='ABANDON'
            )
        log("Lifecycle hook ABANDONed for: {}".format(instance_id))
    except botocore.exceptions.ClientError as e:
            log("Error completing life cycle hook for instance {}: {}".format(instance_id, e.response['Error']))
            log('{"Error": "1"}')    
    

def log(error):
    print('{}Z {}'.format(datetime.utcnow().isoformat(), error))


Now to create a lifecycle hook to trigger your event: Go to EC2 screen, then to ASG, and find the ASG you created earlier Click on the Lifecycle Hook menu (far right)

[screenshot of lifecyce hook selection]


Click create Lifecycle Hook Give it a name, and leave defaults for rest as shown in screenshot

[lifecycle hook edit screenshot]


Click the link that appears to go into Cloudwatch events rules, this is where you’ll define what triggers the lifecycle hook. If no link appears, go into Cloudwatch, and browse to Rules, under Events [Screenshot of Cloudwatch rule creation]

















Go to rules, and click on create rule. Select Autoscaling for the service name, and choose “Instance Launch and Terminate” and then select Specific Instance Events and then “EC2 Instance-Launch Lifecycle Action” as shown in the screenshot. Choose specific group name and choose the name of your ASG, so this rule only applies now to launches of instances in your ASG. Under Targets, Select Lambda Function, and choose the Lambda function created earlier. Click Configure details in bottom right. Give the rule a name, description and make sure it is enabled. You now have everything configured to launch your function when the ASG successfully launches a new instance To avoid exhausting private IP addresses in the subnet and reaching the ENI limit in your account, you should write code that deletes the second network interface when Auto Scaling terminates the instance: You can use a Termination lifecycle hook, and then trigger a Lambda function through CloudWatch Events or Amazon Simple Notification Service (Amazon SNS) to detach and delete the interface with DeviceIndex > 0. You can also write a boto script that periodically deletes the unused interfaces in your account, or you can modify the 'delete on termination' attribute of the network interface using the modify-network-interface-attribute AWS CLI command. You can also use SNS instead of CloudWatch Events to invoke your Lambda function by configuring --notification-target-arn when creating the hook.

  • Twitter Social Icon
  • LinkedIn Social Icon

©2019 by Squanto Tech.

Follow us on