(DEV301) Automating AWS with the AWS CLI

2. AWS Command Line Interface Unified tool to manage your AWS services

3. $ git show 1.0.0 tag 1.0.0 Tagger: James Saryerwinnie <js@jamesls.com> Date: Mon Sep 2 18:38:51 2013 -0700 Tagging 1.0.0 release. commit 7fbdac0b19ea9dcc0380242068af20ef425ac5c3 Merge: 6f76721 469bab6 Author: James Saryerwinnie <js@jamesls.com> Date: Mon Sep 2 18:38:51 2013 -0700 Merge branch 'release-1.0.0' * release-1.0.0: Bumping version to 1.0.0

4. Interactive

5. Shell scripts#!/bin/bash import_key_pair() { echo -n "Would you like to import ~/.ssh/id_rsa.pub? [y/N]: " read confirmation if [[ "$confirmation" != "y" ]] then return fi aws ec2 import-key-pair --key-name id_rsa --public-key-material file://~/.ssh/id_rsa.pub } create_instance_profile() { echo -n "Would you like to create an IAM instance profile? [y/N]: " read confirmation if [[ "$confirmation" != "y" ]]; then return fi aws iam create-role --role-name dev-ec2-instance --assume-role-policy-document "$TRUST_POLICY" || errexit "Could not create Role" # Use a managed policy policies=$(aws iam list-policies --scope AWS) admin_policy_arn=$(jp -u "Policies[?PolicyName=='AdministratorAccess'].Arn | [0]" <<< "$policies") aws iam attach-role-policy --role-name dev-ec2-instance --policy-arn "$admin_policy_arn" || errexit "Could not attach role policy" # Then we need to create an instance profile from the role. aws iam create-instance-profile --instance-profile-name dev-ec2-instance || errexit "Could not create instance profile." # And add it to the role aws iam add-role-to-instance-profile --role-name dev-ec2-instance --instance-profile-name dev-ec2-instance || errexit "Could not add role to instance profile." } compute_key_fingerprint() { # Computes the fingerprint of a public SSH key given a private # RSA key. This can be used to compare against the output given # from aws ec2 describe-key-pair. openssl pkey -in ~/.ssh/id_rsa -pubout -outform DER | openssl md5 -c | cut -d = -f 2 | tr -d '[:space:]' } do_setup() { echo "Checking for required resources..." echo "" # 1. Check if a security group is found for # both windows and non-windows tags. # If not, we'll eventually give the option to # configure this. if resource_exists "aws ec2 describe-security-groups --filter Name=tag:dev-ec2-instance,Values=non-windows"; then echo "Security groups exists." else echo "Security group not found." fi echo "" # 2. Make sure the keypair is imported. if [[ ! -f ~/.ssh/id_rsa ]]; then echo "Missing ~/.ssh/id_rsa key pair." elif has_new_enough_openssl; then fingerprint=$(compute_key_fingerprint ~/.ssh/id_rsa) if resource_exists "aws ec2 describe-key-pairs --filter Name=fingerprint,Values=$fingerprint"; then echo "Key pair exists." else echo "~/.ssh/id_rsa key pair does not appear to be imported." import_key_pair fi else echo "Can't check if SSH key has been imported." echo "You need at least openssl 1.0.0 that has a "pkey" command." echo "Please upgrade your version of openssl." fi echo "" # 3. Check that they have an IAM role called dev-ec2-instance. # There is no server side filter for this like we have with the EC2 # APIs, so we have to manually use a --query option for us. if resource_exists "aws iam list-instance-profiles" "length(InstanceProfiles[?InstanceProfileName=='dev-ec2-instance'])"; then echo "Instance profile exists." else echo "Missing IAM instance profile 'dev-ec2-instance'" create_instance_profile fi echo "" } do_setup

6. aws s3 ls Application with an AWS SDK

7. aws s3 ls Application with an AWS SDK Shell Scripts – what we’ll cover

8. A good shell script • < 100 SLOC • Commands in sequence • Piping input/output • Simple domain logic

9. • Familiar with AWS CLI • Familiar with bash • Familiar with AWS Prerequisites

10. $ aws ec2 describe-instances --filters Name=architecture,Values=x86_64 $ du $(ls | grep cli) | sort -nr | tail -2 Amazon S3Amazon EC2 Prerequisites

11. Getting up to speed • Becoming a Command Line Expert with the AWS CLI • Advanced Usage of the AWS CLI • Installation • Configuration

12. $ We will be in the terminal…

13. $ We will be in the terminal…a lot

14. Common patterns

15. Main patterns • Single output to a single command • Map list output to N AWS CLI commands • Storing JSON documents and querying later • Resource exists check

16. Main patterns • Single output to a single command • Map list output to N AWS CLI commands • Storing JSON documents and querying later • Resource exists check

17. Output to a single parameter

18. Output to a single parameter $ aws ec2 describe… { "Reservations": [ { "Groups": [ { "GroupId": "sg-abc", "GroupName": "Windows" } ], "Instances": [ { "AmiLaunchIndex": 0, "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2015-01-29T21:22:17.000Z", "DeleteOnTermination": true, "Status": "attached", $ aws ec2 run… --foo <here>

19. Output to a single parameter aws ec2 create-tags --tags Key=purpose,Value=dev --resources $(aws ec2 run-instances --image-id ami-12345 --query Instances[].InstanceId --output text)

20. Output to a single parameter aws ec2 create-tags --tags Key=purpose,Value=dev --resources $(aws ec2 run-instances --image-id ami-12345 --query Instances[].InstanceId --output text) Error handling

21. Output to a single parameter instance_ids=$(aws ec2 run-instances --image-id ami-12345 --query Instances[].InstanceId --output text) || errexit "Could not run instance" aws ec2 create-tags --tags Key=purpose,Value=dev --resources "$(instance_ids)" || errexit "<errmsg>"

24. Output to a single parameter • Used to map one part of the output to one part of the input • Use --query and --output text • Make sure to handle the error case before using the value

25. Main Patterns • Single output to a single command • Map list output to N AWS CLI commands • Storing JSON documents and querying later • Resource exists check

26. Mapping one output to N AWS calls

27. Mapping one output to N AWS calls $ aws iam list… [ "a", "b", ”c", "d", "e" ] $ aws iam … --foo <here> $ aws iam … --foo <here> $ aws iam … --foo <here> $ aws iam … --foo <here> $ aws iam … --foo <here>

28. Mapping one output to N AWS calls for name in $(aws iam list-users --query "Users[].[UserName]" --output text); do aws iam delete-user --user-name "$name" done

29. Mapping one output to N AWS calls for name in $(aws iam list-users --query "Users[].[UserName]" --output text); do aws iam delete-user --user-name "$name" done User1 User2 User3 User4 User5

30. Mapping one output to N AWS calls for name in $(aws iam list-users --query "Users[].[UserName]" --output text); do aws iam delete-user --user-name "$name" done

31. Mapping one output to N AWS calls aws iam list-users --query "Users[].[UserName]" --output text | xargs -I {} -P 10 aws iam delete-user --user-name "{}"

32. Mapping one output to N AWS calls aws iam list-users --query "Users[].[UserName]" --output text | xargs -I {} -P 10 aws iam delete-user --user-name "{}"

33. Mapping one output to N AWS calls aws iam list-users --query "Users[].[UserName]" --output text | xargs -I {} -P 10 aws iam delete-user --user-name "{}" Parallel

34. Mapping one output to N AWS calls -+= 72730 james -bash -+- 76343 james xargs -I {} –P 9 aws iam delete-user --user-name {} |--- 76348 james aws iam delete-user --user-name user1 |--- 76349 james aws iam delete-user --user-name user2 |--- 76350 james aws iam delete-user --user-name user3 |--- 76351 james aws iam delete-user --user-name user4 |--- 76352 james aws iam delete-user --user-name user5 |--- 76353 james aws iam delete-user --user-name user6 |--- 76354 james aws iam delete-user --user-name user7 |--- 76355 james aws iam delete-user --user-name user8 --- 76356 james aws iam delete-user --user-name user9

35. Mapping one output to N AWS calls • Use a for loop for functions • If you’re using xargs, use -I {} • Use xargs –P N parallel execution • Use “.[UserName]” instead of “.UserName” to get newline separated output.. • This works because nothing is written to stdout on error

36. Demo

37. Main patterns  Single output to a single command  Map list output to N AWS CLI commands • Storing JSON documents and querying later • Resource exists check

38. Saving the entire output for querying later { "AmiLaunchIndex": 0, "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2015-01-29T21:22:17.000Z", "DeleteOnTermination": true, "Status": "attached", "VolumeId": "vol-12345" } }, { "DeviceName": "xvdg", "Ebs": { "AttachTime": "2015-09-17T19:14:27.000Z", "DeleteOnTermination": false, "Status": "attached", "VolumeId": "vol-12345" } } ], "ClientToken": "12345", "EbsOptimized": false, "Hypervisor": "xen", "IamInstanceProfile": { "Arn": "arn:aws:iam::12345", "Id": "12345" }, "ImageId": "ami-12345", "InstanceId": "i-12345", "InstanceType": "m3.large", "KeyName": "id_rsa", "LaunchTime": "2015-09-17T17:24:13.000Z", "Monitoring": { "State": "disabled" }, } instance =

39. Saving the entire output for querying later { "AmiLaunchIndex": 0, "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2015-01-29T21:22:17.000Z", "DeleteOnTermination": true, "Status": "attached", "VolumeId": "vol-12345" } }, { "DeviceName": "xvdg", "Ebs": { "AttachTime": "2015-09-17T19:14:27.000Z", "DeleteOnTermination": false, "Status": "attached", "VolumeId": "vol-12345" } } ], "ClientToken": "12345", "EbsOptimized": false, "Hypervisor": "xen", "IamInstanceProfile": { "Arn": "arn:aws:iam::12345", "Id": "12345" }, "ImageId": "ami-12345", "InstanceId": "i-12345", "InstanceType": "m3.large", "KeyName": "id_rsa", "LaunchTime": "2015-09-17T17:24:13.000Z", "Monitoring": { "State": "disabled" }, } instance = query $instance BlockDeviceMappings[0].VolumeId

40. Saving the entire output for querying later { "AmiLaunchIndex": 0, "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2015-01-29T21:22:17.000Z", "DeleteOnTermination": true, "Status": "attached", "VolumeId": "vol-12345" } }, { "DeviceName": "xvdg", "Ebs": { "AttachTime": "2015-09-17T19:14:27.000Z", "DeleteOnTermination": false, "Status": "attached", "VolumeId": "vol-12345" } } ], "ClientToken": "12345", "EbsOptimized": false, "Hypervisor": "xen", "IamInstanceProfile": { "Arn": "arn:aws:iam::12345", "Id": "12345" }, "ImageId": "ami-12345", "InstanceId": "i-12345", "InstanceType": "m3.large", "KeyName": "id_rsa", "LaunchTime": "2015-09-17T17:24:13.000Z", "Monitoring": { "State": "disabled" }, } instance = query $instance ImageId query $instance BlockDeviceMappings[0].VolumeId

41. Saving the entire output for querying later { "AmiLaunchIndex": 0, "Architecture": "x86_64", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "AttachTime": "2015-01-29T21:22:17.000Z", "DeleteOnTermination": true, "Status": "attached", "VolumeId": "vol-12345" } }, { "DeviceName": "xvdg", "Ebs": { "AttachTime": "2015-09-17T19:14:27.000Z", "DeleteOnTermination": false, "Status": "attached", "VolumeId": "vol-12345" } } ], "ClientToken": "12345", "EbsOptimized": false, "Hypervisor": "xen", "IamInstanceProfile": { "Arn": "arn:aws:iam::12345", "Id": "12345" }, "ImageId": "ami-12345", "InstanceId": "i-12345", "InstanceType": "m3.large", "KeyName": "id_rsa", "LaunchTime": "2015-09-17T17:24:13.000Z", "Monitoring": { "State": "disabled" }, } instance = query $instance InstanceId query $instance ImageId query $instance BlockDeviceMappings[0].VolumeId

42. jp – The JMESPath CLI • Same language as --query • Cross platform executable • https://github.com/jmespath/jp

43. jp – The JMESPath CLI • Same language as --query • Cross platform executable • https://github.com/jmespath/jp Mac OS X brew tap jmespath/jmespath brew install jp

44. Saving the entire output for querying later instance=$(aws run-instance --image-id ami-1234 --query Instances[0]) query() { jp -u "$2" <<<"$1" } instance_id=$(query "$instance" InstanceId) state_name=$(query "$instance" State.Name) instance_name=$(query "$instance" "Tags[?Key=='Name'].Value | [0]")

45. Saving the entire output for querying later instance=$(aws run-instance --image-id ami-1234 --query Instances[0]) query() { jp -u "$2" <<<"$1" } instance_id=$(query "$instance" InstanceId) state_name=$(query "$instance" State.Name) instance_name=$(query "$instance" "Tags[?Key=='Name'].Value | [0]") instance= {"InstanceId": ”i-12345", ... }

49. Saving the entire output for querying later 1. Use JSON output, not text, --output json 2. Store the entire thing as a variable, or write to a temp file if output size is a concern. 3. Use the “jp” command to filter down results as needed.

50. Main Patterns  Single output to a single command  Map list output to N AWS CLI commands  Storing JSON documents and querying later • Resource exists check

51. resource_exists() { } 1. Determine command to run with query 2. Run command and check errors 3. Check length of query result

52. resource_exists() { local cmd local num_matches if [[ -z "$2" ]]; then cmd="$1 --query length(*[0])" else cmd="$1 --query $2" fi } 2. Run command and check errors 3. Check length of query result

53. resource_exists() { num_matches=$($command) if [[ "$?" -ne 0 ]]; then echo "Could not check if resource exists, exiting." exit 2 fi } 3. Check length of query result 1. Determine command to run with query

54. resource_exists() { if [[ "$num_matches" -gt 0 ]]; then # RC of 0 mean the resource exists, success. return 0 else return 1 fi } 1. Determine command to run with query 2. Run command and check errors

55. Usage: Server-side filtering describe_groups="aws ec2 describe-security-groups" if resource_exists "$describe_groups --filter Name=tag:dev-ec2-instance,Values=linux"; then echo "Security groups exists." else echo "Security group not found." fi

56. Usage: Client-side filtering list_profiles="aws iam list-instance-profiles" if resource_exists "$list_profiles" "InstanceProfiles[?InstanceProfileName=='dev-ec2-instance']" then echo "Instance profile exists." else echo "Missing IAM instance profile 'dev-ec2-instance'" create_instance_profile fi

57. Resource exists • Check if a resource exists by filtering down a list and checking if the length is greater than 0. • Use server side filtering when possible • Use JMESPath query for client side filtering

58. Demo

59. Common Patterns  Single output to a single command  Map list output to N AWS CLI commands  Storing JSON documents and querying later  Resource exists check

60. Deployment tips

61. # * * * * * command to execute # │ │ │ │ │ # │ │ │ │ │ # │ │ │ │ └───── day of week (0 – 6) # │ │ │ └────────── month (1 - 12) # │ │ └─────────────── day of month (1 - 31) # │ └──────────────────── hour (0 - 23) # └───────────────────────── min (0 - 59) Cron

62. $ whoami $HOME/.aws/config $HOME/.aws/credentials export AWS_CONFIG_FILE export AWS_SHARED_CREDENTIALS_FILE

63. $ whoami $HOME/.aws/config $HOME/.aws/credentials export AWS_CONFIG_FILE export AWS_SHARED_CREDENTIALS_FILE • Explicitly set AWS_CONFIG_FILE/AWS_SHARED_CREDENTIALS_FILE • Ensure user running cron job has ~/.aws/config ~/.aws/credentials

64. aws configure set region $(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jp -u 'region') Configure a region

65. Summary • Common patterns • Shell script examples • Deployment • Tools

66. Remember to complete your evaluations!

67. Thank you! • User guide: http://docs.aws.amazon.com/cli/latest/userguide/ • Reference docs: http://docs.aws.amazon.com/cli/latest/reference/ • JMESPath: http://jmespath.org/ • re:Invent 2014 - https://www.youtube.com/watch?v=vP56l7qThNs • re:Invent 2013 - https://www.youtube.com/watch?v=qiPt1NoyZm0

(DEV301) Automating AWS with the AWS CLI

Amazon Web Services

(DEV301) Automating AWS with the AWS CLI

A particular slide catching your eye?