【awslabs探訪】ami-builder-packerのご紹介

クラウドでコスト削減はこちら

はじめに

中山（順）です

みなさん、GitHub上にawslabsってOrganizationがあるのはご存知でしょうか？ ここにはAWSから提供される様々な便利ツールやサンプルコードがあり、お世話になる機会が多いです。

今回は最近使ったリポジトリをご紹介します。 （好評だったらシリーズ化と思うので、フィードバックをお待ちしております）

本日は、ami-builder-packerをご紹介します。

これ何？

AMIの更新を自動化する仕組みのサンプルです。

どんなことができるのかを一言で説明すると、 Packer Template / Ansible Playbook / CodeBuild Build SpecificationをCodeCommitのリポジトリにプッシュするとそれをトリガーにAMIを作成するというCode Pipelineのパイプライン一式を作ってくれるCloudFormationテンプレートです。（長い）

全体像は以下の構成図を見てください。

参照元 : https://github.com/awslabs/ami-builder-packer/blob/master/images/ami-builder-diagram.png

やってみた

コマンドラインで一連の作業を行います。

ami-builder-packerのリポジトリをクローン

まずはリポジトリをクローンします。

リポジトリには、おおまかに以下のものが含まれています。

  • Ansible Playbook
    • /ansible
  • Packer Template
    • packer_cis.json
  • Build Specification
    • buildspec.yml
  • CloudFormation Template
    • /cloudformation/pipeline.yaml
  • Other
1
2
3
git clone https://github.com/awslabs/ami-builder-packer.git
cd ami-builder-packer
ls -R
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
.:
LICENSE README.md ami_builder_event.json ansible buildspec.yml cloudformation images packer_cis.json
 
./ansible:
playbook.yaml requirements.yaml roles
 
./ansible/roles:
common
 
./ansible/roles/common:
tasks
 
./ansible/roles/common/tasks:
main.yaml
 
./cloudformation:
pipeline.yaml
 
./images:
ami-builder-diagram.png ami-builder-pipeline.png create_policy.png deploy-to-aws.png
ami-builder-initial-pipeline.png build_phases.png create_project.png new_build.png

Pipelineを作成

CFnテンプレートを確認してみます。（内容は割愛）

1
cat cloudformation/pipeline.yaml

作成されるリソースは以下のとおりです。

  • CodeRepository(CodeCommit Repository)
    • Ansible Playbook, Packer Template, Build Specificationの保存先
  • BuildArtifactsBucket(S3 Bucket)
    • CodeRepositorのリソースを後続の処理に引き渡す際に利用するバケット
  • CodeBuildServiceRole(IAM Role)
    • CodeBuild用のサービスロール
    • 以下の処理を実行する権限を付与
    • CloudWatch Logsへのログ出力
    • BuildArtifactsBucketへの読み書き
    • PowerUserAccess（EC2インスタンスの作成、AMI作成など）
  • CodeBuildProject
    • Build Specificationに基づく処理を実行（詳細は後述）
  • PipelineExecutionRole(IAM Role)
    • CodeBuild用のサービスロール
    • 以下の処理を実行する権限を付与
    • BuildArtifactsBucketへの読み書き
    • CodeBuildの実行
    • CodeRepositoryへのアクセス
  • Pipeline(CodePipeline Pipeline)
    • AMIの作成を実行するパイプラインを定義
    • BuildArtifactsBucketにアーティファクトを保存
    • 以下の処理を実行
    • CodeRepositoryの変更をトリガーにリソース (Ansible Playbook, Packer Template, Build Specification) を取得
    • CodeBuildでビルド (packer build)
  • AmiBuilderNotificationTopic
    • ユーザーにメール通知するためのSNS Topic
  • AmiBuilderCustomEvent
    • AMIの作成を契機として、AmiBuilderNotificationTopicへ通知
  • AmiBuilderNotificationTopicPolicy
    • AmiBuilderCustomEventがAmiBuilderNotificationTopicに対して通知を行えるようにするためのSNS Policy

その際に利用するパラメーターとして以下の値を指定できます。

  • ServiceName
  • CodeBuildEnvironment
    • ビルドを実行する際に利用するDocker Imageを指定します。
    • 選択できるImageのうち、AWS公式のイメージは以下のとおりです。
    • Docker Images Provided by AWS CodeBuild
  • BuilderVPC
    • packerが作成するEC2インスタンスの配置先VPC
  • BuilderPublicSubnet
    • packerが作成するEC2インスタンスの配置先サブネット
  • NotificationEmailAddress
    • AMI作成の成功時の通知先となるメールアドレス

一応、テンプレートを検証してみましょう。

1
2
aws cloudformation validate-template \
    --template-body file://cloudformation/pipeline.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
{
  "CapabilitiesReason": "The following resource(s) require capabilities: [AWS::IAM::Role]",
  "Capabilities": [
    "CAPABILITY_IAM"
  ],
  "Parameters": [
    {
      "DefaultValue": "AMI-Builder",
      "NoEcho": false,
      "Description": "Name for this service; used in the code repository and pipeline names",
      "ParameterKey": "ServiceName"
    },
    {
      "NoEcho": false,
      "Description": "VPC ID that AMI Builder will use to launch temporary resource",
      "ParameterKey": "BuilderVPC"
    },
    {
      "NoEcho": false,
      "Description": "Public Subnet ID that AMI Builder will use to launch temporary resource",
      "ParameterKey": "BuilderPublicSubnet"
    },
    {
      "NoEcho": false,
      "Description": "Email to receive new AMI ID created by AMI Builder",
      "ParameterKey": "NotificationEmailAddress"
    },
    {
      "DefaultValue": "eb-python-2.6-amazonlinux-64:2.1.6",
      "NoEcho": false,
      "Description": "Docker image to use for CodeBuild container - Use http://amzn.to/2mjCI91 for reference",
      "ParameterKey": "CodeBuildEnvironment"
    }
  ]
}

それでは、CloudFormation Stackを作成してみましょう。

パラメーターについては、デフォルト値が設定されていない"BuilderVPC", "BuilderPublicSubnet", "NotificationEmailAddress"のみ指定します。 とりあえずデフォルトVPCを使います。 （JSONファイルを作成したらjsonlintなどで検証しましょう）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
PARAMETERS_FILE_NAME="parameters.json"
 
cat << EOF > ${PARAMETERS_FILE_NAME}
[
  {
    "ParameterKey": "BuilderVPC",
    "ParameterValue": "vpc-xxxxxxxx"
  },
  {
    "ParameterKey": "BuilderPublicSubnet",
    "ParameterValue": "subnet-xxxxxxxx"
  },
  {
    "ParameterKey": "NotificationEmailAddress",
    "ParameterValue": "xxxxxxxx@example.com"
  }
]
EOF
 
jsonlint -q ${PARAMETERS_FILE_NAME}

パラメーターを用意したら、CloudFormation Stackを作成します。

1
2
3
4
5
aws cloudformation create-stack \
    --stack-name classmethod-ami-builder-packer \
    --template-body file://cloudformation/pipeline.yaml \
    --parameters file://${PARAMETERS_FILE_NAME} \
    --capabilities CAPABILITY_IAM
1
2
3
{
  "StackId": "arn:aws:cloudformation:ap-northeast-1:XXXXXXXXXXXX:stack/classmethod-ami-builder-packer/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

スタックが正常に作成されたことを確認します。

1
2
3
aws cloudformation describe-stacks \
    --stack-name classmethod-ami-builder-packer \
    --query "Stacks[0].StackStatus"
1
"CREATE_COMPLETE"

正常に作成されたらスタックに含まれるリソースを確認してみましょう。

1
2
3
aws cloudformation describe-stack-resources \
    --stack-name classmethod-ami-builder-packer \
    --query "StackResources[*].{LogicalId: LogicalResourceId,Type: ResourceType,PhysicalId: PhysicalResourceId}"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
[
  {
    "PhysicalId": "classmethod-ami-builder-pack-AmiBuilderCustomEvent-1U41B80EUIBJZ",
    "LogicalId": "AmiBuilderCustomEvent",
    "Type": "AWS::Events::Rule"
  },
  {
    "PhysicalId": "arn:aws:sns:ap-northeast-1:XXXXXXXXXXXX:AmiBuilder-Notify",
    "LogicalId": "AmiBuilderNotificationTopic",
    "Type": "AWS::SNS::Topic"
  },
  {
    "PhysicalId": "classmethod-ami-builder-packer-AmiBuilderNotificationTopicPolicy-1Y0Q6VHTRZMUR",
    "LogicalId": "AmiBuilderNotificationTopicPolicy",
    "Type": "AWS::SNS::TopicPolicy"
  },
  {
    "PhysicalId": "classmethod-ami-builder-pack-buildartifactsbucket-1eh9z8jlwbih6",
    "LogicalId": "BuildArtifactsBucket",
    "Type": "AWS::S3::Bucket"
  },
  {
    "PhysicalId": "AMI-Builder_build",
    "LogicalId": "CodeBuildProject",
    "Type": "AWS::CodeBuild::Project"
  },
  {
    "PhysicalId": "classmethod-ami-builder-packe-CodeBuildServiceRole-7TF5PSDB2S4Q",
    "LogicalId": "CodeBuildServiceRole",
    "Type": "AWS::IAM::Role"
  },
  {
    "PhysicalId": "29ffb514-1cf2-4f52-a9c4-2e73efba0981",
    "LogicalId": "CodeRepository",
    "Type": "AWS::CodeCommit::Repository"
  },
  {
    "PhysicalId": "AMI-Builder_pipeline",
    "LogicalId": "Pipeline",
    "Type": "AWS::CodePipeline::Pipeline"
  },
  {
    "PhysicalId": "classmethod-ami-builder-pack-PipelineExecutionRole-1XYFKAG47HIHC",
    "LogicalId": "PipelineExecutionRole",
    "Type": "AWS::IAM::Role"
  }
]

作成した（空の）リポジトリーをクローン

まず、リポジトリのURLを確認します。 説明を割愛しましたが、CloudFormationテンプレートでリポジトリのURLを出力するようになっているので、Outputを確認します。

1
2
3
4
5
repo_url=$(aws cloudformation describe-stacks \
    --stack-name classmethod-ami-builder-packer \
    --query "Stacks[0].Outputs[?OutputKey=='GitRepository'].OutputValue" \
    --output text) \
    && echo ${repo_url}

リポジトリをクローンします。 この際、認証情報が必要になりますので、事前にご用意ください。

Setup for HTTPS Users Using Git Credentials

1
2
3
4
cd ~
mkdir codecommit
cd codecommit
git clone ${repo_url}
1
2
3
4
5
Cloning into 'AMI-Builder_repo'...
Username for 'https://git-codecommit.ap-northeast-1.amazonaws.com': (user name)
Password for 'https://(user name)@git-codecommit.ap-northeast-1.amazonaws.com':
warning: You appear to have cloned an empty repository.
Checking connectivity... done.

以降、このディレクトリ配下で作業を行います。
リポジトリ名はCloudFormation Stackの作成時にパラメーターとして指定した"ServiceName"で決まるようになっています。
適宜読み替えてください。

1
cd AMI-Builder_repo

CodeBuild spec

ami-builder-packerから取得したBuild Specificationをコピーします。
（コピー元は適宜読み替えてください）

1
cp ~/awslabs/ami-builder-packer/buildspec.yml buildspec.yml

このBuildSpecificationでは、CodeBuildのビルドで以下のことを実行するように定義されています。

  • 前処理
    • Packerのダウンロード
  • PackerによるAMIの作成 (packer build)
  • 後処理
    • 結果の通知（CloudWatch Eventsに対するイベントの発行）

Packer template

ami-builder-packerから取得したPacker Templateをコピーします。
（コピー元は適宜読み替えてください）

もちろん、ami-builder-packerを応用するにあたって、独自に作ったPacker Templateに置き換えても問題ありません。

1
cp ~/awslabs/ami-builder-packer/packer_cis.json packer_cis.json

packerで行っていることの説明は割愛します。
機能の詳細は公式のドキュメントなどをご確認ください。

Ansible playbook

ami-builder-packerから取得したAnsible Playbookをコピーします。 （コピー元は適宜読み替えてください）

もちろん、ami-builder-packerを応用するにあたって、独自に作ったAnsible Playbookに置き換えても問題ありません。

1
2
mkdir ansible
cp -r ~/awslabs/ami-builder-packer/ansible/* ansible

Ansibleで行っていることの説明は割愛します。 機能の詳細は公式のドキュメントなどをご確認ください。

Eventsへの通知用テンプレート

CloudWatch Eventsに対してput-eventsを実行する際のテンプレートをコピーします。 （Build Specification内部で利用しています）

1
cp ~/awslabs/ami-builder-packer/ami_builder_event.json ami_builder_event.json

master branchへのpush

必要なリソースが揃ったら、リポジトリにプッシュします。 （今回作成されたパイプラインは、masterブランチの変更を契機に処理が開始されますのでdevブランチからのマージなどでも動き始めるはずです）

1
2
git add .
git commit -m "Developers.IO"
1
2
3
4
5
6
7
[master (root-commit) 03619b4] Developers.IO
5 files changed, 129 insertions(+)
create mode 100644 ansible/playbook.yaml
create mode 100644 ansible/requirements.yaml
create mode 100644 ansible/roles/common/tasks/main.yaml
create mode 100644 buildspec.yml
create mode 100644 packer_cis.json
1
git push origin master
1
2
3
4
5
6
7
8
9
Username for 'https://git-codecommit.ap-northeast-1.amazonaws.com': (user name)
Password for 'https://(user name)@git-codecommit.ap-northeast-1.amazonaws.com':
Counting objects: 11, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (11/11), 2.47 KiB | 0 bytes/s, done.
Total 11 (delta 0), reused 0 (delta 0)
To https://git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/AMI-Builder_repo
* [new branch] master -> master

結果の確認

パイプラインで直近に実行された処理のIDを確認します。 （パイプライン名は適宜読み替えてください）

1
2
3
aws codepipeline list-pipeline-executions \
    --pipeline-name AMI-Builder_pipeline \
    --query "sort_by(pipelineExecutionSummaries,&startTime)[-1].{ID:pipelineExecutionId,Status:status}"
1
2
3
4
{
  "Status": "Failed",
  "ID": "843f006f-5f45-41c1-8577-108172208f85"
}

・・・あれ？失敗してる。。。orz

ログを確認してみたいと思います。

CodeBuildのログはCloudWatch Logsに出力されます。 まず、ロググループ名を確認します。

1
2
aws logs describe-log-groups \
    --log-group-name-prefix /aws/codebuild/AMI-Builder
1
2
3
4
5
6
7
8
9
10
11
{
  "logGroups": [
    {
      "arn": "arn:aws:logs:ap-northeast-1:XXXXXXXXXXXX:log-group:/aws/codebuild/AMI-Builder_build:*",
      "creationTime": 1514014343796,
      "metricFilterCount": 0,
      "logGroupName": "/aws/codebuild/AMI-Builder_build",
      "storedBytes": 0
    }
  ]
}

ログストリーム名を確認します。

1
2
aws logs describe-log-streams \
    --log-group-name /aws/codebuild/AMI-Builder_build
1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
  "logStreams": [
    {
      "firstEventTimestamp": 1514014391000,
      "lastEventTimestamp": 1514014607000,
      "creationTime": 1514014343834,
      "uploadSequenceToken": "49579861923906215366483654023972723455696922525147614674",
      "logStreamName": "040c71e5-8a13-4f95-a3d6-75150b7106bd",
      "lastIngestionTime": 1514014607098,
      "arn": "arn:aws:logs:ap-northeast-1:XXXXXXXXXXXX:log-group:/aws/codebuild/AMI-Builder_build:log-stream:040c71e5-8a13-4f95-a3d6-75150b7106bd",
      "storedBytes": 0
    }
  ]
}

ロググループ名をログストリーム名がわかったので、ログを確認します。

1
2
3
4
5
aws logs get-log-events \
    --log-group-name /aws/codebuild/AMI-Builder_build \
    --log-stream-name 040c71e5-8a13-4f95-a3d6-75150b7106bd \
    --limit 10000 \
    --query "sort_by(events,&timestamp)[].message"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
[
"[Container] 2017/12/23 07:33:06 Waiting for agent ping\n",
"[Container] 2017/12/23 07:33:07 Waiting for DOWNLOAD_SOURCE\n",
"[Container] 2017/12/23 07:33:11 Phase is DOWNLOAD_SOURCE\n",
"[Container] 2017/12/23 07:33:11 CODEBUILD_SRC_DIR=/codebuild/output/src568755513/src\n",
"[Container] 2017/12/23 07:33:11 YAML location is /codebuild/output/src568755513/src/buildspec.yml\n",
"[Container] 2017/12/23 07:33:11 Processing environment variables\n",
"[Container] 2017/12/23 07:33:11 Moving to directory /codebuild/output/src568755513/src\n",
"[Container] 2017/12/23 07:33:11 Registering with agent\n",
"[Container] 2017/12/23 07:33:11 Phases found in YAML: 3\n",
"[Container] 2017/12/23 07:33:11 POST_BUILD: 5 commands\n",
"[Container] 2017/12/23 07:33:11 PRE_BUILD: 4 commands\n",
"[Container] 2017/12/23 07:33:11 BUILD: 1 commands\n",
"[Container] 2017/12/23 07:33:11 Phase complete: DOWNLOAD_SOURCE Success: true\n",
"[Container] 2017/12/23 07:33:11 Phase context status code: Message: \n",
"[Container] 2017/12/23 07:33:11 Entering phase INSTALL\n",
"[Container] 2017/12/23 07:33:11 Phase complete: INSTALL Success: true\n",
"[Container] 2017/12/23 07:33:11 Phase context status code: Message: \n",
"[Container] 2017/12/23 07:33:11 Entering phase PRE_BUILD\n",
"[Container] 2017/12/23 07:33:11 Running command echo \"Installing Packer\"\n",
"Installing Packer\n",
"\n",
"[Container] 2017/12/23 07:33:11 Running command curl -o packer.zip https://releases.hashicorp.com/packer/1.0.3/packer_1.0.3_linux_amd64.zip && unzip packer.zip\n",
" % Total % Received % Xferd Average Speed Time Time Time Current\n",
" Dload Upload Total Spent Left Speed\n",
"\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r 1 11.5M 1 125k 0 0 433k 0 0:00:27 --:--:-- 0:00:27 432k\r100 11.5M 100 11.5M 0 0 21.3M 0 --:--:-- --:--:-- --:--:-- 21.3M\n",
"Archive: packer.zip\n",
" inflating: packer \n",
"\n",
"[Container] 2017/12/23 07:33:12 Running command echo \"Validating Packer template\"\n",
"Validating Packer template\n",
"\n",
"[Container] 2017/12/23 07:33:12 Running command ./packer validate packer_cis.json\n",
"Template validated successfully.\n",
"\n",
"[Container] 2017/12/23 07:33:13 Phase complete: PRE_BUILD Success: true\n",
"[Container] 2017/12/23 07:33:13 Phase context status code: Message: \n",
"[Container] 2017/12/23 07:33:13 Entering phase BUILD\n",
"[Container] 2017/12/23 07:33:13 Running command ./packer build -color=false packer_cis.json | tee build.log\n",
"==> AWS AMI Builder - CIS: Prevalidating AMI Name...\n",
" AWS AMI Builder - CIS: Found Image ID: ami-6a9d2f0c\n",
"==> AWS AMI Builder - CIS: Creating temporary keypair: packer_5a3e06b9-b7d9-cdc3-2469-1794ed74e58c\n",
"==> AWS AMI Builder - CIS: Creating temporary security group for this instance: packer_5a3e06c1-00b3-fc52-77e7-6f4221704434\n",
"==> AWS AMI Builder - CIS: Authorizing access to port 22 on the temporary security group...\n",
"==> AWS AMI Builder - CIS: Launching a source AWS instance...\n",
" AWS AMI Builder - CIS: Instance ID: i-07f04a4ef18f16fc3\n",
"==> AWS AMI Builder - CIS: Waiting for instance (i-07f04a4ef18f16fc3) to become ready...\n",
"==> AWS AMI Builder - CIS: Adding tags to source instance\n",
" AWS AMI Builder - CIS: Adding tag: \"Name\": \"Prod-CIS-Latest-AMZN-23-Dec-17 07_33_13\"\n",
"==> AWS AMI Builder - CIS: Waiting for SSH to become available...\n",
"==> AWS AMI Builder - CIS: Connected to SSH!\n",
"==> AWS AMI Builder - CIS: Provisioning with shell script: /tmp/packer-shell486994840\n",
" AWS AMI Builder - CIS: Collecting ansible\n",
" AWS AMI Builder - CIS: Downloading ansible-2.4.2.0.tar.gz (6.5MB)\n",
" AWS AMI Builder - CIS: Requirement already satisfied: jinja2 in /usr/lib/python2.7/dist-packages (from ansible)\n",
" AWS AMI Builder - CIS: Requirement already satisfied: PyYAML in /usr/lib64/python2.7/dist-packages (from ansible)\n",
" AWS AMI Builder - CIS: Requirement already satisfied: paramiko in /usr/lib/python2.7/dist-packages (from ansible)\n",
" AWS AMI Builder - CIS: Collecting cryptography (from ansible)\n",
" AWS AMI Builder - CIS: Downloading cryptography-2.1.4-cp27-cp27mu-manylinux1_x86_64.whl (2.2MB)\n",
" AWS AMI Builder - CIS: Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages (from ansible)\n",
" AWS AMI Builder - CIS: Requirement already satisfied: markupsafe in /usr/lib64/python2.7/dist-packages (from jinja2->ansible)\n",
" AWS AMI Builder - CIS: Requirement already satisfied: pycrypto!=2.4,>=2.1 in /usr/lib64/python2.7/dist-packages (from paramiko->ansible)\n",
" AWS AMI Builder - CIS: Requirement already satisfied: ecdsa>=0.11 in /usr/lib/python2.7/dist-packages (from paramiko->ansible)\n",
" AWS AMI Builder - CIS: Collecting asn1crypto>=0.21.0 (from cryptography->ansible)\n",
" AWS AMI Builder - CIS: Downloading asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)\n",
" AWS AMI Builder - CIS: Requirement already satisfied: six>=1.4.1 in /usr/lib/python2.7/dist-packages (from cryptography->ansible)\n",
" AWS AMI Builder - CIS: Collecting enum34; python_version < \"3\" (from cryptography->ansible)\n",
" AWS AMI Builder - CIS: Downloading enum34-1.1.6-py2-none-any.whl\n",
" AWS AMI Builder - CIS: Collecting cffi>=1.7; platform_python_implementation != \"PyPy\" (from cryptography->ansible)\n",
" AWS AMI Builder - CIS: Downloading cffi-1.11.2-cp27-cp27mu-manylinux1_x86_64.whl (405kB)\n",
" AWS AMI Builder - CIS: Collecting ipaddress; python_version < \"3\" (from cryptography->ansible)\n",
" AWS AMI Builder - CIS: Downloading ipaddress-1.0.19.tar.gz\n",
" AWS AMI Builder - CIS: Collecting idna>=2.1 (from cryptography->ansible)\n",
" AWS AMI Builder - CIS: Downloading idna-2.6-py2.py3-none-any.whl (56kB)\n",
" AWS AMI Builder - CIS: Collecting pycparser (from cffi>=1.7; platform_python_implementation != \"PyPy\"->cryptography->ansible)\n",
" AWS AMI Builder - CIS: Downloading pycparser-2.18.tar.gz (245kB)\n",
" AWS AMI Builder - CIS: Installing collected packages: asn1crypto, enum34, pycparser, cffi, ipaddress, idna, cryptography, ansible\n",
" AWS AMI Builder - CIS: Running setup.py install for pycparser: started\n",
" AWS AMI Builder - CIS: Running setup.py install for pycparser: finished with status 'done'\n",
" AWS AMI Builder - CIS: Running setup.py install for ipaddress: started\n",
" AWS AMI Builder - CIS: Running setup.py install for ipaddress: finished with status 'done'\n",
" AWS AMI Builder - CIS: Running setup.py install for ansible: started\n",
" AWS AMI Builder - CIS: Running setup.py install for ansible: finished with status 'done'\n",
" AWS AMI Builder - CIS: Successfully installed ansible-2.4.2.0 asn1crypto-0.24.0 cffi-1.11.2 cryptography-2.1.4 enum34-1.1.6 idna-2.6 ipaddress-1.0.19 pycparser-2.1
8\n",
"==> AWS AMI Builder - CIS: Provisioning with Ansible...\n",
" AWS AMI Builder - CIS: Uploading Playbook directory to Ansible staging directory...\n",
" AWS AMI Builder - CIS: Creating directory: /tmp/packer-provisioner-ansible-local/5a3e06b9-af7f-f1c9-d148-fc36cc5051ea\n",
" AWS AMI Builder - CIS: Uploading main Playbook file...\n",
" AWS AMI Builder - CIS: Uploading galaxy file...\n",
" AWS AMI Builder - CIS: Uploading inventory file...\n",
" AWS AMI Builder - CIS: Uploading role directories...\n",
" AWS AMI Builder - CIS: Creating directory: /tmp/packer-provisioner-ansible-local/5a3e06b9-af7f-f1c9-d148-fc36cc5051ea/roles/common\n",
" AWS AMI Builder - CIS: Executing Ansible Galaxy: cd /tmp/packer-provisioner-ansible-local/5a3e06b9-af7f-f1c9-d148-fc36cc5051ea && ansible-galaxy install -r /tmp/pa
cker-provisioner-ansible-local/5a3e06b9-af7f-f1c9-d148-fc36cc5051ea/requirements.yaml -p /tmp/packer-provisioner-ansible-local/5a3e06b9-af7f-f1c9-d148-fc36cc5051ea/roles\n"
,
" AWS AMI Builder - CIS: - downloading role 'cis-amazon-linux', owned by anthcourtney\n",
" AWS AMI Builder - CIS: - downloading role from https://github.com/anthcourtney/ansible-role-cis-amazon-linux/archive/1.1.4.tar.gz\n",
" AWS AMI Builder - CIS: - extracting anthcourtney.cis-amazon-linux to /tmp/packer-provisioner-ansible-local/5a3e06b9-af7f-f1c9-d148-fc36cc5051ea/roles/anthcourtney.
cis-amazon-linux\n",
" AWS AMI Builder - CIS: - anthcourtney.cis-amazon-linux (1.1.4) was installed successfully\n",
" AWS AMI Builder - CIS: - downloading role 'aws-cloudwatch-logs-agent', owned by dharrisio\n",
" AWS AMI Builder - CIS: - downloading role from https://github.com/dharrisio/ansible-role-aws-cloudwatch-logs-agent/archive/v0.1.1.tar.gz\n",
" AWS AMI Builder - CIS: - extracting dharrisio.aws-cloudwatch-logs-agent to /tmp/packer-provisioner-ansible-local/5a3e06b9-af7f-f1c9-d148-fc36cc5051ea/roles/dharris
io.aws-cloudwatch-logs-agent\n",
" AWS AMI Builder - CIS: - dharrisio.aws-cloudwatch-logs-agent (v0.1.1) was installed successfully\n",
" AWS AMI Builder - CIS: Executing Ansible: cd /tmp/packer-provisioner-ansible-local/5a3e06b9-af7f-f1c9-d148-fc36cc5051ea && ANSIBLE_FORCE_COLOR=1 PYTHONUNBUFFERED=1
ansible-playbook /tmp/packer-provisioner-ansible-local/5a3e06b9-af7f-f1c9-d148-fc36cc5051ea/playbook.yaml --extra-vars \"packer_build_name=AWS AMI Builder - CIS packer_bui
lder_type=amazon-ebs packer_http_addr=\" -c local -i /tmp/packer-provisioner-ansible-local/5a3e06b9-af7f-f1c9-d148-fc36cc5051ea/packer-provisioner-ansible-local474467442\n
",
" AWS AMI Builder - CIS: \u001b[0;35m[DEPRECATION WARNING]: The use of 'include' for tasks has been deprecated. Use \u001b[0m\n",
" AWS AMI Builder - CIS: \u001b[0;35m'import_tasks' for static inclusions or 'include_tasks' for dynamic inclusions.\u001b[0m\n",
" AWS AMI Builder - CIS: \u001b[0;35m This feature will be removed in a future release. Deprecation warnings can be \u001b[0m\n",
" AWS AMI Builder - CIS: \u001b[0;35mdisabled by setting deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
" AWS AMI Builder - CIS: \u001b[0;35m[DEPRECATION WARNING]: include is kept for backwards compatibility but usage is\u001b[0m\n",
" AWS AMI Builder - CIS: \u001b[0;35m discouraged. The module documentation details page may explain more about this\u001b[0m\n",
" AWS AMI Builder - CIS: \u001b[0;35m rationale.. This feature will be removed in a future release. Deprecation \u001b[0m\n",
" AWS AMI Builder - CIS: \u001b[0;35mwarnings can be disabled by setting deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
" AWS AMI Builder - CIS: \u001b[1;35m [WARNING]: file /tmp/packer-provisioner-ansible-local/5a3e06b9-af7f-\u001b[0m\n",
" AWS AMI Builder - CIS: \u001b[1;35mf1c9-d148-fc36cc5051ea/roles/anthcourtney.cis-amazon-\u001b[0m\n",
" AWS AMI Builder - CIS: \u001b[1;35mlinux/tasks/level-1/2.2.1.2.yml is empty and had no tasks to include\u001b[0m\n",
" AWS AMI Builder - CIS: \u001b[1;35m\u001b[0m\n",
" AWS AMI Builder - CIS: \u001b[1;35m [WARNING]: file /tmp/packer-provisioner-ansible-local/5a3e06b9-af7f-\u001b[0m\n",
" AWS AMI Builder - CIS: \u001b[1;35mf1c9-d148-fc36cc5051ea/roles/anthcourtney.cis-amazon-\u001b[0m\n",
" AWS AMI Builder - CIS: \u001b[1;35mlinux/tasks/level-1/2.2.1.3.yml is empty and had no tasks to include\u001b[0m\n",
" AWS AMI Builder - CIS: \u001b[1;35m\u001b[0m\n",
" AWS AMI Builder - CIS:\n",
" AWS AMI Builder - CIS: PLAY [localhost] ***************************************************************\n",
" AWS AMI Builder - CIS:\n",
" AWS AMI Builder - CIS: TASK [Gathering Facts] *********************************************************\n",
" AWS AMI Builder - CIS: \u001b[0;32mok: [127.0.0.1]\u001b[0m\n",
" AWS AMI Builder - CIS:\n",
" AWS AMI Builder - CIS: TASK [common : upgrade all packages] *******************************************\n",
" AWS AMI Builder - CIS: \u001b[0;33mchanged: [127.0.0.1]\u001b[0m\n",
" AWS AMI Builder - CIS:\n",
" AWS AMI Builder - CIS: TASK [anthcourtney.cis-amazon-linux : Preflight - Fail if host is not suitable for this benchmark] ***\n",
" AWS AMI Builder - CIS: \u001b[0;31mfatal: [127.0.0.1]: FAILED! => {\"changed\": false, \"msg\": \"This benchmark is not suitable for the destination operating system\"}\u001b[0m\n",
" AWS AMI Builder - CIS: to retry, use: --limit @/tmp/packer-provisioner-ansible-local/5a3e06b9-af7f-f1c9-d148-fc36cc5051ea/playbook.retry\n",
" AWS AMI Builder - CIS:\n",
" AWS AMI Builder - CIS: PLAY RECAP *********************************************************************\n",
" AWS AMI Builder - CIS: \u001b[0;31m127.0.0.1\u001b[0m : \u001b[0;32mok=2 \u001b[0m \u001b[0;33mchanged=1 \u001b[0m unreachable=0 \u001b[0;3
1mfailed=1 \u001b[0m\n",
" AWS AMI Builder - CIS:\n",
"==> AWS AMI Builder - CIS: Terminating the source AWS instance...\n",
"==> AWS AMI Builder - CIS: Cleaning up any extra volumes...\n",
"==> AWS AMI Builder - CIS: No volumes to clean up, skipping\n",
"==> AWS AMI Builder - CIS: Deleting temporary security group...\n",
"==> AWS AMI Builder - CIS: Deleting temporary keypair...\n",
"Build 'AWS AMI Builder - CIS' errored: Error executing Ansible: Non-zero exit status: 2\n",
"\n",
"==> Some builds didn't complete successfully and had errors:\n",
"--> AWS AMI Builder - CIS: Error executing Ansible: Non-zero exit status: 2\n",
"\n",
"==> Builds finished but no artifacts were created.\n",
"\n",
"[Container] 2017/12/23 07:36:45 Phase complete: BUILD Success: true\n",
"[Container] 2017/12/23 07:36:45 Phase context status code: Message: \n",
"[Container] 2017/12/23 07:36:45 Entering phase POST_BUILD\n",
"[Container] 2017/12/23 07:36:45 Running command egrep \"${AWS_REGION}\\:\\sami\\-\" build.log | cut -d' ' -f2 > ami_id.txt\n",
"\n",
"[Container] 2017/12/23 07:36:45 Running command test -s ami_id.txt || exit 1\n",
"\n",
"[Container] 2017/12/23 07:36:45 Command did not exit successfully test -s ami_id.txt || exit 1 exit status 1\n",
"[Container] 2017/12/23 07:36:45 Phase complete: POST_BUILD Success: false\n",
"[Container] 2017/12/23 07:36:45 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: test -s ami_id.txt || exit 1. Reason: exit s
tatus 1\n"
]

"anthcourtney.cis-amazon-linux"のロールの処理の部分（136行目）でエラーが発生しているようです。 細かい原因追求は置いといて、Ansible Playbookの該当部分をコメントアウトして再度リポジトリにプッシュします。

再度、結果を確認していましょう。

1
2
3
aws codepipeline list-pipeline-executions \
    --pipeline-name AMI-Builder_pipeline \
    --query "sort_by(pipelineExecutionSummaries,&startTime)[-1].{ID:pipelineExecutionId,Status:status}"
1
2
3
4
{
  "Status": "Succeeded",
  "ID": "732863af-7e86-479d-9b62-35f8cf50a9e3"
}

今度は成功していました。 設定したメールアドレスをメールが届いているか確認したところ、以下のメッセージが含まれたメールを受信していました。

1
{"version":"0","id":"75db9922-b837-a7f7-445d-f6b15e79f7b3","detail-type":"AmiBuilder","source":"com.ami.builder","account":"XXXXXXXXXXXX","time":"2017-12-23T08:58:37Z","region":"ap-northeast-1","resources":["ami-bb57dddd"],"detail":{"AmiStatus":"Created"}}

最後にAMIの存在を確認します。 無事、作成できていました。

1
2
aws ec2 describe-images \
    --image-ids ami-bb57dddd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
{
  "Images": [
    {
      "VirtualizationType": "hvm",
      "Description": "Amazon Linux CIS with Cloudwatch Logs agent",
      "Tags": [
        {
          "Value": "Prod-CIS-Latest-AMZN-23-Dec-17 08_52_45",
          "Key": "Name"
        }
      ],
      "Hypervisor": "xen",
      "EnaSupport": true,
      "SriovNetSupport": "simple",
      "ImageId": "ami-bb57dddd",
      "State": "available",
      "BlockDeviceMappings": [
        {
          "DeviceName": "/dev/xvda",
          "Ebs": {
            "Encrypted": false,
            "DeleteOnTermination": true,
            "VolumeType": "standard",
            "VolumeSize": 8,
            "SnapshotId": "snap-0aad91e06631d8804"
          }
        }
      ],
      "Architecture": "x86_64",
      "ImageLocation": "XXXXXXXXXXXX/Prod-CIS-Latest-AMZN-23-Dec-17 08_52_45",
      "RootDeviceType": "ebs",
      "OwnerId": "XXXXXXXXXXXX",
      "RootDeviceName": "/dev/xvda",
      "CreationDate": "2017-12-23T08:56:46.000Z",
      "Public": false,
      "ImageType": "machine",
      "Name": "Prod-CIS-Latest-AMZN-23-Dec-17 08_52_45"
    }
  ]
}

まとめ

こういった仕組みを使ってAMIのコード化／作成の自動化を行うことができれば、運用の負担を下げることができると思います。
気になる方は使ってみてはいかがでしょうか。
その際は、自分自身が中身を理解することはもちろん、他の方にも理解してもらえるような仕組みにするよう心がけましょう。

awslabsには他にもいろんなリポジトリがありますので、気になる方は覗いてみましょう。

Invent2017japan portal