(cache) Google Cloud Platform Blog: Going multi-cloud with Google Cloud Endpoints and AWS Lambda

Google Cloud Platform Blog

Product updates, customer stories, and tips and tricks on Google Cloud Platform

Going multi-cloud with Google Cloud Endpoints and AWS Lambda

Monday, April 24, 2017

By Sowmya Kannan, Technical Curriculum Developer, Google Cloud PlatformVisionCloud Video IntelligenceData Loss PreventionGoogle Cloud PlatformGoogle Cloud EndpointsAWS Lambda

Cloud EndpointsOpenAPI specificationApp EngineGoogle Container EngineCompute Enginedownload the complete source code for this sample from GitHub Setting up Cloud EndpointsApp Engine flexible environment

Use the GCP Console to create a new GCP project and an App Engine application.

When prompted, select the region where you want to run your App Engine application located and then enable billing.

Note the ID of your project, because you'll need it later.

Activate Google Cloud Shell

Retrieving source codeGoogle Cloud Shell

Clone the GitHub repository with source code for the example.

Change directory to blogs/endpointslambda/aeflex-endpoints/

Launch the code editor to view source code.

Implementing application code @app.route('/processmessage', methods=['POST']) def process(): """Process messages with information about S3 objects""" message = request.get_json().get('inputMessage', '') # add other processing as needed # for example, add event to PubSub or # download object using presigned URL, save in Cloud Storage, invoke ML APIs return jsonify({'In app code for endpoint, received message': message}) Configuring and deploying the Cloud Endpoints APISwagger object host: "echo-api.endpoints.aeflex-endpoints.cloud.goog"inputMessage# This section configures the processmessage endpoint. "/processmessage": post: description: "Process the given message." operationId: "processmessage" produces: - "application/json" responses: 200: description: "Return a success response" schema: $ref: "#/definitions/successMessage" parameters: - description: "Message to process" in: body name: inputMessage required: true schema: $ref: "#/definitions/inputMessage" security: - api_key: [] definitions: successMessage: properties: message: type: string inputMessage: # This section contains information about the S3 bucket and object to be processed. properties: Bucket: type: string ObjectKey: type: string ContentType: type: string ContentLength: type: integer ETag: type: string PresignedUrl: type: string gcloud service-management deploy openapi.yamlService Configuration [2017-03-05r2] uploaded for service "echo-api.endpoints.aeflex-endpoints.cloud.goog"endpoints_api_service: # The following values are to be replaced by information from the output of # 'gcloud service-management deploy openapi.yaml' command. name: echo-api.endpoints.aeflex-endpoints.cloud.goog config_id: 2017-03-05r2 gcloud app deploy

In the GCP Console, on the Products & services menu, click API Manager > Credentials.

Click Create Credentials > API key. Note the API key.

Setting up the AWS Lambda functionblogs/endpointslambda/lambdafunctioninline.py. Creating the S3 Bucket for your files Creating an IAM Role

In the IAM Management Console, click Roles in the navigation pane.

Create a new role called LambdaExecRole.

Select AWS Lambda Role Type, and then select the following policies:

AWSLambdaExecute: This policy gives the Lambda function Put and Get access to S3 and full access to CloudWatch Logs.
AmazonSQSFullAccess: This policy gives the Lambda function permissions to send messages to the dead-letter queue (DLQ).

Review the settings and create the role.

Creating an SQS queue

In the SQS Management Console, create an SQS queue called IntegrationDLQ that will act as the dead-letter queue for your Lambda function. AWS Lambda automatically retries failed executions for asynchronous invocations. In addition, you can configure Lambda to forward payloads that were not processed to a dead-letter queue.

Creating a Lambda function in the AWS Console

In the Select blueprint page, use the s3-get-object-python blueprint.

In the Configure triggers page, specify the following:

Bucket: images-bucket-rawdata
Event-type: Object-Created (All)

Enable the trigger.

In the Configure Function page, specify the following:

Name: CallEndpoint
Runtime: Python 2.7
Code entry type: Edit code inline
Environment variable key: ENDPOINT_API_KEY
Environment variable: . Note: For production environments, consider securing the API key using encryption helpers in AWS.
Handler: lambda_function.lambda_handler
Role: Choose an existing role
Existing role: LambdaExecRole (that you created earlier)
In Advanced settings, DLQ resource: IntegrationDLQ

In the inline code editor for the Lambda function, replace the original code with the following code. The Lambda function retrieves the bucket and object information from the event, retrieves object metadata, generates a pre-signed url for the object, and finally invokes the Cloud Endpoints API.

from __future__ import print_function

import boto3
import json
import os
import urllib
import urllib2


print('Loading function')

s3 = boto3.client('s3')
endpoint_api_key = os.environ['ENDPOINT_API_KEY']
endpoint_url = "https://aeflex-endpoints.appspot.com/processmessage"

def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))

    # Get the object information from the event
    bucket = event['Records'][0]['s3']['bucket']['name']
    object_key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key'].encode('utf8'))
    try:
        # Retrieve object metadata
        response = s3.head_object(Bucket=bucket, Key=object_key)
        # Generate pre-signed URL for object
        presigned_url = s3.generate_presigned_url('get_object', Params = {'Bucket': bucket, 'Key': object_key}, ExpiresIn = 3600)
        data = {"inputMessage": {
                    "Bucket": bucket,
                    "ObjectKey": object_key,
                    "ContentType": response['ContentType'],
                    "ContentLength": response['ContentLength'],
                    "ETag": response['ETag'],
                    "PresignedUrl": presigned_url
            }
        }

        headers = {"Content-Type": "application/json",
                    "x-api-key": endpoint_api_key
        }
        # Invoke Cloud Endpoints API
        request = urllib2.Request(endpoint_url, data = json.dumps(data), headers = headers)
        response = urllib2.urlopen(request)
        
        print('Response text: {} \nResponse status: {}'.format(response.read(), response.getcode()))

        return response.getcode()
    except Exception as e:
        print(e)
        print('Error integrating lambda function with endpoint for the object {} in bucket {}'.format(object_key, bucket))
        raise e

Review the Lambda function configuration and create the function.

Testing the integration

In the AWS S3 Management Console, upload a file to images-bucket-rawdata.

In the Lambda Management Console, navigate to the CallEndpoint function, and view the Monitoring tab. The CloudWatch graphs show that the function was invoked successfully.

Click View Logs in CloudWatch and view the details for the function’s log stream. The log shows that the Lambda function received a success response code (200) from the API as well as details about the S3 event that the API received from the Lambda function.