This AWS Lambda function allows you to delete the old Elasticsearch indexes using SigV4Auth authentication. You configure the AWS Elasticsearch Access Policy authorizing the Lambda Role or the AWS Account number instead of using the IP address whitelist.
Clone your repository
$ git clone git@github.com:cloudreach/aws-lambda-es-cleanup.git
$ cd aws-lambda-es-cleanup/
Configure in a proper way the IAM policy inside json_file/es_policy.json
and json_file/trust_policy.json
Create the IAM Role
$ aws iam create-role --role-name es-cleanup-lambda \
--assume-role-policy-document file://json_file/trust_policy.json
$ aws iam put-role-policy --role-name es-cleanup-lambda \
--policy-name es_cleanup \
--policy-document file://json_file/es_policy.json
if your lambda is running inside the VPC also attach the these policies
arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
Create your Lambda package
$ zip es-cleanup-lambda.zip es_cleanup.py
Using awscli you can create your AWS function and set the proper IAM role with the right Account ID
$ export AWS_DEFAULT_REGION=eu-west-1
$ ESENDPOINT="search-es-demo-zveqnhnhjqm5flntemgmx5iuya.eu-west-1.es.amazonaws.com" #ES endpoint
$ aws lambda create-function \
--function-name es-cleanup-lambda \
--environment Variables={es_endpoint=$ESENDPOINT} \
--zip-file fileb://es-cleanup-lambda.zip \
--description "Elasticsearch Index Cleanup" \
--role arn:aws:iam::123456789012:role/es-cleanup-lambda \
--handler es_cleanup.lambda_handler \
--runtime python3.6 \
--timeout 180
If you want to send variables and not to use environment
$ export AWS_DEFAULT_REGION=eu-west-1
$ aws lambda create-function \
--function-name es-cleanup-lambda \
--zip-file fileb://es-cleanup-lambda.zip \
--description "Elasticsearch Index Cleanup" \
--role arn:aws:iam::123456789012:role/es-cleanup-lambda \
--handler es_cleanup.lambda_handler \
--runtime python3.6 \
--timeout 180
is it possible to override the default behaviour passing specific payload
$ aws lambda invoke
--function-name es-cleanup-lambda \
outfile --payload \
'{"es_endpoint":"search-es-demo-zveqnhnhjqm5flntemgmx5iuya.eu-west-1.es.amazonaws.com"}'
Create your AWS Cloudwatch rule:
$ aws events put-rule \
--name my-scheduled-rule \
--schedule-expression 'cron(0 1 * * ? *)'
$ aws lambda add-permission \
--function-name es-cleanup-lambda \
--statement-id my-scheduled-event \
--action 'lambda:InvokeFunction' \
--principal events.amazonaws.com \
--source-arn arn:aws:events:eu-west-1:123456789012:rule/my-scheduled-rule
$ aws events put-targets \
--rule my-scheduled-rule \
--targets file://json_file/cloudwatch-target.json
Using AWS environment variable you can easily modify the behaviour of the Lambda function
Variable Name | Example Value | Description | Default Value | Required |
---|---|---|---|---|
es_endpoint | search-es-demo-zveqnhnhjqm5flntemgmx5iuya.eu-west-1.es.amazonaws.com | AWS ES fqdn | None |
True |
index | logstash* |
Index/indices to process comma separated, with all every index will be processed except the one listed in skip_index |
.* |
False |
skip_index | .kibana,.kibana_5 |
Index/indices to skip | .kibana* |
False |
index_format | %Y.%m.%d |
Combined with index varible is used to evaluate the index age |
%Y.%m.%d |
False |
delete_after | 7 |
Numbers of days to preserve | 15 |
False |
Editing the file serverless.yml
, you can deploy your function in AWS using Serverless Framework
$ git clone git@github.com:cloudreach/aws-lambda-es-cleanup.git
$ cd aws-lambda-es-cleanup/
$ serverless deploy
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
.....
Serverless: Stack create finished...
Serverless: Packaging service...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading function .zip files to S3...
Serverless: Uploading service .zip file to S3 (7.13 KB)...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
......................
Serverless: Stack update finished...
Service Information
service: es-cleanup-lambda
stage: prod
region: eu-west-1
api keys:
None
endpoints:
None
functions:
es-cleanup-lambda: es-cleanup-lambda-prod-es-cleanup-lambda
This lambda function can be also build using terraform followings this README.
We encourage contribution to our projects, please see our CONTRIBUTING guide for details.
aws-lambda-es-cleanup is licensed under the Apache Software License 2.0.
Keep It Cloudy (@CloudreachKIC)