AWS Lambda python deployment with Pre-installed packages like pandas, numpy and other needed packages is little complicated. Since AWS Lambda uses amazon linux as its base os, you need to add the packages based on amazon linux for smooth working.
Install the following
Docker - https://www.docker.com/community-edition
AWS CLI - https://aws.amazon.com/cli/
pip install awscli
1. Start docker with following image/container
docker pull dacut/amazon-linux-python-3.6
docker run -it dacut/amazon-linux-python-3.6 /bin/bash
2.Create a virtualenvs directory to store all packages in /tmp/lambda folder virtual environments
bash-4.2$ mkdir /tmp/lambda
3.Make a new virtual environment with no packages
bash-4.2$ virtualenv /tmp/lambda --no-site-packages
4.To use the virtual environment
bash-4.2$ cd /tmp/lambda/bin
bash-4.2$ source activate
5. Pip install the needed package, if you
(lambda) bash-4.2$ pip3 install <package-name>
for example
(lambda) bash-4.2$ pip3 install pandas
6. Zip the site packages
(lambda) bash-4.2# cd /tmp/lambda/lib/python3.6/site-packages/
(lambda) bash-4.2# zip -r9 /tmp/package.zip .
7. In order to copy a file from a container to the MK MAC, you can use the command (get container id by using docker ps command)
docker cp <containerId>:/tmp/package.zip /Users/Projects/Python/Project1
Reference URL:
http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html#with-s3-example-deployment-pkg-python
https://gist.github.com/IamAdiSri/a379c36b70044725a85a1216e7ee9a46
Save the below code in deploy_lambda.sh
echo "## Set paths ##"
# there should not be any space between = and the value, for e.g
# "VAR_AWSLAMBDA_FUNCTION_NAME = mylambdafunction" will throw an error
VAR_PYTHON_FILE_NAME=your_python_code_file.py
VAR_PYTHON_CODE_PATH=/Users/mk/Projects/Python/AWS_Lambda
VAR_PACKAGE_ZIP_FILE_NAME=lambda.zip
VAR_ZIP_FILE_PATH=/tmp/lambda
VAR_AWSLAMBDA_FUNCTION_NAME=mylambdafunction
VAR_S3_BUCKET_NAME=myS3bucket
# Zip any other required packages
echo "## Adding read_excel.py to lambda.zip ##"
echo "########################"
zip -ur $VAR_ZIP_FILE_PATH/$VAR_PACKAGE_ZIP_FILE_NAME $VAR_PYTHON_CODE_PATH/$VAR_PYTHON_FILE_NAME
echo "## DONE Updating ##"
echo "########################"
# Since the function code zip file exceeds 10 MB you have no option but to upload to S3 as mandated by AWS lambda
echo "## Copying function code to S3 ##"
/Users/mk/bin/aws s3 cp $VAR_ZIP_FILE_PATH/$VAR_PACKAGE_ZIP_FILE_NAME s3://$VAR_S3_BUCKET_NAME
echo "## DONE Copying to S3 ##"
echo "########################"
echo "## Updating Function Code ##"
/Users/mk/bin/aws lambda update-function-code --function-name "$VAR_AWSLAMBDA_FUNCTION_NAME" --s3-bucket "$VAR_S3_BUCKET_NAME" --s3-key "$VAR_ZIP_FILE_PATH/$VAR_PACKAGE_ZIP_FILE_NAME"
#aws lambda update-function-code --function-name "S3FileImportToCouchbaseInEC2" --s3-bucket "my-code" --s3-key "lambda_packages/lambda_function.zip"
echo "## DONE Updating ##"
echo "########################"
echo "## Run Lambda Command ##"
echo "########################"
/Users/mk/bin/aws lambda invoke \
--invocation-type RequestResponse \
--function-name $VAR_AWSLAMBDA_FUNCTION_NAME \
--region us-east-1 \
--log-type Tail \
--payload '{"key1":"value1", "key2":"value2", "key3":"value3"}' \
$VAR_PYTHON_CODE_PATH/outputfile.txt
echo "## Finished Lambda Command ##"
echo "########################"
#echo output.txt | base64 --decode
run the shell script to deploy and run the python code, you will get a base64 output, which you can decode using online or other tools to see the results.