Hi @rahul-JFL,

Thanks for posting the guidance question. Please share logs and stack trace (if any) from CloudWatch. Also, it would be helpful if you could add logging statements (using ILambdaContext.Logger) to the above code to trace the execution.

The problem if see in the above code is that await is used to execute Task.Run() which executes loop in the background thread and then use S3Client.PutObjectAsync() which is not awaited. It is quite possible that the background thread for S3Client.PutObjectAsync() are paused (since these are not awaited) when Lambda function returns. Lambda Test Tool is just a simulation of code execution, it is different from the actual Lambda execution environment.

Thanks,
Ashish

Hi @rahul-JFL,

GitHub is the effective way to communicate about issues. Please post the logs (after adding logging statements), code snippet (if different from above) and stack trace for any failures here on GitHub.

Also try using await keyword before S3Client.PutObjectAsync(request) call like below and see if it helps:

await S3Client.PutObjectAsync(request);

Thanks,
Ashish

Thanks Ashis,
My lambda Execution logs is here ( LambdaLogger.Log)

START RequestId: 02906c10-9b66-495e-baeb-27683602d25c Version: $LATEST
fileName: DPI66639_20200819120927.csv
fileContent :1021,20200819,1000,D834,000000000010000492,C009,0.1,KGS,
Call-Class:UploadFileS3;Function:UploadIndent
bucketName: indentbucket/Post
END RequestId: 02906c10-9b66-495e-baeb-27683602d25c
REPORT RequestId: 02906c10-9b66-495e-baeb-27683602d25c Duration: 90090.23 ms Billed Duration: 90000 ms Memory Size: 640 MB Max Memory Used: 112 MB Init Duration: 160.06 ms
2020-08-19T12:10:55.064Z 02906c10-9b66-495e-baeb-27683602d25c Task timed out after 90.09 seconds

                                                             Parent Function

public String PostIndentFile(String StoreId, DataTable dt)
{
try
{
string fileName = string.Format("_{0:yyyyMMddhhmmss}.csv", DateTime.Now);
fileName = fileName.Replace("", StoreId);
LambdaLogger.Log("fileName: " + fileName);
//String fileName = Extennsions.GetFileName(StoreId);
String input = GetDataForFile(dt);
LambdaLogger.Log("fileContent :" + input.ToString());
UploadFileS3 oUploadFileS3 = new UploadFileS3();
var result = oUploadFileS3.UploadIndent(input, fileName);
LambdaLogger.Log("Result1: " + result);
return result;
}
catch (Exception ex)
{
return ex.Message.ToString();
}
}

                                                             Child Function

public async Task UploadFileToS3Bucket(String input, String fileName)
{
S3 = new AmazonS3Client(awsAccessKeyId, awsSecretAccessKey, RegionEndpoint.APSouth1);
LambdaLogger.Log("RegionEndPoint:" + RegionEndpoint.APEast1);
LambdaLogger.Log("bucketName:" + bucketName);
LambdaLogger.Log("fileName:" + fileName);
LambdaLogger.Log("input:" + input);
return await Task.Run(() =>
{
for (int i = 0; i < 6; i++)
{
PutObjectRequest request = new PutObjectRequest
{
BucketName = bucketName,
Key = fileName,
ContentBody = input,
ContentType = "text/csv",
StorageClass = S3StorageClass.Standard,
CannedACL = S3CannedACL.NoACL
};
S3.PutObjectAsync(request);
LambdaLogger.Log("PutObjectAsync Request:" + request);
Task.Delay(TimeSpan.FromSeconds(1)).Wait();
}
LambdaLogger.Log("Put Object Completed");
return "Success";
}).ConfigureAwait(false);
}

Hi @rahul-JFL,

Thanks for your inputs. I made some minor changes to your code (notice the proper use of await keyword specifically) and was able to successfully upload file to S3 inside AWS Lambda:

public async Task<string> FunctionHandler(string input, ILambdaContext context)
{
    string returnMessage = await UploadFileToS3Bucket(fileContent, fileName);
    return returnMessage;
}

public async Task<string> UploadFileToS3Bucket(String input, String fileName)
{
    using (AmazonS3Client s3Client = new AmazonS3Client(awsAccessKeyId, awsSecretAccessKey, awsSessionToken, region))
    {
        LambdaLogger.Log("RegionEndPoint:" + region);
        LambdaLogger.Log("bucketName:" + bucketName);
        LambdaLogger.Log("fileName:" + fileName);
        LambdaLogger.Log("input:" + input);

        return await Task.Run(async () =>
        {
            for (int i = 0; i < 6; i++)
            {
                PutObjectRequest request = new PutObjectRequest
                {
                    BucketName = bucketName,
                    Key = fileName,
                    ContentBody = input,
                    ContentType = "text/csv",
                    StorageClass = S3StorageClass.Standard,
                    CannedACL = S3CannedACL.NoACL
                };
                await s3Client.PutObjectAsync(request);
                LambdaLogger.Log("PutObjectAsync Request:" + request);
                Task.Delay(TimeSpan.FromSeconds(1)).Wait();
            }

            LambdaLogger.Log("Put Object Completed");
            return "Success";
        }).ConfigureAwait(false);
    }
}

Logs:

START RequestId: 64616574-6b98-48a3-875c-7e0a145be654 Version: $LATEST
RegionEndPoint:Asia Pacific (Mumbai) (ap-south-1)bucketName:tests3bucket-1681fileName:DPI66639_20200819120927.csvinput:1021,20200819,1000,D834,000000000010000492,C009,0.1,KGS,PutObjectAsync Request:Amazon.S3.Model.PutObjectRequestPutObjectAsync Request:Amazon.S3.Model.PutObjectRequestPutObjectAsync Request:Amazon.S3.Model.PutObjectRequestPutObjectAsync Request:Amazon.S3.Model.PutObjectRequestPutObjectAsync Request:Amazon.S3.Model.PutObjectRequestPutObjectAsync Request:Amazon.S3.Model.PutObjectRequestPut Object CompletedEND RequestId: 64616574-6b98-48a3-875c-7e0a145be654
REPORT RequestId: 64616574-6b98-48a3-875c-7e0a145be654	Duration: 10242.09 ms	Billed Duration: 10300 ms	Memory Size: 256 MB	Max Memory Used: 104 MB	

I also had to add appropriate role to AWS Lambda so that it's able to upload file to S3.

Since you are also executing set 1 and 3 before uploading the file, you may also try increasing timeout (refer article Configuring functions in the AWS Lambda console) if those steps are causing any delay.

Please let me know if this helps.

Thanks,
Ashish

@rahul-JFL,
AwsSessionToken is not a mandatory token. Please check the permissions and policies. Also modify u you our code to use the await keyword properly since any background threads would be suspended once Lambda function returns.

Hi @rahul-JFL,

Could you confirm if you modified your code to properly use "await" keyword? As mentioned earlier, background threads could be suspended if the Lambda returns the invocation call. In your code, the call s3Client.PutObjectAsync(request) needs to awaited or executed synchronously.
In other words:

await s3Client.PutObjectAsync(request);

or

var putItemResponse = s3Client.PutObjectAsync(request).Result;

Using the modified code snippet shared with you, I'm unable to reproduce the issue.

Thanks,
Ashish

Yes, I tried both Async as well as synchronous. Please find attached my both function class and exception screen shot.

Async-Function.txt
synchronously-Function.txt

@ashishdhingra : I really appreciate your effort and support but for me, this is show stopper task . Please help me.

Hi @rahul-JFL,

In your code, could you please change the following:

private string bucketName = "indentbucket";
...
string returnMessage = UploadFileToS3Bucket(input, "Post/MyText.csv");

Refer to Bucket restrictions and limitations and Object key and metadata for the naming guidelines.

Thanks,
Ashish

Hi @ashishdhingra

I changed my bucket and removed the folder also, I am not able to put my object in the bucket. Please see my attached class code after your suggestion and log.

sync_FunctionClass.txt
Async_FunctionClass.txt
Async-Fun_Log.txt
sync_Fun_Log.txt

Also please check my bucket permission

Thanks
Rahul Gupta

Hi @ashishdhingra
One more thing, Don't have any 'AWS Policy Generator' for my buckets. Do i need to generate policy?

Regards,
Rahul Gupta

Hi @rahul-JFL,

Could you check if your Lambda has proper role attached to access S3.

Here are the steps I used to deploy Lambda:

Notice the region

Notice the Role Name field. You may also try increasing timeout.

Function executes successfully

File Uploaded:

I'm using the same code as shared earlier and it works fine (just changed the Key for the uploaded file to prefix it with "Post/", since you wanted it to upload to sub folder),

It appears be the environmental issue, may be VPC, security groups, etc.

Thanks,
Ashish

Thanks @ashishdhingra
Yes, I am also using the same steps for uploading Lambda Function.
And yes finally found VPC is the main problem area in my case. Without VPC, I am able to upload my csv file. Thanks.

But now another issue, if i will not use VPC in my lambda function so I am unable to connect to RDS service(MySQL).
My function has steps

1. Read json file and logged into MySQL.
2. Read logged records and do some required data conversion through the same lambda function.
   3)creating a csv file and saving it on S3 bucket.

Can we connect RDS-MySQL without VPC?

Thanks & Regards,
Rahul Gupta

My VPC routes table

And Subnet Associations

My Subnet Associations

Hi @rahul-JFL,

Good morning.

As we figured out earlier, Lambda code is correct and working, so the only thing left is to figure out the proper service level configuration within VPC. I would recommend going through service specific documentation which could provide some reference to VPC specific scenarios. Following documentation links might be helpful for your scenario:

• Example Bucket Policies for VPC Endpoints for Amazon S3

• Configuring a Lambda function to access resources in a VPC

For any other support related to VPC, please directly contact service specific support to get expertise guidance related to service.

Please let me know if any more guidance for .NET SDK related Lambda code is required and if this issue could be closed.

Thanks,
Ashish

Thanks @ashishdhingra

Let me check and close this issue soon.
I have another question , how can I pass a csv string to my lambda function as an input parameter.

Actually, my scenario is that
Triggered (if any .csv file pushed) a lambda function then a lambda function will read a csv file from S3- bucket and data saved into RDS.

JFLGetSTN.txt

Thanks
Rahul Gupta

Hi @rahul-JFL,

There is one good article Tutorial: Importing data into Amazon Pinpoint from external sources which closely resembles you scenarios. It basically involves setting up events on S3 bucket whenever a CSV file is pushed to the bucket which later triggers lambda function. Lambda function would receive event data in a specified JSON format which you can use to fetch file from S3, parse CSV content and do further processing.

There are many such code samples online if you look out for it in a search engine.

Kindly note that the above article is just provided as a guidance source and you should develop and fully test your code before production implementation.

Let me know if this helps.

Thanks,
Ashish

Hi Ashish,

Lambda Function to be invoked or triggered by S3(csv file upload). My query is here now, how do i get csv file data in function, because i tried with 'String input' parameter.but it is not working.

Thanks & Regards;
Rahul Gupta

aws/aws-lambda-dotnet#723

Hi @rahul-JFL,

I notice that the new guidance issue aws/aws-lambda-dotnet#723 was created for your question on Lambda triggered via CSV upload in S3 bucket. Hence I will close this issue now since the original issue mentioned here was resolved. We can continue discussion on CSV guidance under new issue.

Thanks,
Ashish