Serverless API makes calling timeout

Since recently we move API to serverless, user got Error Accessing server message and Filed to upload file message if user tried to upload larger csv file. The API endpoint should work on validate each data row of csv file, if it spends over certain amount time, it will fail. The CloudFront and timeout issue is suspicious (CloudFront default timeout is 30s)

structure like above

Note: Since this is general problem on serverless API call, this issue should happen in all batch Imports. We need to find out infra solution (not just adjust timeout period)

Possible solution: we can use BackgroundService in .NET is to provide a structured way to run long-running or periodic tasks in the background—independent of the main application flow. Plus, we have existing service bus schema can apply.
We create new BackGroundService in API, and register it in service startup. BackGroundService will host long-polling service, once the caller from client, it will start service bus work item, and client side will get feedback directly to avoid timeout. Client side will have polling to check if validation process completed until finished.

Get Cookie Jen’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

API Code is like this:

protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
    // Once the runtime requests cancellation, things should wrap up and exit.
    while (!cancellationToken.IsCancellationRequested)
    {
       try
        {
            Heartbeat();
            await Task.Delay(1_000, cancellationToken);

            // This background service is a singleton, but some services it uses are bound InRequestScope,
            // so create a scope we can use to resolve the dependencies
            using (var scope = _ServiceScopeFactory.CreateScope())
            {
                // get validation work item
                var batchValidationWorkItemService = scope.ServiceProvider.GetRequiredService<IBatchValidationWorkItemService>();

                // we will either get 0 or 1 work items (current implementation never returns more than 1 work item)
                var workItems = batchValidationWorkItemService.GetNextWorkItemsToProcess(LocalDevUser);

                if (workItems.Count > 0)
                {
                    var workItem = workItems[0];
                    try
                    {
                        _Log.LogInformation($"....");
                        var sw = Stopwatch.StartNew();

                        // process the work item
                        await ValidateAsync(scope, workItem);

                        sw.Stop();
                        _Log.LogInformation($"...");
                    }
.....

private async Task ValidateAsync(IServiceScope scope, BatchValidationWorkItem workItem)
{
    //....
    // validate process
     var validationService = scope.ServiceProvider.GetRequiredService<Ed.EAP.BatchServices.Services.Skills.FieldTestItems.Transfer.IValidationService>();
    // start service bus
     await validationService.ValidateAndStoreResultsInS3(fileMetadata, BatchUploadRootDirectory, workItem.UserId);
     ......

In API controller:

public BatchUploadBaseController(
            IPublisherService publisherService,
            IBatchValidationServiceBase<TRowValidation> validationService,
            IProcessingHistoryServiceBase<TDataProcessingSummary> processingHistoryService,
            IWebHostEnvironment hostingEnvironment,
            IBatchActivityValidationWorkItemService batchActivityValidationWorkItemService = null
            )
            : base(publisherService, validationService, processingHistoryService, hostingEnvironment)
        {
            _BatchActivityValidationWorkItemService = batchActivityValidationWorkItemService;
        }

        protected async Task<IActionResult> CreateBatchValidationRequestAsync(BatchActivityType batchActivityType, SourceEntityType sourceEntityType = 0, int? entityId = null)
        {
            if (_BatchActivityValidationWorkItemService == null)
            {
                throw new NotImplementedException();
            }

            FileMetadata fileMetadata = await base.GetUploadedFileMetadata();

            await ((IBatchValidationServiceBase<TRowValidation>)_ValidationService).UploadBatchFileToS3(fileMetadata);

            // create work item - validation request will be processed in a background service
            _BatchActivityValidationWorkItemService.InsertWorkItem(new BatchActivityValidationWorkItem
            {
                SessionId = fileMetadata.SessionId,
                UploadFileName = fileMetadata.FileName,
                BatchActivityType = batchActivityType,
                SourceEntityType = sourceEntityType,
                EntityId = entityId,
                UserId = AuditUserId
            });

            return Ok(new { SessionId = fileMetadata.SessionId });
        }

I won’t go detail to stored procedure in DB layer. In API will call server side which have existing service bus schema to take care batch process, follow existing pattern, we create validation work in server side:
1. Repo (DB) layer e.g. deal with CRUD work item to DB
2. Materialization layer (mapping)
3. Service layer (implement ValidateAndStoreResultsInS3)
And, we need to implement 4 new method in BatchServices

 public interface IBatchValidationServiceBase<T>
        : IFileUploadValidationServiceBase<T>
        where T : BatchRowValidationModelBase
    {
        Task ValidateAndStoreResultsInS3(FileMetadata fileMetadata, string batchUploadRootDirectory, int auditUserId, BatchEntity entity = null);
        Task UploadBatchFileToS3(FileMetadata fileMetadata);
        Task<byte[]> GetValidationResultJsonFromS3(string sessionId);
        Task<byte[]> GetErrorReportCsvFromS3(string sessionId);        

Another approach is to adjust Cloudfront timeout in AWS, the origin here should be load balancer.

setup timeout in cloudfront