Datadog CI - Access denied on new layer releases - IAM Principal

[snlblakesperling]

Expected Behavior

Our Lambda CI queries release tags for the latest datadog-lambda-js layer to apply during datadog-ci instrumentation:

LATEST_DATADOG_LAYER=$(git -c 'versionsort.suffix=-' ls-remote --tags --sort='v:refname' https://github.com/DataDog/datadog-lambda-js/ | tail -n1 | cut -d '.' -f2)

datadog-ci lambda instrument \
    ...
    --layer-version $LATEST_DATADOG_LAYER

Ordinarily this works, datadog-ci-plugin-lambda accepts the shorthand layer version and retrieves the full layer ARN.

Actual Behavior

Over the last two releases of datadog-lambda-js (129, 130), instrumentation during the first few hours of release fails, IAM access is denied to the newly-published layer. This may have been ongoing, but our release schedules have coincided these last two times.

... is not authorized to perform: lambda:GetLayerVersion on resource: arn:aws:lambda:us-west-2:464622532012:layer:Datadog-Node18-x:130 because no resource-based policy allows the lambda:GetLayerVersion action

This seems to resolve within a few hours, and I've noticed tags are re-pushed. We simply re-run the CI at that point, and all is well.

Steps to Reproduce the Problem

1. Tag new version of datadog-lambda-js on upstream repo
2. Use version number as arg: --layer-version 130 in datadog-ci
3. If within the first few hours of release, IAM may decline GetLayerVersion.
4. Watch release tags for a re-push, then GetLayerVersion can be ran.

Specifications

• Datadog Lambda Layer version: 129, 130
• Node version: 18

---

While the issue is returned by datadog-ci, the core issue seems to be the IAM configuration for datadog-lambda-js layers, so i've opened the report here. Let me know any additional info we can provide!

I've noticed the deployment script should allow open principal/permissions for datadog-ci's lambda:GetLayerVersion call. Perhaps this is running into difficulty?