IAM policies won’t save you. Encryption at rest won’t either — not when the account administrator has access to the keys and can reach anything in their account. But there is a way to keep your secret …

EC2 Instance Attestation

AWS documentation describes attestation as

Attestation is a process that allows you to cryptographically prove to any party that only trusted software, drivers, and boot processes are running on an Amazon EC2 instance. Amazon EC2 instance attestation is powered by the Nitro Trusted Platform Module (NitroTPM) and Attestable AMIs.

Above doesn’t do a very good job of describing what this feature is good for. There are 2 major use-cases for attestation. First it can be used as evidence for an auditor to show instance has not been tampered and has a known configuration. The second case, which I focus on in this post, is that this evidence can be used in KMS key policy condition to allow or deny decryption of secrets. When this is combined with OS level security measures such as enhanced read-only file system (erofs) and device-mapper verity (dm-verity) it is possible to deliver secrets to an environment that is safe even from the account owner.

Instance attestation is based on Nitro Trusted Platform Module (TPM) which is hardware that acts as a digital vault generating and storing sensitive information. You likely have one in your laptop. It stores biometric data and powers drive encryption.

Test Scenario

The diagram below shows the test scenario across two AWS accounts:

• Builder Account owns the attestable AMI, the KMS CMS key, and the encrypted secret. This is your account as the SaaS vendor.
• Test Account represents the customer’s account where your product will actually run.

1. Instance launch — The test account starts an EC2 instance from the AMI shared by the builder account. Because the AMI is attestable, NitroTPM measures the boot chain into PCR registers.
2. Attestation document — On the running instance, NitroTPM produces a signed attestation document containing the current PCR values and a public key supplied by the caller.
3. Fetch encrypted secret — The instance retrieves the encrypted secret over a public channel. The ciphertext is not sensitive on its own — it can only be opened with the KMS key in the builder account.
4. Decryption request — The instance calls kms:Decrypt against the builder account’s KMS key, attaching the attestation document as the recipient.
5. Policy evaluation — The KMS key policy compares the PCR values in the attestation document to the expected values from the original AMI build. If they match, KMS returns the plaintext wrapped with the public key from step 2, so only the attested instance can unwrap it.

The secret is delivered into the customer’s account but never visible to the account administrator - only to a process running on an instance booted from an AMI the builder built and measured.

EC2 Instance Types with NitroTPM

Even though NitroTPM is widely supported by EC2 instance families, not all families have it, and not all regions have all families. You can find this info from AWS documentation and AWS API using aws ec2 describe-instance-types but it is a bit clumsy. So I wrote a utility script to list NitroTPM supported EC2 families in given region when no arguments are given, or list available instance types when one or multiple families are specified in cmd line.

% export AWS_REGION=eu-north-1
% ./ec2-tpm-support.sh        
c: c5 c5a c5d c5n c6g c6gd c6gn c6i c6in c7a c7g c7gd c7i c7i-flex c8g c8gn
g: g4dn g5 g6 g6e g6f
gr: gr6 gr6f
hpc: hpc6a hpc6id hpc8a
i: i3en i4i i7i i7ie i8g i8ge
inf: inf1 inf2
m: m5 m5d m6g m6gd m6i m6idn m6in m7a m7g m7gd m7i m7i-flex m8g
p: p5 p5e p5en
r: r5 r5b r5d r5dn r5n r6g r6gd r6i r6idn r6in r7a r7g r7gd r7i r8g
t: t3 t4g
u: u7i-12tb u7i-6tb u7i-8tb u7in-24tb
x: x2idn x2iedn x8aedz x8g x8i

% ./ec2-tpm-support.sh c7g m5
c7g: c7g.12xlarge c7g.16xlarge c7g.2xlarge c7g.4xlarge c7g.8xlarge c7g.large c7g.medium c7g.xlarge
m5: m5.12xlarge m5.16xlarge m5.24xlarge m5.2xlarge m5.4xlarge m5.8xlarge m5.large m5.xlarge

Attestable AMI

To use EC2 attestation not just any AMI works. I need to build an attestable AMI. This is an AMI for which I have PCR hash values. PCRs (Platform Configuration Registers) are TPM slots that accumulate hashes of each step in the boot chain. They can’t be written directly — only extended — so the final values are a tamper-evident fingerprint of exactly what booted. KMS can use these values to decide whether to allow decrypting the secret.

To simplify the build process I wrote a template that creates an EC2 instance with either an x86 or arm64 CPU using the latest Amazon Linux 2023 AMI. Instance can be accessed with SSM Session Manager.

AMI build starts by installing required packages …

dnf install -y \
  kiwi-cli python3-kiwi kiwi-systemdeps-core \
  qemu-img veritysetup erofs-utils git cargo \
  kiwi-image-descriptions-examples

Then I build coldsnap utility, which simplifies AMI build process by skipping snapshotting a running instance step. Coldsnap pulls in aws-sdk-ec2, which takes a while to compile and needs 16 GB of RAM. This is the reason build-template is using r7*.large instance.

git clone https://github.com/awslabs/coldsnap.git && (cd coldsnap && cargo install --locked coldsnap)

For my demo case, I’m using the sample appliance description that comes with Kiwi, with minor modifications to make it work better in the demo; I added openssl and tpm2-tools packages and installed amazon-ssm-agent so I can login to instance and show secrets can be decrypted. In real life you would not install ssm-agent as it would let account owner login to instance.

I’m skipping most of the image configuration details in this post but you can find more details in Kiwi documentation and Attestable AMIs & NitroTPM deep dive.

        <!-- NOTE: additional packages for demo -->
        <package name="openssl"/>
        <package name="tpm2-tools"/>
        ...
        <!-- Remove operator access by not installing these packages -->
        <!-- NOTE: SSM agent is installed for demo -->
        <ignore name="openssh-server"/>
        <package name="amazon-ssm-agent"/>
        <ignore name="cloud-init"/>
        <ignore name="cloud-init-cfg-ec2"/>
        <ignore name="update-motd" />
        <ignore name="ec2-instance-connect"/>
        ...       

After modifying /usr/share/kiwi-image-descriptions-examples/al2023/attestable-image-example/appliance.kiwi I was ready to build the AMI and get PCR measurements. If you’re wondering, the PCR measurements below are not from a real AMI — they’re random hex strings ;-)

kiwi-ng system build \
  --description /usr/share/kiwi-image-descriptions-examples/al2023/attestable-image-example \
  --target-dir ./image

cat ./image/pcr_measurements.json
{
  "Measurements": {
    "HashAlgorithm": "SHA384",
    "PCR4": "a3f7c2e891b4d56f0a2c8e1d7b3f4956c0a8e2d1f7b394c5e6a0d2f8b1c4e7a3d5f6c8b0e2a4d7f1c3b5e8a0d2f4c6b9",
    "PCR7": "d2f8a1c4e7b390d5f2c6a8e1b4d7f0c3a5e9b2d6f8c1a4e7b0d3f6c9a2e5b8d1f4c7a0e3b6d9f2c5a8e1b4d7f0c3a5e9",
    "PCR12": "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
  }
}

Next I uploaded the snapshot and register AMI. I learned it the hard way to avoid throttlin and limit upload process using --workers 16 --omit-zero-blocks. If you still get throttling errors from AWS, you can tune it down further.

SNAPSHOT=$(~/.cargo/bin/coldsnap upload --workers 16 --omit-zero-blocks ./image/al2023*.raw)
ARCH=$(m=$(uname -m); [ "$m" = "aarch64" ] && m="arm64"; echo "$m")
aws ec2 wait snapshot-completed --snapshot-ids ${SNAPSHOT} || exit 1
aws ec2 register-image \
  --name "attestable-app-$(date +%Y%m%d-%H%M)" \
  --architecture ${ARCH} \
  --boot-mode uefi \
  --tpm-support v2.0 \
  --ena-support \
  --virtualization-type hvm \
  --root-device-name /dev/xvda \
  --block-device-mappings DeviceName=/dev/xvda,Ebs={SnapshotId=${SNAPSHOT}}

Then I shared the AMI with the AWS test account.

NOTE: Remember to terminate the build instance once the AMI is ready.

KMS Key Policy

Next I need a KMS key to encrypt and decrypt my secret. The focus is on the key policy that must allow all actions from my builder account where the key is created, and allow decryption from remote accounts only if a correct attestation document is provided. I wrote a template that takes PCR4 value (see above) as input and creates a key (and alias) with proper policy. Take note of AttestationKeyAlias in stack output. This is required later when testing.

    - Sid: AllowAttestedDecryptAnyAccount
      Effect: Allow
      Principal: '*'
      Action:
        - kms:Decrypt
        - kms:GenerateDataKey
        - kms:GenerateRandom
      Resource: '*'
      Condition:
        StringEqualsIgnoreCase:
          kms:RecipientAttestation:NitroTPMPCR4:  !Ref PCR4Value
          kms:RecipientAttestation:NitroTPMPCR12: '000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'

Testing

Let’s start by creating an EC2 instance on test account (one that doesn’t have KMS key). This instance must support NitroTPM and can not be T-series because of a limitation in TPM. I wrote a template to help set up an instance with SSM Session Manager access (remember ssm was deployed to AMI for testing). Use AMI ID from build account that was shared earlier.

Encrypting

On build account I use the KMS to encrypt a secret. Take note of encrypted $BLOB. Use key alias id from KMS stack output.

SECRET="Hello World"
B64=$(echo "$SECRET" | base64)
BLOB=$(aws kms encrypt --key-id arn:aws:kms:eu-north-1:111111111111:alias/sample-kms-key --plaintext ${B64} --query CiphertextBlob --output text)

Decrypting

Let’s start by verifying I can decrypt on build account where I encrypted my secret.

aws kms decrypt --key-id arn:aws:kms:eu-north-1:111111111111:alias/sample-kms-key --ciphertext-blob "${BLOB}" --query Plaintext --output text | base64 -d
Hello World

Above works only when called from build account. If I run the same on test account, key policy will prevent decryption with an error.

aws: [ERROR]: An error occurred (AccessDeniedException) when calling the Decrypt operation: User: arn:aws:sts::22222222222:assumed-role/OrganizationAccountAccessRole/user@example.com is not authorized to perform: kms:Decrypt on the resource associated with this ciphertext because the resource does not exist in this Region, no resource-based policies allow access, or a resource-based policy explicitly denies access

When I login to EC2 instance created from attested AMI on test account, I can decrypt the secret with attestation document. First let’s set some variables to hold encrypted data and define the KMS key alias for decryption. As the secret is now encrypted it can be shared over public channels to test instance.

CIPHERTEXT_B64="YOUR_ENCRYPTED_BLOB"
echo "$CIPHERTEXT_B64" | base64 -d > /tmp/ct.bin
KMS_KEY_ID="arn:aws:kms:eu-north-1:111111111111:alias/sample-kms-key"

Then I need an ephemeral key-pair for the decryption process. When called with --recipient, KMS won’t return plaintext over the wire to an attested instance; instead it wraps the plaintext with the public key in the attestation document.

openssl genrsa -out /tmp/priv.pem 2048
openssl rsa -in /tmp/priv.pem -pubout -outform DER -out /tmp/pub.der

I created those into /tmp to keep data in memory even though this isn’t strictly necessary as my AMI boots to read-only filesystem with overlay so all writes are stored to instance memory and not written to EBS volume. The public key is sent to KMS inside the attestation document, where it’s used to encrypt the reply.

ATT=$(/usr/bin/nitro-tpm-attest --public-key /tmp/pub.der | base64 -w0)

NOTE: Attestation document is only valid for 5 minutes after it has been created. If you get error for expired document when decrypting with KMS, just rerun above to refresh.

Now I can decrypt the secret.

aws kms decrypt --key-id "$KMS_KEY_ID" \
  --ciphertext-blob fileb:///tmp/ct.bin \
  --recipient "KeyEncryptionAlgorithm=RSAES_OAEP_SHA_256,AttestationDocument=${ATT}" \
  --query CiphertextForRecipient \
  --output text | base64 -d > /tmp/wrapped.bin

KMS doesn’t return the plaintext directly — the secret arrives encrypted with the ephemeral key-pair I must decrypt with matching private key.

openssl cms -decrypt -inform DER -in /tmp/wrapped.bin -inkey /tmp/priv.pem
Hello World

Summary

The demonstration above showed how to deliver a secret to an EC2 instance in a remote AWS account without exposing it to the account owner. To keep the process secure do remember

• AMI containing the application must have all external access disabled. This includes inbound network access like sshd, SSM agent, and instance connect. Also cloud-init must be disabled to prevent user-data script from adding any access during the instance boot.
• PCR value used in KMS key policy must be kept secret. If it gets compromised, an attacker can create a valid attestation document and gain access to KMS key.
• You can (and should) limit access to KMS key only to your customer accounts. Sharing the key with everyone adds risk if PCR gets compromised.
• The AMI itself should not be considered secret — it’s trivial to create an EBS volume from it and attach it to an attacker-controlled instance. You can embed encrypted secrets in the AMI, but it’s not recommended: you’d need to update the PCR values in KMS policy every time the AMI changes.
• When processing the data you should never write plaintext secrets to disk but keep it in memory so an attacker can’t recover it from a EBS snapshot. Having read-only file system helps to keep data off the disk ;-)
• If you must store a secret data to a disk, use OS block level encryption such as LUKS. AWS native EBS encryption is not secure because it doesn’t support attestation.

You can find templates, scripts and other supporting documents from ec2attestation github repo. There is also further details on tampering the AMI and using encrypted file system. These were too much to be included in a single blog post. Also beware these were written with help from Claude and all details haven’t been verified.

This post demystifies the process of setting up S3 Vector based Knowledge Bases for Amazon Bedrock by walking through complete implementation in AWS CloudFormation. Examining the infrastructure as code will reveal components, dependencies, and permissions required—bringing full transparency to the “magic” that happens when the same is build from Amazon Web Services (AWS) console.

But I will also add some magic ;-) sync mechanism that keeps the knowledge base synchronized with S3 changes automatically. Here’s the complete flow that CloudFormation will build. Click the image to enlarge.

The complete CloudFormation template is available in the aws-bedrock-kb-s3-vector repository.

The Policies

Let’s start from the AWS Identity and Access Management (IAM) roles — this is where the console hides most details.

Bedrock Knowledge Base Role (KnowledgeBaseRole):

AssumeRolePolicyDocument:
  Statement:
    - Effect: Allow
      Principal:
        Service: bedrock.amazonaws.com
      Action: sts:AssumeRole
      Condition:
        StringEquals:
          aws:SourceAccount: !Ref AWS::AccountId
        ArnLike:
          aws:SourceArn: !Sub 'arn:aws:bedrock:${AWS::Region}:${AWS::AccountId}:knowledge-base/*'

This trust policy allows the Bedrock service to assume the role, but ONLY:

• From your AWS account (prevents cross-account access)
• When acting on behalf of a knowledge base resource (not other Bedrock features)

Knowledge Base Permissions:

Policies:
  - PolicyName: S3Access
    PolicyDocument:
      Statement:
        # Read from data source bucket
        - Effect: Allow
          Action:
            - s3:GetObject
            - s3:ListBucket
          Resource:
            - !Sub '${DataSourceBucket.Arn}'
            - !Sub '${DataSourceBucket.Arn}/*'

        # Write to vector storage bucket
        - Effect: Allow
          Action:
            - s3:PutObject
            - s3:GetObject
            - s3:ListBucket
          Resource:
            - !Sub '${VectorStorageBucket.Arn}'
            - !Sub '${VectorStorageBucket.Arn}/*'

        # Invoke embedding model
        - Effect: Allow
          Action:
            - bedrock:InvokeModel
          Resource: !Sub 'arn:aws:bedrock:${AWS::Region}::foundation-model/${EmbeddingModelId}'

NOTE:

• Data source: Read-only access
• Vector storage: Read/write access
• Embedding model: Invoke permission scoped to specific model

AWS Lambda Execution Role (AutoSyncLambdaRole):

Policies:
  - PolicyName: BedrockIngestionAccess
    PolicyDocument:
      Statement:
        - Effect: Allow
          Action:
            - bedrock-agent:StartIngestionJob
            - bedrock-agent:ListIngestionJobs
          Resource:
            - !Sub 'arn:aws:bedrock:${AWS::Region}:${AWS::AccountId}:knowledge-base/${KnowledgeBase}'

NOTE: Lambda can only interact with this specific knowledge base.

The Chunking Strategies

The console shows you chunking options in a simple dropdown. But each strategy requires different configuration resource type in CloudFormation. This makes switching the strategy bit more complex than it first seems.

Fixed-Size Chunking

ChunkingConfiguration:
  ChunkingStrategy: FIXED_SIZE
  FixedSizeChunkingConfiguration:
    MaxTokens: 300              # Max chunk size
    OverlapPercentage: 20       # 20% overlap between chunks

How it works:

• Document is tokenized using the embedding model’s tokenizer
• Split into chunks of 300 tokens
• Each chunk overlaps with the next by 20%
• This overlap ensures context isn’t lost at chunk boundaries

Hierarchical Chunking

ChunkingConfiguration:
  ChunkingStrategy: HIERARCHICAL
  HierarchicalChunkingConfiguration:
    LevelConfigurations:
      - MaxTokens: 1500          # Parent chunk size
      - MaxTokens: 300           # Child chunk size
    OverlapTokens: 60            # Absolute overlap (not percentage)

How it works:

• Document split into parent chunks (1500 tokens)
• Each parent split into child chunks (300 tokens)
• Child chunks overlap by 60 tokens
• Both parent and child chunks are embedded and stored
• Retrieval can leverage both granularities

This preserves document structure better than fixed-size. When searching, you get:

• High-precision matches from child chunks
• Broader context from parent chunks

Semantic Chunking

ChunkingConfiguration:
  ChunkingStrategy: SEMANTIC
  SemanticChunkingConfiguration:
    MaxTokens: 300                           # Maximum chunk size
    BufferSize: 1                            # Sentences to buffer
    BreakpointPercentileThreshold: 95        # Similarity threshold

How it works (this is the most sophisticated):

• Document split into sentences
• Sentences embedded using the model
• Calculate cosine similarity between adjacent sentence embeddings
• When similarity drops below the 95th percentile threshold, create a chunk boundary
• This creates “natural” chunks based on semantic shifts in content
• BufferSize: 1 means look at 1 sentence on either side for context

The result: chunks that respect topic boundaries rather than arbitrary token counts. A long paragraph discussing a single concept stays together, while a transition to a new topic creates a boundary even mid-paragraph.

No Chunking

ChunkingConfiguration:
  ChunkingStrategy: NONE

The entire document is embedded as one unit. This is practical for:

• Short documents
• When you need complete document context
• When document structure itself is important

Supported File Formats

Before diving into the implementation details, it’s important to understand what types of documents you can use with Bedrock Knowledge Bases. The service supports following formats at time of writing:

Text-Based Documents:

• PDF (.pdf) - with support for multimodal parsing of figures, charts, and tables
• Microsoft Word (.doc, .docx)
• Microsoft Excel (.xls, .xlsx)
• Plain text (.txt)
• Markdown (.md)
• HTML (.html)
• CSV (.csv) - supports one content field with metadata fields

Multimodal Content:

• Images: JPEG (.jpeg) and PNG (.png) - maximum 3.75 MB per file
• Audio files (when using S3 or custom data sources)
• Video files (when using S3 or custom data sources)

File Size Limits:

• General documents: maximum 50 MB per file
• Image files: maximum 3.75 MB per file

For the complete and up-to-date list of supported formats and configurations, refer to the Prerequisites for your Amazon Bedrock knowledge base data documentation.

Knowledge Base Sync

Here’s where we go beyond what the console offers. The console doesn’t provide automatic synchronization—you must manually trigger ingestion jobs. When documents change frequently, this is impractical. Instead you should be able to upload documents to S3 and after a short moment knowledge base is updated automatically. And while doing this, avoid running sync for every document separately as update can be the most costly operation as it triggers re-embedding of documents, which incurs model invocation costs.”

The Event Flow in Detail

Step 1: S3 Event Configuration

First, the S3 bucket must enable Amazon EventBridge notifications:

DataSourceBucket:
  Type: AWS::S3::Bucket
  Condition: CreateNewBucket
  Properties:
    NotificationConfiguration:
      EventBridgeConfiguration:
        EventBridgeEnabled: true

When enabled, S3 emits events to EventBridge for:

• Object Created (PutObject, CompleteMultipartUpload, CopyObject, POST object)
• Object Deleted (DeleteObject, DeleteObjects)

Step 2: EventBridge Rule

S3EventRule:
  Type: AWS::Events::Rule
  Condition: AutoSyncEnabled
  Properties:
    EventPattern:
      source:
        - aws.s3
      detail-type:
        - Object Created
        - Object Deleted
      detail:
        bucket:
          name:
            - !If [CreateNewBucket, !Ref DataSourceBucket, !Ref ExistingDataSourceBucket]
    Targets:
      - Arn: !GetAtt SyncEventQueue.Arn
        Id: SQSTarget

This rule filters ALL S3 events in your account to only:

• Events from S3 (source: aws.s3)
• Creation or deletion events (detail-type)
• From your specific data source bucket (detail.bucket.name)

Events matching this pattern are sent to the SQS queue.

Step 3: Amazon Simple Queue Services (SQS) Batching Configuration

SyncEventQueue:
  Type: AWS::SQS::Queue
  Condition: AutoSyncEnabled
  Properties:
    MessageRetentionPeriod: 86400              # 1 day
    VisibilityTimeout: 360                     # 6 minutes

The Lambda event source mapping configures batching:

Events:
  SQSEvent:
    Type: SQS
    Properties:
      Queue: !GetAtt SyncEventQueue.Arn
      BatchSize: 100                           # Process up to 100 events
      MaximumBatchingWindowInSeconds: 60       # Wait up to 60s to accumulate

This batching is critical for efficiency:

• If you upload 1 file, Lambda triggers after 60 seconds with 1 event
• If you upload 50 files within 60 seconds, Lambda triggers once with 50 events
• If you upload 100+ files, Lambda triggers with 100 events (max batch)

Step 4: AWS Lambda Processing Logic

The Lambda function implements intelligent sync management. Let’s examine the key parts:

def lambda_handler(event, context):
    num_records = len(event.get('Records', []))
    print(f"Received {num_records} event(s)")

    # Check if there's already an ingestion job running
    response = bedrock_agent.list_ingestion_jobs(
        knowledgeBaseId=KNOWLEDGE_BASE_ID,
        dataSourceId=DATA_SOURCE_ID,
        maxResults=1
    )

    ingestion_jobs = response.get('ingestionJobSummaries', [])

    if ingestion_jobs:
        latest_job = ingestion_jobs[0]
        status = latest_job.get('status')

        if status in ['STARTING', 'IN_PROGRESS']:
            # Sync is running - schedule a follow-up
            sqs.send_message(
                QueueUrl=SYNC_QUEUE_URL,
                MessageBody=json.dumps({
                    'type': 'retry',
                    'reason': 'sync_in_progress',
                    'timestamp': datetime.now(timezone.utc).isoformat()
                }),
                DelaySeconds=300  # Retry after 5 minutes
            )
            return {'statusCode': 200, 'body': 'Sync deferred'}

Why check for running jobs? Bedrock ingestion jobs can take several minutes. If we start a new job while one is running, we waste resources and create race conditions.

Why send a retry message? Files added during a sync won’t be in the current job. The retry message ensures a follow-up sync happens after the current one completes.

Why 5-minute delay? This prevents busy-waiting. The Lambda won’t be triggered by the retry message until 5 minutes pass, by which time most syncs have completed.

    # No running job - check if queue needs purging
    queue_attrs = sqs.get_queue_attributes(
        QueueUrl=SYNC_QUEUE_URL,
        AttributeNames=['ApproximateNumberOfMessages',
                        'ApproximateNumberOfMessagesDelayed']
    )

    total_messages = (
        int(queue_attrs['Attributes']['ApproximateNumberOfMessages']) +
        int(queue_attrs['Attributes']['ApproximateNumberOfMessagesDelayed'])
    )

    if total_messages > 0:
        try:
            sqs.purge_queue(QueueUrl=SYNC_QUEUE_URL)
            print(f"Purged {total_messages} messages from queue")
        except ClientError as e:
            if 'PurgeQueueInProgress' in str(e):
                print("Queue purge already in progress, continuing with sync")

Why purge the queue? This is non-obvious but critical. When Bedrock starts an ingestion job, it scans the ENTIRE S3 bucket. Individual S3 event messages telling us “file X was added” become redundant — we’re processing ALL files anyway.

Without purging:

• Queue accumulates thousands of S3 event messages
• Retry messages keep accumulating during long syncs
• Lambda gets triggered repeatedly with redundant batches

With purging:

• Before each sync, we clear all queued messages
• The sync processes everything in the bucket
• Clean slate for the next round of events

Why check message count first? SQS enforces a 60-second cooldown between purge operations. If we purge an empty queue, we can’t purge again for 60 seconds. By checking first, we only purge when necessary.

    # Start the ingestion job
    response = bedrock_agent.start_ingestion_job(
        knowledgeBaseId=KNOWLEDGE_BASE_ID,
        dataSourceId=DATA_SOURCE_ID
    )

    job_id = response['ingestionJob']['ingestionJobId']
    print(f"Started ingestion job: {job_id}")

    return {
        'statusCode': 200,
        'body': json.dumps({
            'message': 'Ingestion job started',
            'jobId': job_id
        })
    }

The ingestion job processes:

• All objects in the S3 data source bucket
• Applies the configured chunking strategy
• Generates embeddings via the embedding model
• Stores vectors in the vector storage bucket
• Updates the searchable index

Concurrency Control

AutoSyncFunction:
  Type: AWS::Serverless::Function
  Properties:
    ReservedConcurrentExecutions: 1    # Only 1 instance can run at a time

This setting is critical. Without it multiple batches of S3 events could trigger multiple Lambda instances. Each instance might call list_ingestion_jobs and see “no running job”. Multiple instances start duplicate ingestion jobs simultaneously causing wasted resources and potential race conditions.

Edge Cases Handled

Fast consecutive syncs

If your documents are small and ingestion completes in <60 seconds:

• New files trigger Lambda
• Check finds no running job
• Check message count (might be 0 if queue was just purged)
• Skip purge (avoids 60-second cooldown violation)
• Start new sync

Purge failure during long sync

If files keep arriving during a 5-minute sync:

• Multiple Lambda invocations send retry messages
• Eventually, retry delay expires and Lambda runs
• Tries to purge queue
• Purge fails due to SQS’s 60-second cooldown from previous purge.
• Lambda logs warning and continues anyway
• Sync still succeeds (graceful degradation)

Sync completes during retry delay

Upload 100 files, sync takes 3 minutes:

• During those 3 minutes, new S3 events trigger Lambda
• Each batch schedules a 5-minute retry
• After 3 minutes, sync completes
• At 5 minutes, first retry message triggers Lambda
• No job running, queue is purged, new sync starts
• Subsequent retry messages are purged before they trigger (redundant but harmless)

Deployment

Let’s walk through deployment and observe what gets created.

Deploy with New Bucket

aws cloudformation deploy \
  --template-file template.yaml \
  --stack-name my-knowledge-base \
  --parameter-overrides \
    KnowledgeBaseName=MyKnowledgeBase \
    ChunkingStrategy=SEMANTIC \
    EnableAutoSync=true \
    AutoSyncDelaySeconds=60 \
  --capabilities CAPABILITY_NAMED_IAM

The CAPABILITY_NAMED_IAM flag is required because the template creates IAM roles with custom names. This is CloudFormation’s safety mechanism—you must explicitly acknowledge that you’re creating IAM resources.

Watch the stack events:

aws cloudformation describe-stack-events \
  --stack-name my-knowledge-base \
  --max-items 20

You’ll see resources created in dependency order

Upload Documents and Observe

# Get bucket name from stack output
BUCKET_NAME=$(aws cloudformation describe-stacks \
  --stack-name my-knowledge-base \
  --query 'Stacks[0].Outputs[?OutputKey==`DataSourceBucketName`].OutputValue' \
  --output text)

# Upload a document
aws s3 cp my-document.pdf s3://${BUCKET_NAME}/

Now watch the auto-sync in action:

# Get Lambda function name
FUNCTION_NAME=$(aws cloudformation describe-stacks \
  --stack-name my-knowledge-base \
  --query 'Stacks[0].Outputs[?OutputKey==`AutoSyncFunctionArn`].OutputValue' \
  --output text | cut -d: -f7)

# Tail logs
aws logs tail /aws/lambda/${FUNCTION_NAME} --follow

You’ll see output like:

2026-01-18T13:45:23.456Z START RequestId: abc-123-def
2026-01-18T13:45:23.789Z Received 1 event(s)
2026-01-18T13:45:24.123Z Latest ingestion job xyz-789 status: COMPLETE
2026-01-18T13:45:24.456Z Queue has 0 messages, skipping purge
2026-01-18T13:45:25.789Z Started ingestion job: new-job-456
2026-01-18T13:45:25.890Z END RequestId: abc-123-def

Check the ingestion job status:

KB_ID=$(aws cloudformation describe-stacks \
  --stack-name my-knowledge-base \
  --query 'Stacks[0].Outputs[?OutputKey==`KnowledgeBaseId`].OutputValue' \
  --output text)

DS_ID=$(aws cloudformation describe-stacks \
  --stack-name my-knowledge-base \
  --query 'Stacks[0].Outputs[?OutputKey==`DataSourceId`].OutputValue' \
  --output text)

aws bedrock-agent list-ingestion-jobs \
  --knowledge-base-id ${KB_ID} \
  --data-source-id ${DS_ID} \
  --max-results 5

Output shows the job lifecycle:

{
  "ingestionJobSummaries": [
    {
      "ingestionJobId": "new-job-456",
      "status": "IN_PROGRESS",
      "startedAt": "2026-01-18T13:45:25Z",
      "statistics": {
        "numberOfDocumentsScanned": 1,
        "numberOfNewDocumentsIndexed": 0,
        "numberOfModifiedDocumentsIndexed": 1
      }
    }
  ]
}

Wait a few minutes and query again—status changes to COMPLETE.

Lessons from Building This

The IAM Trust Policy Conditions Matter

The Condition in the Knowledge Base role’s trust policy is important.

Condition:
  StringEquals:
    aws:SourceAccount: !Ref AWS::AccountId

Without this, the role could potentially be assumed by Bedrock from ANY AWS account. Always scope trust policies to your account and specific resource ARNs.

EventBridge Must Be Explicitly Enabled

S3 doesn’t emit EventBridge events by default—you must enable it:

NotificationConfiguration:
  EventBridgeConfiguration:
    EventBridgeEnabled: true

For existing buckets, you must enable this manually as CloudFormation can not modify resources that are not part of the stack.

Queue Purging Prevents Message Accumulation

Without the purge logic, the queue accumulated thousands of messages during testing. Each S3 upload creates an event, and during long syncs, retry messages keep piling up. The purge ensures clean state.

Concurrency Limit is Critical

When I initially tested without ReservedConcurrentExecutions: 1, multiple Lambda instances started duplicate ingestion jobs. The concurrency limit ensures sequential execution and prevents race conditions.

Semantic Chunking Requires Careful Tuning

Semantic chunking with default thresholds created very large chunks (800+ tokens) for some documents. Tuning BreakpointPercentileThreshold from 95 to 90 created more appropriately-sized chunks. The “right” value depends on your document characteristics.

Eventual Consistency

After starting an ingestion job, it takes a few seconds for list_ingestion_jobs to reflect the new job. During that window, another Lambda invocation might see “no running job” and start a duplicate. The concurrency limit prevents this race condition.

With Little Help from LLM

Building this template wasn’t a solo effort — I used an LLM to accelerate development. The process revealed both the power and limitations of AI-assisted infrastructure coding.

The Initial Prompt

I started simple:

“Create CloudFormation template that creates bedrock knowledge base using s3 vector and all supporting resources and configurations required”

The LLM generated a complete template within seconds. Impressive… until I looked closer at the generated code and comments. The solution was using OpenSearch, not S3 Vector storage. This is typical with new technology. S3 Vector support for Bedrock Knowledge Bases became generally available in December 2024—very recent. The LLM’s training data contains far more examples of the previous generation solution (OpenSearch Serverless) than the newer S3 Vector approach. When you ask for “bedrock knowledge base,” the model defaults to the pattern it’s seen most often and might ignore some of your instructions.

The Correction Cycle

I followed up with a clarification:

“I don’t want to use opensearch but S3 Vector bucket”

The LLM responded enthusiastically: “This is much simpler and uses fully managed services without the need for OpenSearch!” That was exactly my point—S3 Vector eliminates the complexity of managing OpenSearch clusters. But when I validated the template with aws cloudformation validate-template, problems emerged. The storage configuration was incomplete.

Another iteration:

“Storage configuration should be S3_VECTOR type and include configuration”

The classic LLM response: “You’re absolutely right! …”

After this correction, I had a working first version. The entire process took maybe 15 minutes — much faster than writing from scratch, even for someone familiar with CloudFormation.

What the LLM Got Right

The LLM excelled at:

• Boilerplate structure: Parameters, conditions, outputs etc.
• IAM role scaffolding: Trust policies and basic permission structure
• Resource dependencies: Using !Ref and !GetAtt correctly
• Common patterns: S3 bucket configurations, Lambda function basics

These are the tedious parts of CloudFormation development. Having them generated instantly let me focus on the interesting problems: auto-sync logic and edge case handling.

What Required Human Expertise

The LLM struggled with:

• New service features: S3 Vector configuration details weren’t in training data
• Edge cases: Queue purging logic, concurrency limits, eventual consistency handling
• Production patterns: Error handling, retry logic, graceful degradation

The auto-sync mechanism—the most valuable part of this template—required human design. The LLM could generate a Lambda function that calls start_ingestion_job, but it didn’t consider on it’s own what happens when multiple S3 events arrive simultaneously, how to prevent duplicate ingestion jobs, or do optimization like purging the queue or setting concurrency limit.

Conclusion

This implementation achieves two complementary goals:

Demystifying the Foundation

The AWS console makes it easy to create a Bedrock Knowledge Base—a few clicks and you’re done. But understanding what’s actually happening beneath that interface is essential for:

• Production deployments: You need to know IAM roles, permissions, and dependencies
• Troubleshooting: When something breaks, you need to know which component failed
• Customization: You can’t extend what you don’t understand
• Cost optimization: You need to know what resources you’re paying for
• Security: You must understand what permissions are granted and why

By building this as CloudFormation, we’ve revealed every component, every permission, every dependency. The “magic” of the console becomes transparent, understandable infrastructure that you can reason about, modify, and confidently deploy to production.

Adding the Right Kind of Magic

The auto-sync mechanism adds automation that works invisibly for end users (documents just sync automatically), remains transparent to developers (every trigger is explicit in code) and scales to zero when there are no document to process.

Whether you’re building a documentation search system, customer support chatbot, or legal research tool, this template provides a solid foundation: transparent enough to understand completely, automated enough to eliminate operational toil.

Brief History of AWS NAT Solutions

First there were NAT instances. You’d spin up an Amazon Elastic Compute Cloud (EC2) instance with a special Amazon Machine Image (AMI), disable source/destination checks, point your route tables at it, and hope it didn’t fall over. Instance sizing, high availability, patching, monitoring - all yours to manage. It worked, but it was a high maintenance solution.

Late 2015 AWS announced Amazon VPC NAT Gateway. Fully managed, highly available, no patching required. Create it in a public subnet, add a route, job done.

In 2021 AWS added Private NAT Gateway to the mix. This one isn’t for Internet access but for connecting VPCs and other networks when you’ve got overlapping CIDR ranges. I wrote about how this enables more agile networking patterns that weren’t really feasible before.

And now, in November 2025, AWS announced Amazon VPC Regional NAT Gateway. Unlike traditional NAT Gateway that lives in a single Availability Zone (AZ), the regional variant spans multiple AZs. AWS handles the high availability and scaling for you.

Simpler

AWS services come in 3 flavours - zonal, regional and global. Zonal services live in a specific availability zone, whilst regional services span the entire region. And here’s the thing - they’re configured with network components that match their scope.

Classic NAT Gateway is a zonal service. You create it in a specific AZ, attach it to a subnet (which is also zonal), and it serves traffic for that AZ. If you want high availability, you must deploy another NAT Gateway in another AZ, with another subnet, and manage separate route tables for each AZ’s private subnets. It works, but you’re thinking about AZ topology every step.

Regional NAT Gateway makes this much simpler. It’s a regional service, so it attaches directly to your VPC that is a regional network construct. One NAT Gateway for the entire VPC, regardless of how many AZs you are using. Routing isn’t specific to availability zone any longer but works the same way as public subnets with an Internet Gateway. Just point the default route to the regional NAT Gateway and you’re done.

This is particularly nice if you don’t need any inbound connections from the Internet. Traditionally you’d still need public subnets for the NAT Gateways even though nothing was accepting inbound traffic. Now you can drop public subnets entirely and run a pure private subnet architecture. Simpler network topology, fewer moving parts.

Note there is one new component in above setup - the Regional NAT Route Table. It is created automatically with a pre-configured route to the Internet Gateway. But you can customize it to route to middleboxes like firewalls, enabling traffic inspection for outbound flows. See AWS blog post for more details.

NOTE: If you read my previous post about Lambda and IPv6, you might notice this looks very similar to IPv6 Egress Only Gateway. But you shouldn’t think Regional NAT gateway as IPv4 equivalent to IPv6 Egress gateway because it still does address translation that isn’t required when using IPv6.

More Secure

Public subnets are where things with public IPs live - load balancers, bastion hosts, anything that needs inbound connections from the Internet. If the only thing you have in public subnets are NAT gateways, then you don’t need them any more after switching to regional NAT gateway.

This matters for security because now you can’t expose something to the Internet by accident if there’s nowhere to expose it. No public subnets means no path for resources to get public IPs. Someone wants to make their database publicly accessible? Not happening - there is no subnet configuration that would allow it. It’s defence in depth through architectural constraint.

However, there is a small catch. You still need an Internet Gateway attached to your VPC. Regional NAT Gateway needs it for outbound connectivity, even though you’re not routing any traffic to it from your subnets. As the AWS documentation notes, when you create a regional NAT Gateway, AWS “automatically creates a route table for it, which comes with a pre-configured route to the internet gateway.” So the IGW must exist, but your private subnets never point routes to it.

Cheaper (?)

If you’re running traditional NAT Gateways in three AZs and thinking “brilliant, I’ll replace all three with one regional NAT Gateway and cut my costs by two-thirds” - this isn’t how it works :-(

Regional NAT Gateway pricing is per AZ, just like traditional NAT Gateway. The difference is it’s automatically scaling. When you have resources in three AZs using the regional NAT Gateway, you’re charged for three AZs.

The clever bit is what happens when your workloads change. Say you’re running a scheduled job that only uses two AZs during off-peak hours. So far you would be still paying for all three NAT Gateways. Regional NAT Gateway does adjust automatically. When no traffic flows through an AZ, you stop paying for that AZ. When traffic resumes, pricing scales back up.

So you’re not saving money if you’re running steady workloads across multiple AZs. You’re getting operational simplicity - one resource to manage instead of three - but you’re paying the same hourly charges. The cost benefit comes if your workloads are dynamic, expanding and contracting across AZs throughout the day.

There is also another way Regional NAT Gateway can save you money (and improve your resilience to AZ failures) - it prevents cross-AZ data transfers from misconfiguration. With traditional NAT Gateways, if you accidentally made route table in AZ-A to point to a NAT Gateway in AZ-B, you were paying cross-AZ data transfer charges on top of everything else.

Think of it this way - you’re trading the complexity of managing multiple NAT Gateways for the flexibility of automatic AZ scaling. The price stays the same for equivalent usage, but you’re getting operational simplicity and protection from costly misconfigurations.

Small Print

Scaling Out

Regional NAT Gateway can take up to 60 minutes to expand to new AZs. This isn’t at speed of EC2 Auto Scaling launching instances in 2-3 minutes or functions scaling in seconds, but it shouldn’t be a major issue because

Until this expansion is complete, the relevant traffic from this resource is processed across zones by your regional NAT Gateway in one of the existing Availability Zones.

Pricing

Regional NAT Gateway charges per AZ, but how is it detected if an AZ is “in use”? The documentation is bit vague but it seems to be based on presence of an elastic network interface (ENI) rather than traffic detection.

regional NAT gateway detects the presence of an network interface(ENI) in that Availability Zone and automatically expands to that zone. Similarly, the NAT Gateway contracts from the Availability Zone that has no active workloads.

Expansion to new AZ is said to typically take 15-20 minutes but can take up to 60 min. Documentation doesn’t say, but I would expect scale in from unused AZs to follow similar process. And if you stop your instance but ENI remains, I would expect that AZ remain “in use” from NAT point-of-view.

Public NAT Only

At launch regional NAT gateway does not support private NAT. For private NAT gateways you should continue to use classic zonal NAT gateways.

Summary

Regional NAT Gateway is what NAT Gateway should have been from the start.

Architectural & Operational simplicity - One NAT Gateway per VPC instead of one per AZ. No more managing multiple resources, no more AZ-specific route tables to keep synchronized. When workloads expand to new AZs, Regional NAT Gateway handles it automatically. For purely outbound architectures, you can eliminate public subnets entirely. Simpler network topology, fewer components to secure and maintain.

Protection from mistakes - No risk of misconfigured route tables causing cross-AZ data transfer charges. Traffic automatically routes through the NAT Gateway capacity in the same AZ as your resources.

Better capacity - Up to 32 IP addresses per AZ compared to 8 for classic NAT Gateway. Automatic IP scaling when connections approach limits. Better suited for shared NAT scenarios and high-connection workloads.

Cost efficiency for dynamic workloads - Only pay for AZs in-use. If your workloads scale across AZs throughout the day, you stop paying for idle NAT capacity.

Advanced routing options - NAT gateway route table enables traffic inspection through middleboxes like firewalls, something that required complex routing workarounds with classic NAT Gateway.

The pricing is the same for equivalent usage, so you’re not paying extra for these benefits. You’re getting operational simplicity, architectural flexibility, and protection from common misconfigurations at no additional cost.

There’s no compelling reason to deploy new classic NAT Gateways. Regional NAT Gateway is available in all commercial AWS regions. If you’re designing new VPCs or refactoring existing ones, make your NAT regional. And if you’re running classic NAT Gateways in production, start planning the migration. The architecture is simpler, the operations are easier, and you’re eliminating entire classes of configuration mistakes that cost money and cause outages.

That’s exactly what we’re doing today with Amazon Web Services (AWS) Lambda functions in Amazon Virtual Private Cloud (VPC).

VPC Dealers - The Electric Conversion

In this episode I’ve got a Lambda function that’s been deployed into a VPC for years. It needs VPC access to reach private resources like databases or internal APIs. But it also needs to call AWS APIs and maybe some external services on the Internet. The classic solution? Deploy to VPC private subnets with a route to Amazon VPC NAT Gateway (NAT) in a public subnet.

It works. No question about it. But here’s the thing - NAT gateway charges you by the hour, every hour, whether you’re using it or not. For a Lambda function that might only run a few times a day, you’re paying for 24/7 infrastructure. It’s like leaving the engine running in the car park all day just in case you need to pop to the shops.

And before you ask - no, you can’t just deploy Lambda to a public subnet. Lambda functions in VPC never get a public IP address, even when deployed to public subnets. They always need NAT gateway or some other route to reach the Internet.

Classic Solutions

Now, there are traditional ways to reduce NAT gateway costs. You could use VPC endpoints for AWS services like Amazon Simple Storage Service (S3) and Amazon DynamoDB to bypass the NAT gateway entirely for that traffic. You could consolidate multiple NAT gateways into one if you’re running several. You could even replace NAT gateway with a NAT instance on a small Amazon Elastic Compute Cloud instance that you manage yourself.

All valid approaches. But they’re a bit like tuning the old engine - you’re still paying for something that sits there burning money even when idle. VPC endpoints help but only for specific AWS services. NAT instances save money but add operational overhead.

Why NAT Gateway Exists

Let’s step back and understand what NAT gateway actually does. With IPv4, you’ve got a limited pool of public IP addresses. Your VPC uses private IP addresses internally (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) that aren’t routable on the Internet. When your Lambda needs to reach the Internet, NAT gateway translates that private IP to a public IP address, sends the request out, receives the response, and translates it back to your private IP. It’s Network Address Translation (NAT). The gateway sits in a public subnet with an Elastic IP, acting as a middleman for all your outbound traffic.

IPv6 Changes Everything

Here’s the thing about IPv6 - there’s no address shortage. IPv6 has 2^128 (that’s 39 digits!) Every resource in your VPC can have a globally unique, routable IPv6 address. No translation needed.

When your Lambda function has an IPv6 address it can talk directly to the Internet. No middleman. No NAT gateway sitting there charging you by the hour. You just need an egress-only internet gateway. Like a regular Amazon VPC Internet Gateway, it’s a highly available regional service that scales automatically. It allows outbound IPv6 traffic while blocking inbound connections - keeping your Lambda in private subnet just as secure as it was behind the traditional NAT gateway.

Electric Conversion

So what does this conversion actually look like? I’ve put together a AWS CloudFormation template that shows the key changes. Let’s walk through what’s different from a traditional IPv4 VPC setup.

First, I need to associate an IPv6 CIDR block with the VPC. AWS provides these for free, no need to worry about paying for public IPs like with IPv4.

IPv6CidrBlock:
  Type: AWS::EC2::VPCCidrBlock
  Properties:
    VpcId: !Ref VPC
    AmazonProvidedIpv6CidrBlock: true

Then subnets need IPv6 CIDR blocks. Notice AssignIpv6AddressOnCreation - this ensures Lambda ENIs get IPv6 addresses automatically.

PrivateSubnet1:
  Type: AWS::EC2::Subnet
  DependsOn: IPv6CidrBlock
  Properties:
    VpcId: !Ref VPC
    CidrBlock: 10.0.0.0/25
    Ipv6CidrBlock: !Select [0, !Cidr [!Select [0, !GetAtt VPC.Ipv6CidrBlocks], 2, 64]]
    AvailabilityZone: !Select [0, !GetAZs '']
    AssignIpv6AddressOnCreation: true

Instead of a NAT gateway, I create an Amazon VPC Egress-Only Internet Gateway. This is attached to VPC like Internet gateway. No public subnets required.

EgressOnlyInternetGateway:
  Type: AWS::EC2::EgressOnlyInternetGateway
  Properties:
    VpcId: !Ref VPC

Route table points IPv6 traffic (::/0) to the Egress-only gateway instead of a NAT gateway.

PrivateRouteIPv6:
  Type: AWS::EC2::Route
  Properties:
    RouteTableId: !Ref PrivateRouteTable
    DestinationIpv6CidrBlock: ::/0
    EgressOnlyInternetGatewayId: !Ref EgressOnlyInternetGateway

Security groups need to allow IPv6 egress traffic.

LambdaSecurityGroup:
  Type: AWS::EC2::SecurityGroup
  Properties:
    GroupDescription: Security group for Lambda function
    VpcId: !Ref VPC
    SecurityGroupEgress:
      - IpProtocol: -1
        CidrIp: 0.0.0.0/0
      - IpProtocol: -1
        CidrIpv6: ::/0

And finally, the Lambda function needs one crucial setting - Ipv6AllowedForDualStack: true. This tells Lambda to use IPv6 for outbound connections:

MyLambdaFunction:
  Type: AWS::Serverless::Function
  Properties:
    VpcConfig:
      SubnetIds:
        - !Ref PrivateSubnet1
        - !Ref PrivateSubnet2
      SecurityGroupIds:
        - !Ref LambdaSecurityGroup
      Ipv6AllowedForDualStack: true

That’s it. No public subnets. No NAT gateway. No Elastic IPs. Just private subnets with IPv6 and an Egress-only gateway. Same functionality, zero ongoing infrastructure costs.

Other Considerations

Almost all AWS services support IPv6 these days. Your Lambda can talk to S3, DynamoDB, Amazon API Gateway, Amazon Simple Queue Service (SQS), Amazon Simple Notification Service (SNS) - all over IPv6. No issues there. And surprisingly, many Internet services also speak IPv6. Major CDNs, cloud providers, and APIs have had IPv6 support for years.

And if you’re exposing your own services to IPv4 clients through an Application Load Balancer or Amazon CloudFront, dual-stack mode lets the load balancer handle IPv4 clients connecting to your IPv6 backend.

Your VPC resources can still use their private IPv4 addresses for internal communication. This is dual-stack mode - both IPv4 and IPv6 work simultaneously. Your Lambda talks to Amazon Relational Database Service (RDS) using the same private IPv4 address it always has. Only outbound Internet traffic uses IPv6. This makes the conversion backward compatible and low risk.

Now, what about services that only speak IPv4? AWS offers NAT64 and DNS64 services that translate between IPv6 and IPv4. But here’s the catch - NAT64 gateway has the similar hourly charges as a regular NAT gateway. You’re back to paying for infrastructure. If you need to reach IPv4-only services, you might be better off keeping a traditional NAT gateway.

The sweet spot for this approach is Lambda functions that primarily call AWS services and APIs that support IPv6. For those workloads, you can eliminate NAT gateway costs entirely while keeping everything else exactly as it was.

Summary

In this episode of VPC Dealers we took a Lambda function running in a traditional VPC with NAT gateway and converted it to use IPv6 with an egress-only Internet gateway. Same functionality, same security posture, but without the hourly infrastructure charges. Just like Ant’s electric Maserati - modern technology under the same classic bonnet.

If you’ve got Lambda functions in VPCs that are costing you money in NAT gateway charges, give this approach a try. The template is on GitHub ready to deploy.

Until next time. Ta-Da!

What is the best way to debug AWS Lambda functions?

Answer was a long list of advice including test events, logging, AWS X-Ray, and running functions locally on your laptop using sam local or LocalStack. While all great advice, all of those require rather complex setup and don’t really let you debug code in its native environment (to avoid works-on-my-machine -syndrome). While setup could be justified for full-time dev(ops), it isn’t really feasible when you are on-duty and suddenly pulled into debugging a function that you have never seen before.

Fortunately there is a way to debug ad-hoc lambda functions running in their native environment, with minimal setup.

Lambda Remote Debugging

This is the new feature my chatbot wasn’t aware of (yet)

AWS Lambda now supports remote debugging in Visual Studio Code (VS Code), enabling developers to debug their Lambda functions running in the cloud directly from their local IDE. With this new capability, developers can use familiar debugging tools like breakpoints, variable inspection, and step-through debugging with functions deployed in the cloud without modifying their existing development workflow, thus accelerating their serverless development process.

Wow! Also the list of pre-requirements for remote debugging doesn’t look too bad for casual debugger, especially when compared to any of above options.

• VS Code
• AWS Toolkit
• AWS Credentials

Getting started

To get started you just browse to AWS Lambda Console, find your function and click.

This will open the function in your VS Code and if you don’t have recent version of AWS Toolkit, it will take care of that too. For now AWS Console can only open VS Code, but actually any VS Code derivative like Kiro will work. Important part is AWS Toolkit. You can also open debugging session from your editor and select the function from AWS Toolkit Explorer. By default you will see only the default region configured for your credentials. If you don’t see the function you were looking for, verify you have enabled the correct region.

I’d also recommend you to spend 5 minutes and watch Eric Johnson’s demo below.

Debug this

Sometimes seeing is believing, but nothing beats getting hands-on. For this I created a template that creates a sample python function with a bug. You can get the template from github -repo and deploy it with AWS CLI or from AWS CloudFormation console. Code and minimal infrastructure are both included in single file for easy deployment.

% aws cloudformation deploy --template-file template-bug.yaml --stack-name lambda-debug-demo --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM CAPABILITY_NAMED_IAM

And then get the function URL from stack outputs using AWS CLI or from AWS CloudFormation console

% aws cloudformation describe-stacks --stack-name lambda-debug-demo --query 'Stacks[0].Outputs[].[Description,OutputValue]'
[
    [
        "Lambda Function URL",
        "https://LAMBDA-FUNCTION-URL.lambda-url.eu-west-1.on.aws/"
    ],
    [
        "Hello World Lambda Function ARN",
        "arn:aws:lambda:eu-west-1:123456789012:function:lambda-debug-HelloWorldFunction-OI7DQCVooNx7"
    ]
]

If you now call plain function URL https://LAMBDA-FUNCTION-URL.lambda-url.eu-west-1.on.aws/ or with URL parameter user_id=foo you can see how the value from from process_user_data() keep growing after each call.

Lambda Function Debug Info
=============================

BUG DEMONSTRATION - Mutable Default Argument:
Notice how the list keeps growing between calls!
process_user_data() returns: 
[{'timestamp': '2025-07-25T18:27:55.517622',
  'user_id': 'anonymous'},
 {'timestamp': '2025-07-25T18:28:06.864674',
  'user_id': 'anonymous'},
 {'timestamp': '2025-07-25T18:28:21.197582',
  'user_id': 'foo'}]

But looking at the code, it should only return info about current call

    def process_user_data(user_id, metadata=[]):
        # BUG: Mutable default argument - list persists between calls!
        metadata.append({
            'timestamp': datetime.now().isoformat(),
            'user_id': user_id
        })
        return metadata

I will leave fixing this for reader to test debugging and your favorite LLM. Or you can check debug.md.

Things to consider

While remote debugging of Lambda functions is a great feature it doesn’t come without some limitations. 1) At the moment it works with recent Python, Node.js and Java functions. 2) Debugging attach an extra layer to function. This will add about 40MB on top of functon total size that must remain within limit of 250MB. 3) Connection from remote function to AWS Toolkit running in local IDE is established using IoT tunnel. If you have strict least privilege IAM policies, it might be necessary to add permission for iot:OpenTunnel for credentials you use in AWS Toolkit.

And finally, eventhough Lambda remote debugging is marketed as developer feature, I think it also has a lot of value for the Ops in devops as it allows ad-hoc debugging with minimal pre-requirements.

If you missed the first episode you can find it here.

To improve your mean time to innocence (MTTI)

Those of you who have been working as network admin or someone who’s work depends on network, you know when things stop working, network is always first to blame until proven innocent. Legal principle of one presumed innocence until prove guilty just doesn’t apply here. To get to correct root cause it is important to quickly rule out, or find evidence of, network problems. This is where AWS Reachability Analyzer can help network admins but also let anyone to verify basic requirements for connectivity from point A to B (and back again) are in-place.

What is AWS Reachability Analyzer?

AWS Reachability Analyzer is a network diagnostic service that helps to analyze and troubleshoot network connectivity within your VPC environment. It is designed to verify whether there is path between 2 points in network and identify potential connectivity blockers on the path. Hop-by-hop analysis and pointing out configuration issues can help also non-network specialist to fix connectivity issues. It isn’t actually a new service but has been around since Dec 2020.

Since then it has got more features such as ability to analyze path across multiple VPC connected via AWS Transit Gateway and most recently ability exclude resources that should not part of the path. The later got me interested again to take another look what it would be capable of.

Analysis is performed by looking at the configuration of AWS network resources. Reachability Analyzer doesn’t sent any packets. This makes it safe to use in any environment but can also limit it’s ability test paths traveling through some networks. See “Small print” -chapter later in this post.

Is there a path from A to B?

To demonstrate how Reachability Analyzer works I wrote a template you can use to create VPC similar to below diagram. It shows just a single availability zone but you can extend it using AvailabilityZones parameter and list extra AZs as lower-case letters separated by commas. For initial deployment you shoud leave BrokenNACL to false. We will get back to that later.

Template will not create any EC2 instances but use ENIs as source or target in network analysed paths. From analysis point of this is the same as EC2 instances and other resources are seen as one or multiple network interface in VPC. ENIs itself are free of charge but you would pay for public IPs and analysis.

Template also creates 3 paths you can test and see how the analysis works.

• Path from Public ENI to Private ENI.
• Path from Private ENI to Internet (8.8.8.8).
• Path from Public ENI to Internet (8.8.8.8).

If you deploy the template for multiple AZs then there is going to be a similar set of 3 paths created for each AZ. You will find Reachability Analyzer in AWS Network Manager in AWS console. There you can select one of the paths and click Analyze path from Actions -menu as shown below.

Let’s analyze first the path from private subnet to Internet. Here is how the results look like. Besides just telling you this path works it is worth while reading through as it also shows you step-by-step what are all things you need to have to make that happen. Or if there is a misconfiguration in any of those it would block the access. And when you expand “Outbound header” you see how the packet source address changes when it goes out from Internet Gateway, not at the NAT gateway as you might first think.

Let’s try next path from public subnet to Internet. This should be more simple as there is no NAT gateway in the path but route is directly from ENI to Internet Gateway. If you look how public subnets are configured, you can see there is MapPublicIpOnLaunch: true. This does assign a public IP for things launched to subnet and you would expect the route from instance to Internet work. Right?

Type: AWS::EC2::Subnet
    Properties:
        VpcId: !Ref VPC
        AvailabilityZone: !Sub ${AWS::Region}${X}
        MapPublicIpOnLaunch: true       

But it fails and says there is no public IP associated for egress ?!? To understand what happened here you would have to read the VPC documentation rather carefully, especially the part that says

Therefore, when you launch an instance into a subnet that has this attribute enabled, a public IP address is assigned to the primary network interface that’s created for the instance. A public IP address is mapped to the primary private IP address through network address translation (NAT).

But since we created just stand-a-lone ENIs, these are not considered as primary interfaces and thereforee don’t get a public IPs. The NAT above text refers is not your NAT Gateway but Internet Gateway that is a NAT device too. I will leave fixing this as an excersice for the reader ;-)

Now you can run analysis for the last remaining path from private ENI to public ENI to verify you VPC routing and security groups are properly configured. Once you have verified this path is working, go back to Cloudformation console and change this parameter from false to true to add misconfigured NACL to VPC.

Then re-run analysis for private to public path to see how analysis finds a common mis-configuration where NACL allows only one-way traffic, and because it is stateless, return packets are not automatically allowed as it would be for security groups.

Advanced use-cases

Since the introduction Reachability Analyzer has gained many new skills. It can analyze paths through Transit Gateway and multiple AWS accounts as shown in this demo. Another feature that caught my eye was ability to exclude components that can be in the path. Being able to say certain component can not be part of the path lets you to find paths that you didn’t want to have, but were accidentially created due to misconfiguration.

In my simple setup one such case could be testing path from private ENI to Internet with exclusion of NAT Gateway in the same AZ. If this analysis fails, you know your routing is correctly configured. If it succeeds, e.g. because you added route from private subnet to internet gateway (and attached public IP to private ENI) or it goes via NAT gateway in another AZ, you know there is something wrong. With this feature you can now not only test if you have a connectivity between A and B, but also test various compliancy scenarios where you want traffic being blocked.

So it is good to remember not all analysis that have failed mean there is something broken.

Small print

Reachability Analyzer supports most of common AWS network resources, but not all. You can find the list of supported resources and scenarios from documentation.

Service is priced per analysis run ($0.10/analysis). If this is cheap or expensive depends on your viewpoint. If you think this can save couple of hours debugging time when your production systems are down, it is very cheap. But if you run continous analysis of complex multi-vpc network it can be very expesive solution to feed data to network operations center dashboard showing everything is working.

And if you are using Amazon Q Developer to help you connectivity issues, it is actually using Reachability Analyzer behind the scenes.

Summary

AWS Reachability Analyzer does a good job in analyzing network paths. It all happens by analyzing configuration and not sending any packets. This can be a good thing, separating network from application level issues, but one should understand it doesn’t actually test if your service is functioning.

Pre-configured paths can help non-network specialists to check most common paths to either rule out or get insight what might be the problem in network very quickly, without support from network team. Another tool to keep in mind is Q Developer that can be more human friendly interface for debugging network issues, using Reachability Analyzer.

But lets face it, format is getting aged and fixing cars is a bit niche topic these days. So here is my proposal for the face-lift and synopsis for the pilot episode of …

VPC Dealers

In this episode I’ve inherited an old web service build on common VPC design for its time. Architecture is basicly ok and fit for the purpose. Public and private resources are in their separate subnets. Only the application load balancer (ALB) is exposed to internet and EC2 instances in private subnets are protected from direct internet access. But today we could do better and with little effort, make it run better, cheaper and be more secure. Pick any 3 ;-)

More Secure

Security is job zero, so lets start from there. If you could remove public subnets and not to expose any part of your VPC to internet, it would mitigate the possibility of misconfiguration exposing server or database to internet. As current configuration is using NAT gateways in public subnets as a route to internet you would have to have an alternative route before decommisioning NAT gateways and being able to remove public subnets. Luckily the centralised egress is easy to setup with Transit Gateway. If you don’t already have that in your organisation here is how you can configure a shared route to internet for all your VPCs. And it comes with CDK build too.

Great thing about central egress VPC is it doesn’t affect your developer experience and it gives you a perfect locatation to add egress filtering later as additional layer of security. See Centralized egress in AWS Prescriptive Guidance for best practices of securing egress traffic.

Cheaper

As we managed to replace multiple NAT gateways with TGW attachment it will be cheaper to run. This is especially important when you have 10’s or 100’s VPCs but every VPC counts. At the same time we also removed 2 public elastic IPs (for NAT gateways) and 2 or more public IPs for ALB. As you might remember all public IPv4 addresses cost you. Removing those will keep your FinOps -team happy.

Better

But what happened to public ALB and why I didn’t delete the internet gateway with public subnets?

In November 2024 a new feature Cloudfront VPC Origins was announced. This is was such a big thing there was not just one but also a second announcement of the same feature on that very same day.

You should read the details how to deploy VPC Origin from above blog posts but in short this allows us to serve content via CloudFront from EC2, ALB or NLB in private subnets. An Origin will be deployed as AWS managed ENIs in your subnets.

For this to work you must have an internet gateway attached to VPC but you don’t have to have a route pointing to it. This is just a safety measure preventing building an unallowed access from internet, similar to Global Accellerator -case in the past. There is also the usual small print you should read before deploying.

Obviously adding CloudFront will enchance user experience by cutting down the latency between browser and service. But I don’t think this is the most important aspect of “better” in this case. This setup solves the problem many development teams have with centralized ingress where front-end load balancer is placed into central VPC out of their reach.

I believe this way it is possible to both enable developers to deploy their service independently of network team using any tools and workflows they are familiar with, and at the same time not have route (as routable public IPs) from the internet to every VPC. Keeping both devs and net/sec happy is no small feat!

With Little Effort

All of above would be nice but not very impressive if that would only apply to new VPCs build from ground-up, or require re-deploying all existing resources. Luckily you can apply these changes to existing VPCs with running services with only a short glitch.

• First deploy internal ALB and register the same targets as with original internet facing ALB.
• Configure CloudFront VPC Origin pointing to internal ALB.
• Attach TGW for cental egress into private subnets.
• Change default route of private subnets from NAT GWs to TGW attachment.
• Switch DNS from ALB to CloudFront and wait until there is no more traffic to ALB.
• Delete NAT GWs and attached EIPs. If you suspect you might have out-bound integrations that depend on your IP address, you can keep EIPs for a moment. Just don’t forget to delete them once you are sure you don’t need them.
• Delete ALB, public subnets and public route table(s).
• Done.

Summary

In this episode of VPC Dealers we modernised a classic VPC architecture to run better, cheaper and be more resilient to operator errors. All this without sacrificing developer team independence or having a service break. I’m sure there will be many happy deployments ahead with new improved VPC architecture.

If you liked this, please stay tuned for next episode of VPC Dealers by subscribing RSS-feed.

Until then. Ta-Da!

The Red Hat Knowledge Base is an extensive repository of resources related to Red Hat products and services. It contains technical documentation, how-to articles, FAQs, and community contributions designed to help users effectively utilize Red Hat technologies, such as Red Hat Enterprise Linux, OpenShift, Ansible, and more. Users can search the knowledge base to find solutions to issues or to learn more about specific features and functionality of Red Hat products.

Unfortunately good things are seldom free. To access Red Hat Knowledge Base content you would typically need:

• Red Hat Account: You should have an active Red Hat account. This is required to log in and access specific articles or support resources.

• Subscription: Some content in the knowledge base may require a valid subscription to Red Hat products. This grants access to more in-depth resources and support.

Problem can be not just the price of subscription but the purchase process of your company, especially if you don’t use RHEL but one of it’s free derivatives. Luckily AWS console has a shortcut for you …

Access Red Hat Knowledge Base from Fleet Manager

This is actually an old news but you might have missed it if RHEL administration on EC2 wasn’t your main focus back in 2021 when the feature was announced. All you need is an EC2 instance running RHEL AMI registered to AWS Systems Manager (SSM). Unfortunately RHEL AMIs don’t come with SSM agent pre-installed but you have to take care of installation yourself or bake your own AMI from one provided by Red Hat.

SSM agent can be installed to RHEL 8 & 9 with the following user-data script. This will take care of adding Python 3 and downloading SSM agent from S3 bucket in the same region as your instance is. Note that if you are deploying this from Cloudformation template you don’t have to resolve REGION in the script but REGION can be supplied during stack creation.

#!/bin/bash
ARCH="linux_amd64"
REGION=`curl -s http://169.254.169.254/latest/meta-data/placement/region`
[ `uname -m` == "aarch64" ] && ARCH="linux_arm64"
sudo dnf install python3
sudo dnf install -y https://s3.${REGION}.amazonaws.com/amazon-ssm-${REGION}/latest/${ARCH}/amazon-ssm-agent.rpm
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent

Now Go Build

To make testing easy, I wrote you a template for VPC with single t2.small instance with supporting resources as shown below. Note that default AMI ID is for RHEL9 at eu-west-1. If you deploy to any other region you must find the RHEL9 AMI ID yourself *

* System Manager Paremeter Store has redhat -namespace where you could find aliases for RHEL AMI IDs. Unfortunately this feature is mostly not maintained, except for AWS provided AMIs.

After deploying the stack from the template you can navigate to Systems Manager in AWS console, and open Fleet Manager. From Account management -pulldown you will find “Access to Red Hat Knowledgebase” -link.

Clicking the link used to open a new browser window to Red Hat support portal, but now there is a delay build into the process …

After a day or two, you are forwarded to support portal with access “subscription required” content.

You can now access the subscriber content, but to view or write comments you’d have create account and login.

Authorization

Access to support portal is granted by readOnlyAuth cookie from access.redhat.com. The Cookie is valid for 24h from authorization at AWS console, so you can continue your session for a moment after your RHEL instance has been stopped or terminated.

And while it took 2 days from provisioning RHEL EC2 instance for authorization start working, it seems to work other way around too. I was able to refresh my session from AWS console on a day after I shutdown my RHEL instance. But this was just a single test case. You mileage may vary.

Summary

Fleet Manager link to Red Hat support portal can be useful especially for random needs to find solutions for common RHEL problems, especially if you prefer expert advices over genAI content. But if this your full-time job, I’d recommend creating an account to Red Hat portal and maybe keep AWS console as your backup solution.

AWS Services

In the past there was no that many complete services being depricated but this year (2024) AWS has been busy with EOLs. Some of these services weren’t really main stream so if you haven’t heard of Amazon Lookout for Vision|Metrics|Equipment or Monitron, you’re not alone. But that was a whole set of industrial AI/ML services that were put into maintenance mode, with many others, in October 2024.

Similar, but more main stream, clean up did happen in development tools where CodeCommit, CodeStar and Cloud9 are no longer available for new customers.

Common way for AWS to depricate a service or feature, is block new customers from start using it. It can be that your existing deployments will continue work, and you might be able to make changes too (but not always) or you could even be able to deploy new instances of service on accounts where you had given service in use.

Any deprication can have a major effect for a customer if critical services are build to depend on it, but in global scale there is one breaking change that will dwarf all others …

EC2 Autoscaling Launch Configurations

Through a series of deprecations spanning multiple years, AWS is slowly killing off EC2 Auto Scaling Launch Configurations in favor of EC2 Launch Templates. On October 1, 2024, only those accounts that have previously used Launch Configurations can continue to do so.

While this has been public information since October 2021, I think there are still a lot of ‘legacy’ Cloudformation and Terraform code depending on launch configuration being available. Just a simple search from aws-samples github repositories found +180 references to Cloudformation resource type AWS::AutoScaling::LaunchConfiguration

Now after October 1st, 2024, those templates will continue work if you have previously used launch configurations on the account, but on any account that was created after October 1st, or didn’t previously have launch configurations, the same template will fail.

Conversion process from launch configuration to launch template isn’t very complex in most of cases, but tricky part can be finding the all the code and CI/CD processes. For services that are not in active development mode this can be a challenging reverse engineer task.

3rd Party Software

When the service is all about hosting 3rd party software, there is obvious dependency to availability and support for given software version. Major services in this category are

• EKS
• RDS for PostgreSQL / SQL Server / Oracle / MySQL / MariaDB / Db2
• Lambda runtimes
• … plus others like Elasticache, OpenSearch and even Amazon Linux.

For databases minor and even major version upgrades are often backwards compatible, so your application will work with next version without changes. Upgrade effort is then reduced to testing (which shouldn’t be underestimated) and scheduling the upgrade. However it can be your application is certified only with certain versions and while it could work after upgrade, loosing your application vendor support might be too risky for business critical service.

Kubernetes control plane (EKS) upgrades takes a bit more planning because there is no warranty new version is compatible with previous one. It is your responsibility to judge if breaking changes between versions are something you should be worried or not. To resolve upgrade issues you might need to make changes to your applications or find a new combination of versions for middleware services you run in the cluster.

If you feel dead-lines for RDS and EKS updates are coming too fast for you, there is an option to buy extra time but it doesn’t come for free.

For Lambda the serverless promise was you don’t need to manage servers (OS) or runtimes. While that is true, no-one ever said you can just deploy and be done with it. When Lambda runtime is deprecated functions you have deployed will continue working but you can not create any new or update existing functions, ie. deploying bug fixes or new features to your code isn’t possible. Python 2.7 EOL announcement is old but good example how Lambda runtime EOL process works.

Keep Yourself Updated

I wish there would be a single timeline of all EOL plans for AWS services. Unfortunately there isn’t, or if you find one, please let me know too. The Best thing I’ve found is aws_breaking_changes -git repo mainteined by Scott Piper. README has the list recent and upcoming deprecations, or breaking changes. Past changes are archieved in archive.md.

EDIT: Also aws-news.com dedicated feed for service deprecations is worth subscribing to.

Other good sources are AWS service documentation (e.g. above links for RDS, EKS & Lambda) and subscribing AWS blogs from categories that are most relevant for you.

I am a complete newby to AWS and HTML and want to create a pdf file sharing solution for my social group. I have created a s3 bucket with my pdf’s in a folder. when I specify a single file the web page works ok ie

<iframe src="Songs/my file name.pdf""> </iframe>

I cannot find a way to specify all the files using a wild card and I cannot specify each file individually so how can it be done.

When you ignore some of the context, what is left is the question how to list objects from S3 bucket on a webpage or an application. This may first sound trivial but keep reading …

Server-side solutions

Obvious solution would be rendering the page on server-side. You could write a piece of code to list the bucket content and transform that into something that can be presented in browser or application. As everyone should know by now, inserting AWS API key (or any secrets) into your code base is serious anti-pattern. Luckily this is very easy to avoid using instance profiles or IAM roles that take transparently care of assigning temporary API keys for your code to use.

And even if you would be running server outside of AWS, there is similar functionality (with little bit more setup effort) called IAM Roles Everywhere you could use instead of static API keys.

All above makes server-side implementation too trivial to be interesting. But what if you don’t want to have any of application code running on your servers, including serverless solutions ;-)

Problem with client-side secrets

Challenge with client-side solutions isn’t in writing the code that can run in a browser or creating a client application. Just pick the SDK for your favourite language and go build! Problem is how to authenticate for AWS API without exposing your API key & secret to the client. By default you should assume anything distributed with client will be exposed to the public. You could craft the IAM policy very carefully so it would allow access to things that are already public but even then there is a risk that you made a small mistake in IAM policy (we all know how easy that is) or someone accidentially alters the policy later.

Amazon Cognito helps in building end-user identity and access management. It can also provide temporary access to AWS APIs similar to instance profiles and IAM roles for server-side solutions. Cognito also allows you to define a separate IAM role with limited permissions for guest users who are not authenticated. This removes unnecessary login step, but still enables you to use temporary, limited privilege credentials to access AWS resources.

While Cognito helps when you want to provide access to AWS resources for authenticated users, identity pool and permission setup is bit complex if you only want to provide anonymous access to S3 (or similar).

Anonymous access

For S3 there is also truly anonymous access, meaning you don’t need any API key to access public buckets and objects. Easiest way to test this is with AWS S3 CLI. When you add -no-sign-request option, no credentials are used but I can still get the list of objects.

% aws s3 ls s3://s3.carriagereturn.nl/ --region eu-north-1 --no-sign-request
2024-07-31 18:42:27     119409 hello.jpg
2024-07-31 18:41:17         15 hello.json
2024-07-31 18:41:17         12 hello.txt

And to verify I didn’t have any credentialsi, the same command fails when I try to sign the request.

% aws s3 ls s3://s3.carriagereturn.nl/ --region eu-north-1                  
An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.

For above to work, bucket policy must allow s3:ListBucket and s3:GetObject.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicRead",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::s3.carriagereturn.nl",
                "arn:aws:s3:::s3.carriagereturn.nl/*"
            ]
        }
    ]
}

As the challenge was to create a webpage with dynamic listing of bucket content I created a small demo project using AWS Javascript SDK V3 and anonymous access to S3 API.

You can get the code from GitHub repository and build it yourself. If things worked as planned you should see page similar to this screenshot.

In earlier versions of Javascript SDK there was s3.makeUnauthenticatedRequest method but in current versions that doesn’t exists and documentation isn’t very helpful either :-(

Work-a-round is to define signer as shown below.

    // Initialize an S3 service with anonymous credentials.
    const s3 = new S3({
        region: region,
        signer: {
            sign: async (request) => request
        }
        ...

Summary

If you think you must include static credentials with any client, or server, side code, think again. As shown above, there are multiple ways to avoid using long term AWS API secrets. In some cases you can also make the code work without any API authentication. Yes, it is only limited S3 use-cases but when that is all you need, anonymous access can simplify your application architecture and avoid running application code on server-side, making the application truly serverless ;-)

PS. If you are looking for a bit more ambitious implementation of S3 access from a browser, using Cognito authentication, take a look at public-file-browser-for-amazon-s3.