AWS DOP-C02: Secrets Manager versions and rotation staging labels, the four rotation steps and rollback, walked through by hand

Readers follow AWSCURRENT; rotation writes AWSPENDING first, moves AWSCURRENT last, and leaves AWSPREVIOUS for the way back.

The problem

A deployment pipeline and the application it ships both read a database password from AWS Secrets Manager, a managed service that stores and rotates credentials. The password must change on a schedule with no outage. A rotation that fails halfway must never leave the application holding a password the database rejects. When a new password turns out to be bad, someone must be able to return to the old one quickly.

Exam questions here are short incident stories (a half-finished rotation, a wrong value, a deleted secret) that ask which label, API (application programming interface) call or parameter fixes it. The AWS Certified DevOps Engineer - Professional (DOP-C02) exam guide covers this in two places. Content Domain 1, "SDLC Automation" (software development life cycle, 22% of scored content), has Task Statement 1.1 "Implement CI/CD pipelines." (continuous integration and continuous deployment), with the skill "Managing build and deployment secrets (for example, AWS Secrets Manager, AWS Systems Manager Parameter Store)". Content Domain 6, "Security and Compliance" (17%), has Task Statement 6.1 "Implement techniques for identity and access management at scale.", with the skill "Automating credential rotation for machine identities (for example, AWS Secrets Manager)". Parameter Store is the other service the guide names; only Secrets Manager is covered here.

The lab walked one secret through every label move by hand. It used AWS command-line interface (CLI) commands against a local AWS emulator (moto) on 127.0.0.1, with throwaway credentials and fixed version IDs. The emulator diverged from the documented service in three places, and each one is flagged where it happens.

The idea, in plain words

A secret holds many versions, and a staging label is a movable name attached to one version at a time. Readers ask for AWSCURRENT, so the rotation recipe is to write the new value under AWSPENDING, change the real credential in the database, test the new value, and only then move AWSCURRENT onto it. Secrets Manager moves AWSPREVIOUS onto the old version by itself when AWSCURRENT moves. That automatic step is what makes rollback a single call, because the last known good value is always labeled and waiting. Each step touches a different label, so a failure at any point leaves readers on a value that still works.

Versions and staging labels

A secret is a named item in Secrets Manager. A version is one stored value of that secret, and a version ID is the identifier of that version. A staging label is a name attached to a version, and it can be moved from one version to another. Secrets Manager defines three: AWSCURRENT, "the current version"; AWSPREVIOUS, "the previous version"; and AWSPENDING, "the pending version (during rotation)". A secret always has a version labeled AWSCURRENT, and that is the version returned by default when you retrieve the value.

Labels exist because version IDs change on every rotation, while consumers need a stable name for "the value to use now". A secret can carry up to 20 labels across its versions, and versions can have several labels at once. Two versions can't share a staging label, so attaching one to a new version takes it from the old one.

The lab created the secret and then stored a second value without naming any label. In the commands below, $S holds the secret name app/orders/db and $V1 and $V2 hold the fixed version IDs.

aws secretsmanager create-secret --name $S --client-request-token $V1 --secret-string '{"username":"orders","password":"first-pass"}' --query VersionId --output text
aws secretsmanager put-secret-value --secret-id $S --client-request-token $V2 --secret-string '{"username":"orders","password":"second-pass"}' --query '[VersionId,VersionStages]' --output json

After each command the lab printed the VersionIdsToStages field of describe-secret. After step 1 the first version alone carried AWSCURRENT, because a new secret must have a current version. After step 2:

[
    "22222222-2222-4222-8222-222222222222",
    [
        "AWSCURRENT"
    ]
]
{
    "11111111-1111-4111-8111-111111111111": [
        "AWSPREVIOUS"
    ],
    "22222222-2222-4222-8222-222222222222": [
        "AWSCURRENT"
    ]
}

The second put-secret-value omitted --version-stages, and the documentation says that in this case Secrets Manager automatically moves AWSCURRENT to the new version. AWSCURRENT is on the second version, and the first version picked up AWSPREVIOUS without being asked.

Step 3 stored a third value with --version-stages AWSPENDING, the way the first rotation step does. The lab then read the secret twice.

-- default read:
22222222-2222-4222-8222-222222222222	{"username":"orders","password":"second-pass"}
-- AWSPENDING read:
33333333-3333-4333-8333-333333333333	{"username":"orders","password":"third-pass"}

The third version exists and holds third-pass, but AWSCURRENT stayed on the second version. A read with no version specified returns the AWSCURRENT version, so the default read returned second-pass. Only a read that names the AWSPENDING stage reaches the new value.

On the exam: expect a question where a value was stored successfully but consumers do not see it. The answer is almost always about which label the new version carries.

Symptom: right after storing a new password, the application still gets the old one.

Cause: the new version was stored with --version-stages AWSPENDING (or another label), so AWSCURRENT never moved and default reads return the old version.

Fix: move AWSCURRENT with update-secret-version-stage once the new value is tested, or store the value without --version-stages when no rotation steps are needed.

The four rotation steps and idempotent retries

Rotation is the scheduled replacement of a credential with a new one. In the Lambda-based form, a rotation function does the work. AWS Lambda is a service that runs a small piece of code on demand, and the rotation function is that code. It runs four steps, and each one reads or moves a specific label.

The documentation explains why createSecret stores the value under AWSPENDING: "Storing the new secret value in AWSPENDING helps ensure idempotency. If rotation fails for any reason, you can refer to that secret value in subsequent calls." An operation is idempotent when repeating it has the same effect as running it once. The rotation function needs this property because a crash after the database password changed, but before the secret was finished, must not produce a second, different password on retry.

The tool for that is the client request token, a caller-supplied identifier that becomes the version ID of the new version. The rule for PutSecretValue reads: "If you call this operation with a ClientRequestToken that matches an existing version's VersionId, and you specify the same secret data, the operation succeeds but does nothing. However, if the secret data is different, then the operation fails because you can't modify an existing version; you can only create new ones."

Step 4 of the lab retried the third version's write twice. First it used the same token and the same value, then the same token with other-pass.

33333333-3333-4333-8333-333333333333
exit 0
33333333-3333-4333-8333-333333333333
exit 0

The first retry matches the documented behavior, since it succeeded and changed nothing. The second also exited 0, and the documentation says the real service fails here. The damage shows up in step 5, where a read by the AWSCURRENT label returned {"username":"orders","password":"other-pass"} for the third version. The emulator silently rewrote the stored value of an existing version. That is emulator behavior. The documented service refuses it, because "you can't modify an existing version; you can only create new ones."

Secrets Manager describes two designs for the rotation function. A single-user strategy "updates credentials for one user in one secret", with a short window in which the database may deny the rotated credentials, which a retry strategy mitigates. An alternating-users strategy "updates credentials for two users in one secret" and alternates which user's password each rotation updates. The lab tested neither.

On the exam: idempotency questions hinge on the token and on the AWSPENDING version being the saved copy of the new value.

Symptom: a rotation retry fails at createSecret with an error about the version, after a first attempt crashed halfway.

Cause: the retry reused the same client request token with a newly generated value, and the data of an existing version cannot change.

Fix: on retry, read the existing AWSPENDING value for that token and reuse it, which is why createSecret stores it first. This follows from the documentation; the emulator did not reproduce the failure.

Moving AWSCURRENT, and rolling back

The last rotation step moves AWSCURRENT, and the API for it is UpdateSecretVersionStage. Its MoveToVersionId parameter names the version that receives the label. Its RemoveFromVersionId parameter names the version that loses it. The documentation states the rule this way: "If the staging label you are trying to attach to one version is already attached to a different version, then you must include this parameter and specify the version that the label is to be removed from." Without it, a move could silently strip a label readers depend on.

Step 5 tried the move without it, then repeated it correctly.

aws: [ERROR]: An error occurred (InvalidParameterException) when calling the UpdateSecretVersionStage operation: The parameter RemoveFromVersionId can't be empty. Staging label AWSCURRENT is currently attached to version 22222222-2222-4222-8222-222222222222, so you must explicitly reference that version in RemoveFromVersionId.
exit 254
app/orders/db
exit 0
{
    "11111111-1111-4111-8111-111111111111": [],
    "22222222-2222-4222-8222-222222222222": [
        "AWSPREVIOUS"
    ],
    "33333333-3333-4333-8333-333333333333": [
        "AWSPENDING",
        "AWSCURRENT"
    ]
}

The error names the version holding AWSCURRENT, so it doubles as a lookup. With --remove-from-version-id $V2 added, the move succeeded: AWSCURRENT on the third version, AWSPREVIOUS on the second. The first version holds no labels at all, because the move pushed AWSPREVIOUS off it. The third version still carries AWSPENDING because the lab never removed it; the leftovers section covers that.

Here the emulator diverged. After the move, the default read still returned the second version (second-pass), while the read by label AWSCURRENT returned the third. The documentation says the default read returns AWSCURRENT, so on the real service both reads should return the third version.

Step 6 rolled back with one call that moved AWSCURRENT to the second version and removed it from the third. The documentation says: "To revert to the previous version of a secret, call UpdateSecretVersionStage."

{
    "11111111-1111-4111-8111-111111111111": [],
    "22222222-2222-4222-8222-222222222222": [
        "AWSCURRENT"
    ],
    "33333333-3333-4333-8333-333333333333": [
        "AWSPENDING",
        "AWSPREVIOUS"
    ]
}

The second version is current again, and AWSPREVIOUS landed on the third version without being named in the command. That matches the documented rule: "Whenever you move AWSCURRENT, Secrets Manager automatically moves the label AWSPREVIOUS to the version that AWSCURRENT was removed from."

On the exam: "revert to the previous value" points to UpdateSecretVersionStage, and a failure naming RemoveFromVersionId means the call omitted the version currently holding the label.

Symptom: an automation that moves AWSCURRENT fails with InvalidParameterException about RemoveFromVersionId.

Cause: the label is attached to another version and the call did not name it.

Fix: read the version that holds AWSCURRENT (describe-secret, field VersionIdsToStages) and pass it as --remove-from-version-id.

Leftovers: stale AWSPENDING, deprecated versions and deletion

Rotation leaves three kinds of trace. First, a version that loses its last label is deprecated. Secrets Manager "never removes labeled versions", but it removes deprecated versions when there are more than 100, and it never removes versions created less than 24 hours ago. In step 7, the first version had no labels. The documentation says ListSecretVersionIds leaves such versions out unless IncludeDeprecated is set. The emulator listed the first version both ways, the third divergence.

-- list-secret-version-ids:
11111111-1111-4111-8111-111111111111	
22222222-2222-4222-8222-222222222222	AWSCURRENT
33333333-3333-4333-8333-333333333333	AWSPENDING,AWSPREVIOUS
-- with --include-deprecated:
11111111-1111-4111-8111-111111111111	

The second listing, trimmed here, was identical to the first. The unlabeled version still read by ID with exit 0: deprecated means eligible for removal, not gone.

Second, a failed rotation can leave AWSPENDING behind. Step 8 simulated a rotation that stopped right after createSecret by storing a fourth value as AWSPENDING. Because a label can sit on only one version, it left the third version, which kept only AWSPREVIOUS. Then update-secret-version-stage --version-stage AWSPENDING --remove-from-version-id $V4, with no --move-to-version-id, cleared it. That works because, per the documentation, "To remove a label from a version, then do not specify" MoveToVersionId. The final stages were:

{
    "11111111-1111-4111-8111-111111111111": [],
    "22222222-2222-4222-8222-222222222222": [
        "AWSCURRENT"
    ],
    "33333333-3333-4333-8333-333333333333": [
        "AWSPREVIOUS"
    ],
    "44444444-4444-4444-8444-444444444444": []
}

The fourth version ends with [], so the pending marker is gone and the next rotation starts clean. The troubleshooting guide gives the same advice: "If rotation consistently fails, clear the AWSPENDING staging label and retry rotation".

Third, deletion. Step 9 ran delete-secret --recovery-window-in-days 7. The documentation says the minimum recovery window is 7 days and the default is 30, and that Secrets Manager attaches a DeletionDate stamp marking the end of the window. The lab replaced the real date in its output with <DeletionDate>. Reads then failed, and restore-secret brought the secret back.

aws: [ERROR]: An error occurred (InvalidRequestException) when calling the GetSecretValue operation: An error occurred (InvalidRequestException) when calling the GetSecretValue operation: You tried to                 perform the operation on a secret that's currently marked deleted.
exit 254
app/orders/db
22222222-2222-4222-8222-222222222222	{"username":"orders","password":"second-pass"}
exit 0

After the restore, the read returned the second version again, so the labels survived the round trip. The documentation: "When a secret is scheduled for deletion, you cannot retrieve the secret value. You must first cancel the deletion with RestoreSecret and then you can retrieve the secret."

On the exam: a deleted secret inside its recovery window is restored, not recreated, and stuck rotations call for clearing AWSPENDING.

Symptom: every rotation attempt for a secret fails, though the last successful one was days ago.

Cause: an earlier rotation stopped after createSecret and left AWSPENDING on a version that was never finished.

Fix: check which version holds AWSPENDING, clear the label with update-secret-version-stage using --remove-from-version-id only, then retry rotation.

Putting it together

1. createSecret writes AWSPENDING under a client request token, so a retry finds the same version.
2. setSecret and testSecret change and try the credential using AWSPENDING.
3. finishSecret moves AWSCURRENT with RemoveFromVersionId, and AWSPREVIOUS becomes the rollback target.
4. A stale AWSPENDING is cleared before retrying; a deleted secret is restorable for 7 to 30 days.

The lab used aws-cli 2.35.11 against moto 5.2.4.dev0. Its three emulator divergences, not AWS behavior:

The lab did not test a Lambda rotation function, managed rotation (which the documentation says does not use a Lambda function), a real database, rotate-secret itself, KMS (Key Management Service) keys and resource policies, or replication.

Check yourself

Sources

• AWS Certified DevOps Engineer - Professional (DOP-C02) Exam Guide
• Secrets Manager concepts: secret versions
• PutSecretValue, AWS Secrets Manager API Reference
• UpdateSecretVersionStage, AWS Secrets Manager API Reference
• Lambda rotation functions, AWS Secrets Manager User Guide
• Rotation strategies, AWS Secrets Manager User Guide
• Troubleshoot rotation, AWS Secrets Manager User Guide
• GetSecretValue, AWS Secrets Manager API Reference
• ListSecretVersionIds, AWS Secrets Manager API Reference
• DeleteSecret, AWS Secrets Manager API Reference
• Rotate AWS Secrets Manager secrets, AWS Secrets Manager User Guide