Pulumi state on an Azure backend

The Pulumi Service backend is seamless to get going and has been brilliant to work with so far. We are handling multiple simultaneous projects at any one time, with any number of new ones on a weekly basis generated by our SaaS application. Each project has a number of environments, and hence a stack per environment. With about 150–200 resources per environment, this can quickly build up.

For a number of reasons, including cost as we scale up and keeping specific data within our own controlled environment, we looked at moving certain projects to our Azure hosted backend and we’ll share our experiences with that move.

Architecture

We have a core Pulumi project which deploys a number of our central management infrastructure, such as DNS. We’ve kept the Azure state backend also deployed through this core project and we’ll keep this managed by the native Pulumi Service backend. Avoids the chicken-egg issue of how to manage our Azure backend deployment through Pulumi if the state needs to be on that same Azure.

State

We will store our state on an Azure storage account, set up as with geo-redundancy to ensure any outage recovery options. And we create a number of containers in this storage account which will segregate our states.

AZURE_STORAGE_ACCOUNT=
AZURE_STORAGE_KEY=

Encryption

We use Azure Key Vault for our encryption handling and created a vault and a key for Pulumi to use. When setting this up we had to add our Pulumi service principal to the Key Vault Crypto Officer role assignment for the key vault, otherwise it can not create the needed keys.

In order for Pulumi to connect to the Keyvault the following variables need to be set:

AZURE_CLIENT_ID=
AZURE_CLIENT_SECRET=
AZURE_SUBSCRIPTION_ID=
AZURE_TENANT_ID=

Then to set up a new project with set up a

$ pulumi login --cloud-url azblob://internal
pulumi new azure-python -n <project name> -s <stack name> -y --secrets-provider="azurekeyvault://<azure kv url>"

Or similarly, a new stack using the following:

$ pulumi stack init <stack name> --secrets-provider=”azurekeyvault://<azure kv url>"

Lessons learnt

1. Project/stack naming is different. There is no concept of <org>/<project>/<stack> hierarchy like in Pulumi Service backend This is actively being worked on to resolve these (and other) differences, hopefully an update will be coming soon. As a workaround we are using the convention of calling stacks with this format: “someproject.stack”.
2. However, this also means that when doing a stack list, there is a very long list of stacks, because obviously they are all in the same backend container. Not ideal, but avoids the complexity of multiple workflows for each SaaS deployment we do
3. Cross-backend stack calling is not doable. An Azure backend stack calling outputs from a Pulumi Service backend stack will not work. It’s not even possible for one Azure backend to call outputs from another Azure backend. Basically what the credentials/settings all Pulumi to seen on the current backend is what’s available.

End result is all our projects and stacks are in an Azure storage account. There are actually a few storage accounts/containers, which we segregate based on type of usage (client, internal, development testing, etc.). Apart from the few gotchas as shown above, it’s been fairly seamless to both set up and migrate to.