Stripe has two modes, Live Mode and Test Mode. Both of these represent isolated environments, and Test Mode will provide simulated responses for some requests. It's a really useful sandbox to build against as you implement your Stripe integrations. It also comes with a few gotchas.
As we encountered these issues while building Tier, we decided to create tier switch to make it easy to spin up test and preview environments.
To learn more about the switch command and how to use it to avoid these issues, please read this post and give it a try today.
Inconsistent UI
One quirk of test mode is how it shows up in the stripe Dashboard. It’s easy to miss as you move between modes.
Test Clocks will usually isolate some objects from each other in the API, however, they do not show up that way when using the UI in Test Mode.
For example, if you list all customers in the API without specifying a test clock, you will only see customers without test clocks, but in the UI, you will see all customers with and without test clocks.
To distinguish between objects with test clocks, and those without, objects associated with a test clock get orange banners and/or an icon indicating they are part of a test clock. This is helpful, but it does not show any indication of which objects are grouped together by a test clock. To see this, you’ll need a new skill to operate and see in a new dimension, as we’ll quickly see:
In the above screenshot, there are two test clocks. Which customers are on the same clock? You can probably deduce which are associated by looking at the CREATED field, but what if I advance both clocks to the same time, or what if I ran two simulations each with nearly identical timelines? It all becomes blurry in the “normal” UI mode.
The only way to find out who is associated with what clock is to drill down to the test clock’s screen.
Having to keep this in mind as you navigate around the UI requires a new mental shift. One you must enter into and switch out of rapidly to gain a sense of what you're looking at depending on the screen, and how the things you're seeing all relate to each other in this new, additional dimension.
Product and Price Debris
Products and Prices remain unisolated from subscriptions associated with a test clock, as well as with subscriptions not associated with a test clock. This means products and prices will not be cleaned up when a test clock cleans up (if you ever clean up test clocks right away or just wait for them to GC on their own).
This may not seem like a big deal, but when you’re in a trial-and-error mode, working out the kinks of your pricing model, or running an integration test, this quickly becomes a major source of friction and frustration. That is because you cannot start a new simulation in a totally clean room, so you must manually delete each product and price you modified, otherwise it may interfere with the next simulation.
Key and ID Collisions
It's important to note that deleting products and prices can not happen if they have been burned into the audit log. In this situation, you can only archive them, but their IDs and lookup keys remain. This will often cause collisions across simulations and tests.
One of my favorite things about the product and price APIs in Stripe is that you can bring your own IDs and lookup keys. This allows you to assign a unique ID or lookup key that may only be used by a single product or price, respectively, at a time.
But if you can’t delete the price or product because a subscription has burned the price into the audit log, then the next test run might get an error if attempting to create a new price with the same lookup key or product with the same ID.
We can side-step this by namespacing our keys across tests, but this means our code must be very smart and careful to not let this behavior leak into production, and know how to plumb the namespace from the request received by the user all the way to stripe, or risk clobbering or duplicating objects on write and potentially reading back the wrong one. This isn’t ideal either.
The Manual process of Deleting Test Data
With the above gotchas in mind, we probably want to think about wiping the slate clean.
Unfortunately, there is no Stripe API to start with a fresh Test Mode. To do so programmatically requires cascading delete calls to the API per object, which is a slow and error-prone process.
Even if “done right”, you’re likely to run into the problem where you can’t delete prices already burned into the audit trail (which in production is a good thing, but the reality is different for your test environments as they become impossible to clean completely).
Keep the babies. Ignore the bath water.
When you hit that “Delete Test Data” button, you’re potentially erasing the artifacts of valuable lessons learned. Lessons that could be used to refer back to as you experiment with new approaches to your pricing and billing.
To keep these for reference, a common approach is to simply sign up for a new Stripe account. Maybe we call it “test2”, but this comes with new problems:
- Extra API keys to manage
- Extra login credentials to manage
This also does not fix the problem of throwing out our history, because this new account could end up with data as valuable as the previous, so away we go again to sign up for a new Stripe account, and then again, and so on.
We can delay decluttering our account drop-down by triaging all of those accounts until another day.
Continuous Integration
It isn’t hard to imagine getting into a situation where changing a price or a product in the “Holy” CI Stripe account wreaks havoc on CI. This usually happens because the once “pristine” account is now out of whack according to the code expecting Stripe to look another way.
If engineering was already sheepish about making changes to the pricing model, this just further adds ammo to their argument against it, because no one wants to breathe on that code, for fear of breaking it.
Make the switch
Tier is your new set of power tools for Stripe. By using the switch command, you can avoid the 5 gotchas of test mode and gain the power of a PriceOps-compatible approach to monetization while building powerful new workflows on top of Stripe. This gives you the best of both worlds: The power of Tier and the reliability and reach of Stripe.
With Tier, you get powerful features such as switch along with our SDKs, metering, and feature flagging style entitlement checking.
If you aren't already a Tier user, you can get started quickly and easily.
