How we're building end-to-end tests that keep up with coding agents
Most of the code we merge is written with coding agents. Find out how we built it, and how we use a sandbox to check fake providers against real ones.
Doruk Güneş
Founding Engineer

At Seapoint, most of the code we merge is now written with coding agents. In October we merged over 1,000 pull requests, more than twice as many as a year before.
As writing code has gotten a lot faster (and cheaper), checking it has become all the more important. Reviewers can’t run every change by hand, so they lean on the agent’s own checks, and those don’t catch every regression.
Since there's no perfect check we aim to improve the ones we already have, with one important example being the end-to-end suite. This part of our stack, using Playwright under the hood, signs into Seapoint, approves a payment with a second factor and checks the money has moved, as a customer would.
In this post I'll share how we built it, and how we use a sandbox to check fake providers against real ones.
Step one: test on a shared login
Our first browser tests all ran as one user on our test environment with data we'd set up by hand, like transactions and beneficiaries.
It was a decent place to start. Every run went through the login, and we were able to keep costs down by signing in once and reusing the session in every test. It ran once a day, and all it told us was that staging was partly healthy.
But as more people and agents wrote tests, nearly every problem we hit came back to only having one shared user. A crashed run left data behind, and the next run failed because of it. Tests couldn't run in parallel so the suite ran one at a time, and it only worked on staging since nowhere else had that exact data.
Step two: a company for every test
To stop sharing data between tests where it could cause problems, we created a separate login for each worker whenever a test modified server-side data, as recommended by Playwright’s own docs.
In our case, most of that data belongs to a company, so we decided to give each test file its own company, with only the data it needs. We also moved our signup logic into a separate service so test companies go through the same creation process as real customers, and from there each test can request whatever else it needs, like accounts or beneficiaries.
With this approach, an agent could write and run a test on its own, so we built a write-e2e-test skill that writes a test using a one-line request. After the test has been reviewed and approved by an engineer, the test runs and sends back a report with a video, ready for engineer analysis.
While creating a company for every test worked, it wasn’t free. Everything we provision is another round trip to our server, our database and often a third-party provider, making this approach not only expensive, but also very slow.
Step three: faking the providers
As well as being pricey and slow to run, provisioning had another problem. Every test was talking to our providers' sandboxes, and getting those into the exact state we needed was often difficult, or sometimes not even possible.
For example, UK banks can stop sharing transactions older than 90 days until the user signs in again. Each bank handles this differently, but our open banking sandbox only has one mock bank, so we couldn't really test those different behaviours.
Our coding agents were running into the same problem. They could make changes, but when it came to testing them, they often struggled to get the app into the right state.
So in October, we built our own sandbox. With a single command, sandbox create, we spun up an isolated, temporary version of Seapoint with fake implementations for 33 providers. In doing so, every developer or agent had their own sandbox, one that didn’t interfere with anyone else.
Importantly, the application code never knew whether it was talking to a real provider or a fake one. The only thing that ever changed was the provider URLs.
Of course, there's a trade-off. Our fakes might slowly drift away from how the real providers behave, and we might not even notice.
Step four: checking the fakes
To find out when a fake has drifted, we want to run every test in two ways.
A hermetic run uses the sandbox, with every provider faked. It's fast enough for every PR and agent task. Meanwhile, a live run uses staging and the providers' real sandboxes. It's slower, so it runs less often, and always includes payment flows.
If a test passes with the fake but fails against the real sandbox, that's usually a sign that our fake has drifted from the actual provider.
We're still pretty early with all of this, but so far it’s looking positive. And as Seapoint continues to grow, the data we get from our sandbox tests is only going to get more important.