How are people testing their applications in development?
I have a repo, “store” (example). I have Postman which I can setup to commit the collection to a Git repo.
I use a normal Git Flow. I want to commit to a feature branch. I want to create tests in that branch before I merge the develop/master. I want to run those tests on that branch, but they’ll fail on develop/master until the code is merged.
Is Postman setup to do that? It doesn’t seem like it can do that now. If not, how are other people doing this with a team?
@singhsivcan I looked and didn’t see a solution before I posted.
The problem is that Postman/Newman isn’t actually integrated with the development pipeline. It’s external to it. I want to know what/when broke my code. I figure if people are using Postman/Newman in a CI/CD pipeline, they aren’t exporting their collection and environment on every commit. That seems like a huge burden and very prone to breaking. Even then, how do you isolate your changes from everyone else’s if you’re working in a team?
I’m excited to bring my API Request GUI inline with my CI/CD pipeline, but I’m having a hard time figuring out how Newman is going to help me more than OAS and Dredd.
The integration tests get updated constantly as things change.
What is BVT? The collection I have contains all of our API endpoints. The plan is to run the tests on every PR, but we can’t do that if we are going to have tests breaking in master because we’re adding the code in an upcoming branch.
The integration tests get updated constantly as things change.
Nice. So you have multiple developers working off the same collections or do you use something else for your integration tests?
What is BVT?
Build Verification Test. These tests should give you an idea if your build is testable (not necessarily shippable).
The collection I have contains all of our API endpoints.
Great so you run tests on all your endpoints? What about critical workflows?
The plan is to run the tests on every PR, but we can’t do that if we are going to have tests breaking in master because we’re adding the code in an upcoming branch.
The integration tests should be added with the new feature.
Would you mind elaborating on your SDLC? It’s odd to hear about tests breaking in master (instead of develop).
Right now we’re using OAS 2.0 with Dredd. I’m evaluating a move to Postman.
All our endpoints have tests. At least, that they return a 200 status. Not very much is tested past that.
The SDLC isn’t unusual. How I’m describing it is, heh. What I mean is that Postman is missing a critical feature that the requests for Newman aren’t tied to the code. Say I have a collection and it is running all of the tests OK while running tests on develop and master. Then, I create a feature branch. I want to test the new feature so I add a request to the collection. That test fails for anyone that doesn’t have their environment set to my feature branch.
If I don’t have that test, how do we know if my feature works? How do we know when it broke? It doesn’t pass our “definition of done” if it doesn’t have tests so I can’t even merge it. It’s a catch 22 if the tests aren’t along side the code.
Joshua, I believe you are asking if you have a API breaking change on a branch in your GitHub and have the corresponding updates to the integration tests on a forked version of that API’s collection, can you configure your CI pipeline to use Newman and run the matching branches?
If so, in my experience, Postman does not support this. The Postman API does not expose the Postman branch name when querying collections. So even if you know the Postman branch you want to use with Newman, you won’t be able to programmatically get the collection uid. You can do some hacky things, like include a branch name in the collection description or include a collection uid in your repo. But even doing this, it would be hard to trigger a merge of the forked Postman collection in sequence with the merge of your GitHub branch.
My approach so far has been to avoid breaking API and test changes as much as possible. This allows for the master Postman collection to be updated and for CI to still function. When changes can’t be avoided, manually executing tests on a Postman branch and including evidence in the PR for reviewers (then merge the two branches manually at the same time). In the most extreme cases, we would need to block the CICD queue for that API to perform testings/merges.
My current approach is not sustainable or scalable. Hopefully, someone else has a real solution or Postman will provide better support for this. Anytime your tests and code are in different repos CICD gets much more complicated.
I think James covered it well, if your tests and features are being merged asynchronously, that can be a bit problematic. In my team we try develop the tests and feature in parallel or as soon as possible, at least this has been our interpretation of the Shift Left approach.
One other thing jumped out at me in your response:
At least, that they return a 200 status. Not very much is tested past that.
That’s a good start, but I assume your consumers of your APIs are looking for more than just status codes. Our team gets a lot of value out of asserting expected responses here. I’d start with something simple and validate one workflow that can never break.