I have some tests where I pass data to an endpoint and then make another call to verify the data is correct. I repeat this a number of times with different data sets and sometimes the data verification call is made before the data setting call is completed.
The first call is to an internal service that calls a number of other services and so it can take up to 20s for the call to return a response. The second call is almost instantaneous.
When I run the tests using newman locally then the calls are handled correctly and the tests pass. When the tests are run using jenkins and newman then some of the tests fail intermittently as the 2nd call is run before the 1st is completed and the data is not correct.
After some investigation I worked out why my collection is failing.
I tried the flag as recommended and it had no effect on the run results. So I console.log’d everything and found out that my dynamic request body is not being filled correctly in the pre-request state.
I used a data table of about 20 variables and in the pre-request tab I would reset the values and then set them based on the data table. After it’s gone through about 3 iterations of this, one of the fields would not get filled and remained empty causing my tests to fail. I’ve changed the order and it is always after 3-4 tests that it fails.
I’ve worked around this by duplicating the request and hardcoding the values rather than using a data table.