I am trying to implement a test collection running against an API hosted in Kubernetes. The API implements a ready endpoint that will return a 200-OK status code as soon as the service is ready to serve requests. The desired behavior should be:
- Build pipeline builds and deploys the API.
- Build pipeline runs the postman collection testing the API.
- Test collection waits for the service to be ready by retrying to call the
/readyendpoint. (E.g. 10 times with a delay of 10 seconds between calls.)- Any failed calls during the retry can be ignored and they should not affect the test result.
- Test collection should be executed after the service is ready. Right now I am just continuing with the test execution if the ready probe fails and obviously all tests will then fail as well as the service was not started. This behavior is fine but I would also be ok with aborting/failing the complete test if the service is not ready.
I found the following article that describes how to implement a "wait for ready"check inside the pre-request of a postman collection: http://stevetarver.github.io/2018/03/27/postman-waiting-on-pod-ready.html
This works like a charm in the Postman Collection runner.
However, executing the same collection inside newman results in a failed request as the call to the /ready endpoint has failed. Executing the collection from within Azure Devops then causes my tests to be marked as failed when the service was not ready to respond to one or more ready request. (Which is expected as the service has just been deployed and is now starting…)
Is there:
- A better way to wait for an API to be ready inside newman?
- Some way to allow specific request to fail and not consider it as a test failure?
- Tell newman to ignore the all failed requests and just focus on test-scripts?
Attached is a minimum sample collection showing the issue. (It is configured to some local API which happens to exist on my PC. Replace http://localhost:8081/ with anything that you can connect to.)
{
"info": {
"_postman_id": "d436e8e7-0f20-4b96-82ff-e7ce60cec9f3",
"name": "PodReadyTest",
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
},
"item": [
{
"name": "Working Call",
"event": [
{
"listen": "test",
"script": {
"id": "169c9380-7eca-498e-a08f-9913845e5c09",
"exec": [
"pm.test(\"Status code is 200\", function () {",
" pm.response.to.have.status(200);",
"});"
],
"type": "text/javascript"
}
}
],
"request": {
"method": "GET",
"header": [],
"url": {
"raw": "http://localhost:8081/",
"protocol": "http",
"host": [
"localhost"
],
"port": "8081",
"path": [
""
]
}
},
"response": []
}
],
"event": [
{
"listen": "prerequest",
"script": {
"id": "30679626-27fa-4b6b-9688-e4d38eaa5849",
"type": "text/javascript",
"exec": [
"// Make a call to a URL which does not exists - ready probe will fail",
"// In a real use case retry limit would be set to > 1 and multiple retires",
"// should run to wait until the service is ready. Any failure during those",
"// retry calls can be ignored.",
"var url = 'http://someinvalidrul/health/ready';",
"var retryDelay = 10000;",
"var retryLimit = 1;",
"",
"function isServiceReady(retryCount) {",
" pm.sendRequest(url, function (err, response) {",
" // Service endpoint is not up yet - any kind of connection error",
" var notUp = (null !== err);",
" // Service is up, but still initializing or waiting on upstream partners",
" // NOTE: there is always a response object, but .code only exists if we connected",
" var notReady = (!response.hasOwnProperty('code') || response.code !== 200);",
"",
" if(notUp || notReady) {",
" if (retryCount < retryLimit) {",
" if(notUp) {",
" console.log('Service is down. Retrying in ' + retryDelay + 'ms');",
" } else {",
" console.log('Service is not yet ready. Retrying in ' + retryDelay + 'ms');",
" }",
" // If not ready, retry this function after retryDelay",
" setTimeout(function() {",
" isServiceReady(++retryCount);",
" }, retryDelay);",
" } else {",
" console.log('Retry limit reached, giving up.');",
" postman.setNextRequest(null);",
" }",
" }",
" });",
"}",
"",
"isServiceReady(1);"
]
}
},
{
"listen": "test",
"script": {
"id": "19786870-0354-45b8-b82f-23dc3573fc04",
"type": "text/javascript",
"exec": [
""
]
}
}
],
"protocolProfileBehavior": {}
}