Notify Slack When Test Failures Are Found

Since Iā€™m a ā€œnew memberā€ Iā€™m unable to post links/screenshots so bare with meā€¦

Hi All,
Iā€™ve shown this to a few people starting out with Postman in the Slack channel and thought it might have some user to others who are new to Javascript/Postman (Iā€™m sure the more experienced users can suggest a better way of doing this :stuck_out_tongue: )

What This Does:

  • Every test run I have a ā€œpassedā€ and ā€œfailedā€ variable which gets set to 0 at the start of each test run.
  • Every time a test passes or fails, +1 is added to the passed or failed postman variables.
  • At the end of the test run, Postman checks if the failed variable value is greater than 0
  • If itā€™s not, I use the postman.setNextRequest(null) command to end the test on that request.
  • If the failed variable is greater than 0, Postman will automatically go to the next request which is a Slack hook.
  • This will send a message saying that the particular test collection has just ran and it contained X failures.

How To Do This:

  1. I have a ā€˜setupā€™ request which I use to clean and setup the environment for testing. This involves clearing all environment variables that may have been left over from the previous test run. This will also set the ā€œpassedā€ and ā€œfailedā€ variables to 0.
// This is a GET request calling https://postman-echo.com/delay/0
// Place the below in the Pre-Request Script tab
postman.clearEnvironmentVariables();
postman.setEnvironmentVariable("passed", 0);
postman.setEnvironmentVariable("failed", 0);
  1. In the same request, in the Tests tab, I have the below code. Donā€™t crucify me for using evals!
// Place the below in the Tests tab in the same request

// Instructions on what's happening below:
// Creates a postman variable called 'testResultCounter'
postman.setEnvironmentVariable("testResultCounter", () => {
// Creates an array which holds true or false values for tests that pass or fail
    var test_results = Object.values(tests);
// It will loop through the above array:
    for(var i = 0; i < test_results.length; i++) {
        if(test_results[i] === true) { 
// If it finds a true value, it converts the "passed" Postman Variable variable into an actual number so we can do maths with it. Postman Variables are converted to "string" when saved.
//A variable called thisPassed is created and it equals the number stored in the "Passed" Postman Variable. If this is the first time running, it will be 0 obviously.
            thisPassed = parseInt(postman.getEnvironmentVariable("passed"));
// This will then get the value from the above "thisPassed" variable and add one to it. So if it's the first time running, 0 + 1 = 1.
//Then it'll overwrite the current postman variable by adding one to it
            postman.setEnvironmentVariable("passed", thisPassed+ 1); 
//This will keep repeating for every passed test by adding 1 to the "passed" Postman Variable.
        } else {
// Now, if it finds a false in the array. It'll do the same as above but for the "failed" variables.
// So basically, every time "false" is found in the array +1 is added to the "failed" postman variable 
            thisFailed = parseInt(postman.getEnvironmentVariable("failed"));
            postman.setEnvironmentVariable("failed", thisFailed + 1);
        }
    }
});

Since we used a ā€œdirtyā€ eval we can now re-use the above code in all our tests by just calling ā€œtestResultCounterā€.
Re-usable code in Postman!? Nice!

  1. Now, in the Tests tab for all your future requests, you can just include the one line code to call the above code.
//Every test failure, +1 to the "failed" postman variable.
//Every test passed, +1 to the "passed" postman variable.
eval(environment.testResultCounter)();
  1. The second-to-last request in my collection is always a ā€œteardownā€ phase. This will clear all environment variables, with the exception of the passed/failed variables.
// Once again, I'm just hitting the GET "https://postman-echo.com/delay/0" request.
// Place the below in the Pre-Request Script tab
// Store the passed and failed Postman Variables in memory so they're not truely "deleted"...yet.
var passed = environment.passed;
var failed = environment.failed;

postman.clearEnvironmentVariables();

//Re-Create the Postman Variables and give them the values saved in memory :)
postman.setEnvironmentVariable("passed", passed);
postman.setEnvironmentVariable("failed", failed);

If you find the above code ugly, feel free to upvote my Feature Request :slight_smile:

  1. In the Tests tab in the same request:
if(environment.failed > 0) {
// If there were Test Failures go to the next request as planned.
    postman.setNextRequest("EXACT NAME OF SLACK REQUEST WILL GO HERE");
} else {
// If no failures were found, don't go to the next request as usual. Just end the test here :)
    postman.setNextRequest(null);
}
  1. Go to https://api.slack.com/apps and create an account/register your app.
  • Donā€™t feel intimated by the word ā€œappā€. This will just allow you to use POST API calls in Postman to send messages to either yourself or a channel in Slack.

  • Once registered, go to Your Apps at the top right.

  • Click your ā€œapp nameā€

  • Then click ā€˜Incoming Webhooksā€™

  • Make sure the button is toggled to ON for ā€œActive Incoming WebHooksā€ at the top right.

  • Click ā€œAdd New WebHook To Workspaceā€ and then select the channel or user you want to be notified from the dropdown.

  • I just selected myself as I personally want to be ā€œpingedā€ on Slack whenever the API tests fail.

  • Once youā€™ve confirmed everything youā€™ll be given a URL that you can now use in Postman :smiley:

  1. Click copy and go back to Postman and create a POST request to the WebHook URL.

  2. In the Body, click the ā€˜rawā€™ radio button and select 'JSON(application/json) from the menu to the right of it.

  3. Now, in the POST body create a JSON object which has the ā€œtextā€ field and whatever value that field has will be displayed in Slack.

{
	"text": "Postman:\n {{failed}} Tests have failed PANIC!!!"
}
  1. Send the POST request and you should be notified in Slack!

image

  1. The Slack request is the last request in my collection and Iā€™ve named it ā€œSlack - Postman Test Reporterā€
  2. In the second-to-last request (which I called teardown) Iā€™ll rename the setNextRequest command to use the Slack Request.
if(environment.failed > 0) {
    postman.setNextRequest("Slack - Postman Test Reporter");
} else {
    postman.setNextRequest(null);
}
  1. Now, if you run the collection through the Postman Runner or Newman, itā€™ll use the setNextRequest() depending if any failures were found.

Feel free to learn more about the Slack integration at the below URL (such as adding the Postman icon and much more)

https://api.slack.com/incoming-webhooks

Itā€™s been a long time since I configured the Slack integration so apologies if the Slack section of the tutorial was a bit bareā€¦

Hopefully you get some value from this :+1:

Advantages:

  • As soon as a Postman Monitor contains a failure you/the relevent team are notified imediately on Slack (I know emails are sent from the Monitor serivce, however itā€™s harder to avoid an IM than an email)
  • Youā€™re notified imediately if a test fails in your CI (TeamCity etc.)
  • If youā€™re running a larger test (100ā€™s of iterations/requests) in the Postman Runner, you can minimise the window and let the test run and be notified once itā€™s finished and found any failures.

Disadvantages:

  • If youā€™re sharing the collection with your team, if a team member runs the test locally and has made changes to the tests on their machine - If a failures occur from their test run the notification is sent to the Slack room which may cause some false alarms :stuck_out_tongue:
11 Likes

Woah! Awesome. Thank you for sharing!

@abhinav
ACTUALLY! Is this possible when using the new pm.tests() ?
Seems like the tests object is now empty when using the pm.tests command :frowning:

pm.tests might override existing behavior if you decide to use it AFAIK. Let me check with the team.

Hi @abhinav
Any update on this?
Iā€™d like to edit the original post so itā€™ll work with the new pm.test() commands

@postman-paul540 pm.test function and the tests object are completely separate and they donā€™t influence each other. @kamalaknn we could keep a note that when we next improve on run control API is sandbox, we could also expose API to know how many tests failed.

@shamasis this sounds useful. We track assertion count internally. It would be useful to expose it with passed, failed and skipped count as a summary. Probably pm.test.summary?

1 Like

Yes, can we also expose other items too?
Such as the name of the test collection and even the sub-folders within that collection too?
That way I can dynamically create the Slack Message to say something like:

{
        "username": "Postman Reporter",
        "text": pm.currentTestCollection() + ":\n {{failed}} Tests have failed"
}

If Iā€™m running my test collection named ā€œBB - Order Historyā€ and failures occur, Slack will output

image

1 Like

Hi @shamasis @kamalaknn :wave:t3:,

is pm.test.summary available ?
as of now, we are getting null object?
any news upon this?

as we all have migrated from tests[ ] to pm.test( ) .

it would be very very useful to know the count of passed and failed and skipped tests and the tests that are actually passed and tests that are failed.

thanks
@gopikrishna4595 :cowboy_hat_face:

any update @kamalaknn and @shamasis :grinning: ?

Hi @kamalaknn | @shamasis ,

can you plz provide any info on this ?
much awaiting for your response.

Thanks
@gopikrishna4595 :cowboy_hat_face:

This has to be a feature request as we no longer store the test result and instead bubble it out of Postman sandbox in real time.

Thereā€™s an undocumented API pm.test.index() to get the total test in the script at the time of calling the function. We could add pm.test.result() to get total and failure count.

@udit

2 Likes

Thanks you @shamasis and @udit for the info.

checked pm.test.index() which is giving the total count of tests.

we are highly anticipating for the availability of pm.test.result() api to get pass and failure count.

Thanks and Regards
@gopikrishna4595 :cowboy_hat_face:

Hey,

This tool looks really cool! :slight_smile:

As a hackathon this week, I had a go at creating something that could kick off Newman collection runs from with Slack.

I would love some feedback on it.

Cheers,

Danny

1 Like

Really looking forward to - pm.test.result()

Hi there! Guys, do you know where to read the development of this topic? I would like to study it in more detail. I know many companies who are using Slack and they would be interested to know such things. What area code is located for the forum? I will also look for similar information on other resources. If I find something useful, I will post here for everybody to have access.

Use newman as a library in node and console the summary in for loop
Var summary=summary [I]. Errors. Test and stringify the errors and then use node mailor for sending mails with custom html reports and pass the error variables in that.

Hey! I was just curious as to why you have to clear the environment variables at the beginning. Is it possible for this to work successfully without that?

Just chiming in to let yā€™all know Iā€™ve moved this thread to Help instead of Community Showcase. Hope this can keep the discussion alive and the answers flowing.

@postman-paul540 love seeing one of your early posts!

1 Like

My suggestion is to make a video tutorial, itā€™s very helpful:)