Notify Slack When Test Failures Are Found

slack
integration

#1

Since I’m a “new member” I’m unable to post links/screenshots so bare with me…

Hi All,
I’ve shown this to a few people starting out with Postman in the Slack channel and thought it might have some user to others who are new to Javascript/Postman (I’m sure the more experienced users can suggest a better way of doing this :stuck_out_tongue: )

What This Does:

  • Every test run I have a “passed” and “failed” variable which gets set to 0 at the start of each test run.
  • Every time a test passes or fails, +1 is added to the passed or failed postman variables.
  • At the end of the test run, Postman checks if the failed variable value is greater than 0
  • If it’s not, I use the postman.setNextRequest(null) command to end the test on that request.
  • If the failed variable is greater than 0, Postman will automatically go to the next request which is a Slack hook.
  • This will send a message saying that the particular test collection has just ran and it contained X failures.

How To Do This:

  1. I have a ‘setup’ request which I use to clean and setup the environment for testing. This involves clearing all environment variables that may have been left over from the previous test run. This will also set the “passed” and “failed” variables to 0.
// This is a GET request calling https://postman-echo.com/delay/0
// Place the below in the Pre-Request Script tab
postman.clearEnvironmentVariables();
postman.setEnvironmentVariable("passed", 0);
postman.setEnvironmentVariable("failed", 0);
  1. In the same request, in the Tests tab, I have the below code. Don’t crucify me for using evals!
// Place the below in the Tests tab in the same request

// Instructions on what's happening below:
// Creates a postman variable called 'testResultCounter'
postman.setEnvironmentVariable("testResultCounter", () => {
// Creates an array which holds true or false values for tests that pass or fail
    var test_results = Object.values(tests);
// It will loop through the above array:
    for(var i = 0; i < test_results.length; i++) {
        if(test_results[i] === true) { 
// If it finds a true value, it converts the "passed" Postman Variable variable into an actual number so we can do maths with it. Postman Variables are converted to "string" when saved.
//A variable called thisPassed is created and it equals the number stored in the "Passed" Postman Variable. If this is the first time running, it will be 0 obviously.
            thisPassed = parseInt(postman.getEnvironmentVariable("passed"));
// This will then get the value from the above "thisPassed" variable and add one to it. So if it's the first time running, 0 + 1 = 1.
//Then it'll overwrite the current postman variable by adding one to it
            postman.setEnvironmentVariable("passed", thisPassed+ 1); 
//This will keep repeating for every passed test by adding 1 to the "passed" Postman Variable.
        } else {
// Now, if it finds a false in the array. It'll do the same as above but for the "failed" variables.
// So basically, every time "false" is found in the array +1 is added to the "failed" postman variable 
            thisFailed = parseInt(postman.getEnvironmentVariable("failed"));
            postman.setEnvironmentVariable("failed", thisFailed + 1);
        }
    }
});

Since we used a “dirty” eval we can now re-use the above code in all our tests by just calling “testResultCounter”.
Re-usable code in Postman!? Nice!

  1. Now, in the Tests tab for all your future requests, you can just include the one line code to call the above code.
//Every test failure, +1 to the "failed" postman variable.
//Every test passed, +1 to the "passed" postman variable.
eval(environment.testResultCounter)();
  1. The second-to-last request in my collection is always a “teardown” phase. This will clear all environment variables, with the exception of the passed/failed variables.
// Once again, I'm just hitting the GET "https://postman-echo.com/delay/0" request.
// Place the below in the Pre-Request Script tab
// Store the passed and failed Postman Variables in memory so they're not truely "deleted"...yet.
var passed = environment.passed;
var failed = environment.failed;

postman.clearEnvironmentVariables();

//Re-Create the Postman Variables and give them the values saved in memory :)
postman.setEnvironmentVariable("passed", passed);
postman.setEnvironmentVariable("failed", failed);

If you find the above code ugly, feel free to upvote my Feature Request :slight_smile:
https://github.com/postmanlabs/postman-app-support/issues/3106

  1. In the Tests tab in the same request:
if(environment.failed > 0) {
// If there were Test Failures go to the next request as planned.
    postman.setNextRequest("EXACT NAME OF SLACK REQUEST WILL GO HERE");
} else {
// If no failures were found, don't go to the next request as usual. Just end the test here :)
    postman.setNextRequest(null);
}
  1. Go to https://api.slack.com/apps and create an account/register your app.
  • Don’t feel intimated by the word “app”. This will just allow you to use POST API calls in Postman to send messages to either yourself or a channel in Slack.

  • Once registered, go to Your Apps at the top right.

  • Click your “app name”

  • Then click ‘Incoming Webhooks’

  • Make sure the button is toggled to ON for “Active Incoming WebHooks” at the top right.

  • Click “Add New WebHook To Workspace” and then select the channel or user you want to be notified from the dropdown.

  • I just selected myself as I personally want to be “pinged” on Slack whenever the API tests fail.

  • Once you’ve confirmed everything you’ll be given a URL that you can now use in Postman :smiley:

  1. Click copy and go back to Postman and create a POST request to the WebHook URL.

  2. In the Body, click the ‘raw’ radio button and select 'JSON(application/json) from the menu to the right of it.

  3. Now, in the POST body create a JSON object which has the “text” field and whatever value that field has will be displayed in Slack.

{
	"text": "Postman:\n {{failed}} Tests have failed PANIC!!!"
}
  1. Send the POST request and you should be notified in Slack!

image

  1. The Slack request is the last request in my collection and I’ve named it “Slack - Postman Test Reporter”
  2. In the second-to-last request (which I called teardown) I’ll rename the setNextRequest command to use the Slack Request.
if(environment.failed > 0) {
    postman.setNextRequest("Slack - Postman Test Reporter");
} else {
    postman.setNextRequest(null);
}
  1. Now, if you run the collection through the Postman Runner or Newman, it’ll use the setNextRequest() depending if any failures were found.

Feel free to learn more about the Slack integration at the below URL (such as adding the Postman icon and much more)

https://api.slack.com/incoming-webhooks

It’s been a long time since I configured the Slack integration so apologies if the Slack section of the tutorial was a bit bare…

Hopefully you get some value from this :+1:

Advantages:

  • As soon as a Postman Monitor contains a failure you/the relevent team are notified imediately on Slack (I know emails are sent from the Monitor serivce, however it’s harder to avoid an IM than an email)
  • You’re notified imediately if a test fails in your CI (TeamCity etc.)
  • If you’re running a larger test (100’s of iterations/requests) in the Postman Runner, you can minimise the window and let the test run and be notified once it’s finished and found any failures.

Disadvantages:

  • If you’re sharing the collection with your team, if a team member runs the test locally and has made changes to the tests on their machine - If a failures occur from their test run the notification is sent to the Slack room which may cause some false alarms :stuck_out_tongue:

#2

Woah! Awesome. Thank you for sharing!


#3

@abhinav
ACTUALLY! Is this possible when using the new pm.tests() ?
Seems like the tests object is now empty when using the pm.tests command :frowning:


#4

pm.tests might override existing behavior if you decide to use it AFAIK. Let me check with the team.


#5

Hi @abhinav
Any update on this?
I’d like to edit the original post so it’ll work with the new pm.test() commands


#6

@pfarrell pm.test function and the tests object are completely separate and they don’t influence each other. @kamalaknn we could keep a note that when we next improve on run control API is sandbox, we could also expose API to know how many tests failed.


#7

@shamasis this sounds useful. We track assertion count internally. It would be useful to expose it with passed, failed and skipped count as a summary. Probably pm.test.summary?


#8

Yes, can we also expose other items too?
Such as the name of the test collection and even the sub-folders within that collection too?
That way I can dynamically create the Slack Message to say something like:

{
        "username": "Postman Reporter",
        "text": pm.currentTestCollection() + ":\n {{failed}} Tests have failed"
}

If I’m running my test collection named “BB - Order History” and failures occur, Slack will output

image


#9

Hi @shamasis @kamalaknn :wave:t3:,

is pm.test.summary available ?
as of now, we are getting null object?
any news upon this?

as we all have migrated from tests[ ] to pm.test( ) .

it would be very very useful to know the count of passed and failed and skipped tests and the tests that are actually passed and tests that are failed.

thanks
@gopikrishna4595 :cowboy_hat_face:


#10

any update @kamalaknn and @shamasis :grinning: ?


#11

Hi @kamalaknn | @shamasis ,

can you plz provide any info on this ?
much awaiting for your response.

Thanks
@gopikrishna4595 :cowboy_hat_face:


#12

This has to be a feature request as we no longer store the test result and instead bubble it out of Postman sandbox in real time.

There’s an undocumented API pm.test.index() to get the total test in the script at the time of calling the function. We could add pm.test.result() to get total and failure count.

@udit.vasu


#13

Thanks you @shamasis and @udit.vasu for the info.

checked pm.test.index() which is giving the total count of tests.

we are highly anticipating for the availability of pm.test.result() api to get pass and failure count.

Thanks and Regards
@gopikrishna4595 :cowboy_hat_face:


#14

Hey,

This tool looks really cool! :slight_smile:

As a hackathon this week, I had a go at creating something that could kick off Newman collection runs from with Slack.

I would love some feedback on it.

Cheers,

Danny