Need some help to get the Postman Results integrated with the XRay test cases.
I’m not too familiar with that tool and how those results are uploaded/imported.
Would you be able to explain this in a bit more detail, please?
Something that you could look into is using Newman with a Custom Reporter - You could create your own reporter and feed the results back to XRay (maybe using the JIRA API or something).
More information about Custom Reporters can be found here: https://github.com/postmanlabs/newman#creating-your-own-reporter
Thanks for the response. Currently we are using JIRA and would like Postman integrated with it so that we can send the reports back to JIRA and update the user stories accordingly. The test case we have in JIRA have the corresponding API tests in Postman. So the ask is whenever there is a deployment related to these test cases after i execute the test scripts in Postman the results should be automatically updated to the related test cases in JIRA. Your help in this regard is much appreciated.
Thanks in advance
Are you running these as part of your CI/CD pipeline using Newman?
We want to eventually, but for now i would like to see how can i get JIRA updated with the Postman results
If you’re not running your Collections with Newman, getting the results will be a bit more involved.
I would guess that you would need to use
pm.sendRequest() in multiple places to push these over or store the results as you go through the Collection, then have a final request send them over via the JIRA API.
Using Newman would make this a lot less painful as you could use one of the many reporters to export the data in a cleaner way.
I don’t think that there is one created for JIRA but there is a community created one for confluence, that might get you closer to what you need.
Hi Danny, thanks for the information. I have tried using the commands mentioned in the link to export results into the Confluence page. But when executing the command its says Access Denied. How can i export the Postman API results into a specific folder in confluence?
We are using X-Ray for JIRA as well. Or test collections run in Bamboo using Newman and then we have customized the JUNIT Newman Reporter to publish in a format that X-Ray can understand and import our tests into JIRA. All of our tests now are automatically imported into JIRA and show results.
I massively encourage you to publish that reporter for the whole community to benefit from @mick.mcbride.
You know for sure there’s someone here that would get value from it but I’m sure many other Postman uses would too
I forgot that we already have it published.
I know I personally got a lot out of creating
htmlextra and seeing different folks using it so hopefully seeing more people using this will do the same.
Hi! We are trying to integrate Newman, executed by a Jenkins Job, with XRay as well. We do have a set of test cases that we are executing but we need to run several iterations for those test cases with different data. We have been struggled with exporting from Newman the results of the several iterations. We don’t want to have aggregate information, we need to know which test case fail for which data set. Does the reporter you published had that feature? Thank you very much for your help!
Looking at the project description for that reporter - It says that it’s a full non-aggregated report so I would say yes but I think that @mick.mcbride would be able to clarify that though.
The issue we had with the default junit reporter and XRay for JIRA was that it imported not just the test case, but any of the tests that were run. For example for a single test case in the collection we might have 8 tests that get run against it, so the junit reporter would create 9 test cases in XRay for JIRA.
The customize newman-reporter-juintxray that my coworker put together basically allows only the test case (test case name) to be created in XRay for JIRA. Then each successive run of the tests it records the pass/fail of each iteration. If the test case name is change it will create a new test case in XRay for JIRA.
Let me know if this is what you were looking for,
Thanks @mick.mcbride and @dannydainton ! Actually we do have your same issue, but the problem we are trying to solve is a different one. We have a Json Data file with data to execute multiple iterations of the test cases. So we run the same collection of Newman test cases 22 times, each time has its own data set, and the results might be different at each iteration: one test case could pass in one iteration and could fail at other iteration. The problem we are facing is that, we can’t import back to Jira the results of the test cases for the 22 iterations, it only imports the result of the last iteration. So my question was if your reporter could import back to Jira the results of the test cases collection but for the 22 iterations? I’m not sure how would be the best approach to do that, I guess by having one xRay test execution per Newman execution iteration maybe?
awesome work, thanks!