Is it possible to have 1 html report on all collections run in newman?

tohlaymui35 · June 10, 2019, 3:42am

t=mytest1

newman run --bail -e $MY_ENV -g $GLOBAL_ENV $DIR/API/Collection/$t.collection.json --reporters=cli,htmlextra --reporter-htmlextra-title=“Platform-API-TestReport” --reporter-htmlextra-export $API_REPORT_DIR/$API_HTML_FILE 2>&1 | tee -a $API_REPORT_DIR/$API_LOG_FILE

t=mytest2
newman run --bail -e $MY_ENV -g $GLOBAL_ENV $DIR/API/Collection/$t.collection.json --reporters=cli,htmlextra --reporter-htmlextra-title=“Platform-API-TestReport” --reporter-htmlextra-export $API_REPORT_DIR/$API_HTML_FILE 2>&1 | tee -a $API_REPORT_DIR/$API_LOG_FILE

this will produce 2 html report
how can I make it into 1 report for all my collections?

mick-mcbride · June 10, 2019, 7:18pm

@tohlaymui35

If you had just one collection you could have multiple folders under it that represent the different sets of tests. If you needed different environment variables between the two tests you could have it switch those in the pre-request of each folder. Then you could only have one Newman run and one report. The reporter you are using will show the different folders that were run. It would take a bit to figure out the correct way to get the variables right for each, but I think it would work.

Mick

maschmidt · January 7, 2020, 11:55am

I’m facing the same issue as @tohlaymui35. What you generally want to have is:

One collection per service you own
One CI/CD job that does a regular health check or regression test over all services you own
A nice report that aggregates results from test runs in all your collections

I don’t see a way to do this without custom coding right now. Breaking this one general job into one job per service/collection or merging all collections into one doesn’t seam like a desirable option to me. I’d love to use newman-reporter-htmlextra but don’t find report aggregation of multiple collections as a feature.

Does anyone have experience with these kind of aggregated reports?

Cheers,
Matthias

danny-dainton · January 8, 2020, 2:40pm

Hey @maschmidt,

Unfortunately, it’s currently a single Collection run at a time. That’s the same for the native app and Newman. All the available reporters would work in the same way too (A single report per run).

There are a few NPM modules out there that would combine collections into one so that it could be run through Newman but that would come with its own set of limitations/problems.

Thanks for mentioning that you would like to use my reporter, if there was an easy way from me add the functionality you wanted in there, I would give it a go but I honestly wouldn’t know where to start.

There are a few other versions of the Newman reporters that would output the data in a format that might help you to aggregate that all together after a certain CI/CD job but I would suspect that you would need so way to store that data after each run and then have some custom logic to give you the final result you need.

maschmidt · January 8, 2020, 7:22pm

Hi @danny-dainton,

thanks for getting in touch.

I agree, merging collections into one doesn’t seam like a good option. I expect issues with collection-wide settings such as variables or authentication. Also, I tried postman-combine-collections and it doesn’t look very promising.

It should be some sort of post-processing step. By default all reporters output their a file into a /newman subfolder which is the place to store the data after each run. There could be a post-processing reporter that picks up those files and aggregates general data such as total assertions, total failed tests, total skipped test, etc. This data could be presented in an overview which links to each collection run html file separately. No need to show details such as request headers or response body from all collection runs on one page.

Here is a draft of such an overview I derived from your htmlextra report. Each row of the collection run table could link to a separate htmlextra report.

Wdyt? Just brainstorming here …

Cheers,
Matthias

grfgomez · August 10, 2020, 6:45pm

I am looking exactly at the same thing, as I plan to run a set of tests in as script. Request the user for some manual actions and update environment variables accordingly to run an additional test case set. What I wanted is to join the generated reports into one, as it is actually a single test set.

Thanks for any help!!!
Gonzalo

nishant.shah2 · September 1, 2020, 5:07pm

I am also looking for a similar thing. I need separate collections for different modules and it does not make sense to combine it.
However, Once in a week we execute all the collections via CI integration and currently it generates different reports (one per collection)…
If there is a way we can show a single aggregated report of all the collection runs, it would be awesome…I liked the example of htmlextra report shown above by @maschmidt

@danny-dainton by any chance this is in your backlog or something?
However, thanks for the current HTML Report which htmlextra provides. Its comprehensive as well as very user friendly.

Thanks,
Nishant Shah

danny-dainton · September 2, 2020, 11:40am

Nope

Newman runs a single collection at a time and with that, comes a single summary object, where the reporter gets it’s data.

It’s got nothing to do with the reporter, Newman would need to be able to combine and run multiple collections at once before I could even look at the data to see what I could do in the reporter.

rahul_rana · October 13, 2020, 1:12am

Yes, its possible.
You can use allure report to get reports from multiple collections in one reports.
Add below piece of code to all your tests.

Check this out :

You can do something like below

   newman.run({
        collection: 'your collection path ',
        environment: 'env file path',
        reporters: ['html', 'cli', 'junit', 'json', 'allure'],
        reporter: {
            html: {
                export: '/your/path'
            },
            junit: {
                export: '/your/path'
            },
            json: {
                export: '/your/path'
            },
            allure: {
                export: 'results/allure'
            }
        },
        color: true
    }

Once execution is complete, run following commands:

npx allure generate results/allure --clean -o results/allure-htmlReport
npx allure open results/allure-htmlReport

before running above commands, make sure allure CLI is installed Allure Framework

nishant.shah2 · November 9, 2020, 12:38pm

Thanks @rahul_rana for sharing this.
It does collate results of multiple collections and show in a single allure report.
However, one thing I found weird in the report is :-
Under the SUITES section, it populates list of all the folders (inside all the collections) which were executed, i.e. it does not bifurcate result at the collection level and show.

But naturally, anyone would want the bifurcation at collection level…i.e. Under the SUITES section, list of all the collections executed should appear -> On clicking a collection it should populate list of all the folders in that collection which got executed -> On clicking a folder it should populate list of all the Requests inside that folder which got executed

Is there a way by which I can achieve it ?

Thanks in advance,
Nishant Shah

rahul_rana · November 9, 2020, 11:29pm

@nishant.shah2 yeah, it is bit annoying. But you can use this workaround.

Create a folder at the root of your collection folder and name it whatever you want to see in the report
Put all your test folders or tests under the above folder.

see the example below.

Example:

Weather(collection)

Weather API(create this folder)
```
  -- Test folder 1
  -- Test folder 2
```

felipecassiors · March 2, 2021, 6:23pm

Is it possible to automate this somehow without needing to change the collections?

felipecassiors · March 3, 2021, 5:18pm

Okay, I forked the original allure reporter and added a new option for that in case someone needs (I do).

Here’s how to use it:

$ npm install --global newman @felipecrs/newman-reporter-allure
$ newman run https://www.getpostman.com/collections/d48ca7a0e871d2f530e2 --reporters cli,@felipecrs/allure --reporter-@felipecrs/allure-collection-as-parent-suite
$ allure generate --clean
$ npx serve allure-report

mariaelisabethweegar · April 22, 2021, 6:23am

If you are ok with not having a html-report but reading the report in terminal, you can create a bash-script that run through all collections.
Echo out the name of Collection.
You will get a lot of information about every request and a table with summary for each collection.
We got tired of all these information and made a filter in ‘awk’ . Filter out only if there is any error.

newman run --environment ${TEST_COLLECTION} --reporters progress,cli --color on | filter.awk

And in the awk-file you only need a few lines, newman writes the error with # first so:

#!/usr/bin/awk -f
  ## Print errors
  /\#/,EOF { print }

annapoornagb · October 17, 2023, 2:50am

@danny-dainton Noticed that in summary page there is total & failed column. When assertion of the perticular Request/test fails it means that Request fails, this failed request count is not updated in failed column next Total
Only failed assertion list are updated

danny-dainton · October 17, 2023, 7:12am

Hey @annapoornagb

In the context of Newman, where the data is coming from to populate that table, those are 2 different metrics.

A request can be successful and have multiple failed assertion.

What would like to see there?

The report you see displayed is only a template and is something that I created to visualize the underlying data. If you’d like to present that same data in a totally different way - that can be possible with a new template.

Templates can be created and passed into the reporter when running the Collection.

newman run collection.json --reporter-htmlextra-template <template.hbs>

annapoornagb · October 26, 2023, 6:06am

Thank you was able to customise the report