Edit the page Number in the headers and loop it till the last page POSTMAN

Jprajapatipickering · July 30, 2019, 4:04pm

Hi Team,

I am trying to get all the pages from API.

We have 71 pages in our API. So, I wanted to use loop to print all the pages in postman.
So, I got the total page and page number from JSON API. I used it in a for loop along with the send request.

However, I did not get expected results. It just prints the page 1.

Also, I tried to change page parameter manually. That works.

But, when I try to script it. It does not change the page number.

Please advise on how to do this.

Sample script.
let jsondata = pm.response.json();
let page_number = jsondata.headers.page_info.page_number;
let total_page = jsondata.headers.page_info.total_page;

for(let I=page_number; I<=total_page;I++)
{
//console.log(I);
pm.sendRequest({
url: ‘’,
method: ‘GET’,
headers:{
page_info: {page_number:I}
},

body: {
    mode: 'raw',
    raw: JSON.stringify({ key: jsondata })
}

}, function (err, res) {
console.log(res);
});

}

Also, I tried to just use simple loop to get all the page numbers and pass that value in a variable to the headers using {{I}} this syntax. However, no success.

My Requirement is to get all the records from all the pages and save the json file.

Header Structure:

Thanks,
Jinal

kaustavdm · August 1, 2019, 6:24am

If I understand your question correctly, you are trying to access all the pages of a paginated endpoint. This is better done by looping on the current request using postman.setNextRequest(requestName), where you set the requestName to the same request.

You can use variables for the query params/headers of your request that control the pagination (depending on what the endpoint requires). You can use “Pre-request Script” to set these values based on each iteration, and avoid looping once the total number of records have been received. You can use a variable to store the data across all the pages. Assuming this variable is an array, you will have to push to that array in the “Tests” tab. In the final loop, you can access that variable which by then should have all the data that you pushed to it in the loop.

Example

Your pre-request script can look like this:

const offset = parseInt(pm.variables.get("_rec_offset"));
if (!offset) {
    pm.variables.set("_rec_offset", 0);
}

pm.variables.set("_rec_limit", 10);

And then, the Tests tab can push to the data like this:

// Push the current dataset to this array
let results = [];

// If this is not the first loop, load existing data
if (pm.variables.get("_rec_results")) {
    results = JSON.parse(pm.variables.get("_rec_results"));
}

let jsonData = pm.response.json();

// Process pagination
if (total > offset + limit) {
    // Needs paginated run
    offset = offset + limit;
    // Loop on current request
    postman.setNextRequest("Current request name");
    // Save offset
    pm.variables.set("_rec_offset", offset);
    // Add current dataset to the variable that stores all results
    pm.variables.set("_rec_results", JSON.stringify(results));
} else {
    // Final run. Set the temporary results into a final variable
    pm.environment.set("results", JSON.stringify(results));
}

Bonus

I have added an example collection which you can import by clicking the Run in Postman button below. The first request in this collection does exactly this sort of pagination on another API.

Jprajapatipickering · August 1, 2019, 12:55pm

Hi,

Thanks For your reply.

So, Our goal is to retrieve all the records from the API. for example, customer API has 40000 records. Because of the performance issue, records are paginated. Also, we can not pass the page parameter on URL because it is not designed that way. In addition, the maximum record limit is set to 200.

So, We want to get all 40000 records from those pages.
So, we are trying to automate this using loop.

Hope this provides clear explanation.

kaustavdm · August 1, 2019, 2:20pm

How is the endpoint designed to access pagination if not as part of the URL? I assume there would be some request headers or query parameters or URI segment that you need to set. Whichever method your endpoint uses, you will have to use a variable and keep on updating it on a loop.

Also, if there are that many records, I won’t recommend storing them in the memory, but rather write them to some external data store, like Google Sheets document or a Airtable or something similar.

The core idea will remain the same.

Jprajapatipickering · August 1, 2019, 2:41pm

So, the limitation is we can pass the page parameter on URL Itself. so I can not pass that variable in the URL and send the request till the loop ends. Also, We can just use GET method.

That page parameter is in the header (Please see attached screen shot in my first email).

so lets see when I send a request to an API.

It will give me the body which contains the actual data and headers which provides information regarding sorting and paging. Also, I tried to change those parameter with the code. But it’s not working. That’y I put my query in this community.

However, When I change the page number from the postman application header page. It gives me the result. But now, I do not want to change manually from the headers. I would rather use loop.

Jprajapatipickering · August 1, 2019, 2:42pm

Also, they did not set any limit or offset while designing the page.

So, we can not pass that too.

singhsivcan · August 2, 2019, 9:13am

Hi @Jprajapatipickering, I hope you don’t mind me dropping in!
In addition to @kaustavdm 's answer which clearly explains how you can build the flow of chaining requests, I’d like to help you out with the header parameter.

So to go about that, you can replace your page_info's header value to a variable completely, i.e.: {{page_info}}

Now, in your pre-request script you need to build this request header.
You can read the values of the current page number and other things from an environment variable or some other scope (refer Kaustav’s answer here), and to build the header do something like this:

let totalRecordsPerPage = 2500; // Read this value from some environment or the chaining loop that you've built earlier
  pageNumber = 55; // Read this value from some environment
  pageInfo = {
    "total_records_per_page": totalRecordsPerPage
    "page_number": pageNumber
  };

// Set this new header to a local variable, which will be used during request sending
pm.variables.set('page_info', JSON.stringify(pageInfo));

Jprajapatipickering · August 2, 2019, 1:08pm

Hello SIVCAN,

Thanks for your update. Much Appreciated.

As I mentioned, we know how to setup this parameters in headers and as well as in pre-request script. My question is that we are not able to retrieve response with changed page parameter value.

Also, @kaustavdm, the script which you provided give us the same results as our script.

Do guys think that It may be the issue from the server side pagination set up?

Thanks,
Jinal

kaustavdm · August 2, 2019, 1:58pm

At this point, I’m inclined to say yes, it looks like some quirks of the API itself. The loop and what Sivcan mentioned are what you need to access paginated data in Postman.

technical-astronaut5 · September 19, 2021, 6:33am

sorry reply in this post, I also have the same goal for this topic. I also want to download all pageNumber and loop it to the last page.
My case has a total of 216 pages, and each page is limited to 1000 records.
I’m learning how to write a pre-request script but don’t know how (because I don’t know much about IT)
same post owner, but my case is Query params instead of header.
Can anyone help me in this case? Thanks a lot!!!