Postman assertion tests are passing when they actually should fail

I hope you guys can help me here. I’m not sure if I’m missing something or it’s a Postman bug:

As you can see in the screenshot, JSON response shows a validation error because some of the required fields are missing from the body. However, my test is still passing, even tho there are no “name”, “logline” or “userdId” properties in the json response. It should have failed at the very first step because jsonData.name would be “undefined”.

Notes:

  • So far, it seems like Postman is skipping pm.test() and marking it as Passed if the response is some kind of validation error. So it doesn’t matter what the assertion is, it will always pass.

Any advice is appreciated. Thanks.


Postman v. 7.3.4
Mac OS

1 Like

Hey @atereshkov,

Are you able to add an image of the test tab too? What do you have in the pre-request tab? What version of the app is this happening on?

Have you tried commenting the tests out or causing a failure to see something different happening?

Might be worth adding a status code check in there too as that would alert straight away able that endpoint returning a 400.

It’s quite difficult to give a decent answer without a little more context about what you’re doing. :pensive:

Hi @dannydainton, Thank for the reply.
I’ve added Pre-request script as well as the Postman version.

The pre-request script works fine and I can see those variables in my Environment variables.

I’ve changed the test to check “jsonData.nnnnn” property, which is non-existing and it’s still passing.

I did add a status check and that’s the only test that is failing now (as it should).
What ridiculous is that even “pm.expect(44444).to.eql(55555);” is passing.

Have you tried duplicaring or recreating the request in a new tab and see if there is any change in behaviour?

You’re on a slightly older version of the app, are you in a position to update this to the latest version and see if the issue still persists?

I can’t think of anything that is not an environment/caching type issue.

Without being there and seeing the state of the app myself, it’s tricky to offer anything more than a few things to try and some guess work :pensive:

I have tried to duplicate and recreate it. it’s actually reproducible with other PUT requests as well. Considering that I have just found a typo in one of my other assertions, and it was still passing, I assume that this issue has been happening for a while :worried:

Postman fails to update to 7.8.0: “Something went wrong while trying to update your app. Check the DevTools for more details.”. When I’m trying to download it manually, it only gives me 7.3.4 build. (I’m on Mac)

I’ll try to do some more testing around to get more data and try to narrow it down.

Thanks for your help @dannydainton

If you quit the app and then try the update again that should help.

Interesting that it’s only doing that for a PUT request :thinking:

Something like:

pm.expect(jsonData.foo).to.eql(pm.environment.get("bar"));

can pass because foo is undefined and pm.environment.get(“bar”) is undefined as well.

Can you maybe share a collection?

@vdespa I can’t share the collection, unfortunately. I hope this screenshot will clarify some stuff for you.

I did some more testing around and here’s what I’ve found:

  • The console.log() function doesn’t have the output if it’s within the pm.test(), even though it should.
  • So I’m assuming that Postman is skipping the test and marking it as PASS for some reason.

@atereshkov

think I have spotted your error:

the way you have written your test, it cannot fail, as it does not contain any assertions.

pm.test accepts two params, the test name, and a callback function.

You have closed the function right after the test name, this is why it is always true.

WRONG:

pm.test("Body is correct"), function () {
    pm.expect(1).to.eql(2);
};

RIGHT:

pm.test("Body is correct", function () {
    pm.expect(1).to.eql(2);
});

I always preach the following: make sure you test fail when you write them.

6 Likes

Ahhh, that’s it!!

See, this is why this guy is a legend!!

It was there right in front of me and I couldn’t see it!! That’s always why it didn’t fail, when I suggested intentionally making it fail :pensive:

@vdespa :trophy:

2 Likes

You are a legend indeed!! Thank you, Valentin!
It will be a good lesson for me to always recheck my code and make sure it fails when it should.

1 Like

Glad could help. Thank you guys!

1 Like