Mantis Bugtracker          
testlink.org

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0005240TestLinkInternal issuepublic2012-09-20 09:422012-09-21 11:00
Reportersujataverma3 
Assigned To 
PrioritynormalSeverityminorReproducibilityalways
StatusnewResolutionopen 
PlatformWindowsOSWinXPOS Version
Product Version1.9.4 (2012 Q3 - bug fixing) 
Fixed in Version 
Summary0005240: Query: Regarding issue 5226 resolution
DescriptionI have doubt regarding issue 5226 resolution, I do not agree with results given by @asimon.

http://mantis.testlink.org/view.php?id=5226#c17499 [^]

For filter "All builds" the assumption is "the same test case" has to be "passed/failed/blocked/not run" on both the builds.

So if Build1->Platform A->TC1-> Passed, Build2->Platform A-> TC1-> Passed , so if we put filter "Passed/All builds" the result displayed is 1.

we had some discussion on this in below bug:
http://mantis.testlink.org/view.php?id=5142 [^]


Steps To Reproduce
Assuming the steps to be done to reproduce below cases :

1. Create new Project.
2. Create new Test Plan. Add one Platform A.
3. Create new build.
4. Create new test suite.
5. Create 3 test cases in the above test suite.
6. Add these 3 test cases to the Test Plan and assign to myself. Add platform.
7. Click on "Execute Tests".
8. Verify that the correct Test Plan and Build are selected.
9. You should now have 3 test cases.
10. Select the first test case and mark it as passed.
11. Create another build and be sure and copy tester assignments.
12. Click on "Execute Tests".
13. Verify that the correct Test Plan and new Build are selected.
14. You should now have 3 test cases.



As per comments
======================


2. When *using platforms* and filtering in advanced filter mode for multiple statuses combined, e.g. "not run or blocked on all builds", the results were also incorrect.

Example:
- filter "not run on all builds" finds two test cases
Sujata > Is correct as we have only "two test cases" which are not run in "all builds"

- filter "failed on all builds" returns no test cases
Sujata > Is correct as we have not marked any test case as failed in above steps.

=> The advanced filter combination "not run/failed on all builds" should return two test cases but did not return anything at all.

Another example:
- filter "passed on all builds" finds two test cases

>>> Is wrong it should not return anything as no "same" test cases is passed on
all builds.

- filter "failed on all builds" returns no test cases
>>> Is correct as we have not marked any test case as failed on both the builds.

- filter "blocked on all builds" returns no test cases
>> is correct same as above

=> The advanced filter combination "passed or failed or blocked on all builds" should return two test cases but did not return anything at all.
>> why it will return two test cases ? it should return "none" , the correct behavior.

----------------------------------------------------------

Let me know if the the steps used for giving above comments are different and whether the expected results in above scenario are correct or not.

Also what changes are expected after the resolution of the bug 5226.




TagsNo tags attached.
Database (MySQL,Postgres,etc)MySQL 5.1.30
BrowserFirefox 14
PHP Version5.2.8
TestCaseID
QA Team - Task Workflow StatusTBD
Attached Files

- Relationships
related to 0005226closedasimon 1.9.4 - Execute Tests - Filter by "Not Run" and "All Builds" returning empty 
related to 0005142closedfman [1.9.4 Preview] : Test Execution : Some "Result" on "build" filters are not working 

-  Notes
(0017509)
fman (administrator)
2012-09-20 10:06

@sujataverma3
I appreciate your analisis, and I will try to read it this night.
An important issue with this kind of issue is: LEVEL OF DETAIL used to explain by reporters and developers.

I think you remember that time ago for a similar situation I've shared a Google doc with an step by step details, with a small but clear explained env.
WAIT!!!: I'm not saying you have not done this (I repeat I'm going to read this issue later) what I want to said is than we do not take the real needed time to write a good test case that can be used on regression.
We need to be PARANOIC and write absolutely in detail, or we will be lost on sea of confusion.

regards

Francisco
(0017513)
fman (administrator)
2012-09-20 10:13

Reminder sent to: asimon

Please give a look to user question and create a very detailed explanation ( a google doc will be great)
I'm going to give a look this night
(0017514)
asimon (developer)
2012-09-20 11:29

This was a misunderstanding. In my comment on issue 5226 I was not only talking about the exact test cases or steps that were provided there.

That was only one scenario in which the filters didn't work correctly. After fixing that I used a larger test project to find the remaining problems in different scenarios.

A larger combination of different test cases, builds and results is needed in a test project to make the specific filter problems a bit clearer. As already requested on issue 5226 I will provide a more detailed test case on our repo or as a Google doc or both.

I can't promise that I will find the time to do that today or in the remaining week, but I won't forget it.
(0017518)
fman (administrator)
2012-09-20 21:08
edited on: 2012-09-20 21:11

IMHO to find this kind of errors a test projects with 4 or 6 test cases is enough.
What is important is to document very well what are the action taken.

Google doc present on 0005124 has to be taken as example and/or starting point

(0017519)
fman (administrator)
2012-09-20 21:14

@sujataverma3
Please clarify in this sentence the piece on bold:
>> For filter "All builds" the assumption is "the same test case" has to be
>> "passed/failed/blocked/not run" on both the builds.
(0017524)
sujataverma3 (reporter)
2012-09-21 04:40

With this I meant that the same test case has to be in the same status e.g. "passed" on "all the builds" for it to work.

 
So if Build1->Platform A->TC1-> Passed, Build2->Platform A-> TC1-> Passed , so if we put filter "Passed/All builds" the result displayed is 1.

If in one build it is passed and in other it is fail , this mean the result displayed for this combination will be zero.

Build1->Platform A->TC2-> Passed, Build2->Platform A-> TC2-> Failed , so if we put filter "Passed/All builds" the result displayed will be 0. Also the result for filter "failed/All build" will be 0.

To quote you from bug 5142 comments

"The same test case has been executed on BOTH (=> ALL) BUILDS"

the Google doc location is , its in bug 0005142


https://docs.google.com/document/d/1DeJAeMgdJMzrFf3lWAgKcMbOa2GjtSQcYrG4bRazjL0/edit [^]


@asimon

I will try to update the test cases in demo test repository today covering all these combination. Please have a look and let me know if you are looking for some specific combinations.

1) 1 Build 0 Platform ->All Filters "Result" on "All/any/specific/latest/build chosen" -> test cases only 4

2) 2 builds 0 P -> same

3) 1 build , 1 Platform -> same

4) 2 build , 1 Platform-> same


FYI..As i tested 1 Platform, 2 builds -> all results seems ok to me for 1.9.4 release.
(0017526)
fman (administrator)
2012-09-21 07:42

@sujataverma3
Thanks for your work

regarding
>> FYI..As i tested 1 Platform, 2 builds -> all results seems ok to me for 1.9.4 >> release.
nobody has nothing to say about work done, this is only a proof of HOW DIFFICULT is to test an application with complex filter schemes like TL.
This also has to teach us, that we have to evaluate VERY WELL any request of make filter more and more complex.

regards

Francisco
(0017528)
sujataverma3 (reporter)
2012-09-21 11:00

Ok.

My intention for FYI...was just to tell that this combination , i think is working fine .."no more no less" intended meaning.

I am updating the demo rep, and i agree to test all combinations is "really complex".

At-least we will try to update repo with myself and @asimon tried combinations and found issues.

------

I updated demo site with two test cases TL-164/165 , its in Test Execution-> Execution filters -> Filter by Result Status

@asimon,@fman please review the test case, as and when time permits.

- Issue History
Date Modified Username Field Change
2012-09-20 09:42 sujataverma3 New Issue
2012-09-20 10:06 fman Note Added: 0017509
2012-09-20 10:13 fman Note Added: 0017513
2012-09-20 11:29 asimon Note Added: 0017514
2012-09-20 21:08 fman Note Added: 0017518
2012-09-20 21:09 fman Relationship added related to 0005226
2012-09-20 21:09 fman Relationship added related to 0005142
2012-09-20 21:10 fman Note Edited: 0017518 View Revisions
2012-09-20 21:11 fman Note Edited: 0017518 View Revisions
2012-09-20 21:14 fman Note Added: 0017519
2012-09-21 04:40 sujataverma3 Note Added: 0017524
2012-09-21 07:42 fman Note Added: 0017526
2012-09-21 11:00 sujataverma3 Note Added: 0017528



Copyright © 2000 - 2019 MantisBT Team
Powered by Mantis Bugtracker