Mantis Bugtracker          
testlink.org

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0006828TestLinkReports - Req. Based Reportpublic2014-12-10 11:512014-12-31 14:22
ReporterArteliar 
Assigned Tofman 
PrioritynormalSeveritymajorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version1.9.12 (2014 Q3) 
Fixed in Version1.9.13 (2015 #1) 
Summary0006828: Req Based Report - When more than one platform exists, execution status is always on SAME platform (last created)
DescriptionIn 'Requirements based Report' default for 'Platform' is set '[Any]' and seems it doesn't work well when i have more than one platform for test case. The test result is same as on last platform.

Problem:
We have 3 platforms:
PLAT1
PLAT2
PLAT3
2 tests on platforms:
TEST1 - on PLAT1, PLAT2
TEST2 - ON PLAT1, PLAT2, PLAT3

In 'Requirements based Report' with default 'Platform' in 'Test Results based on Requirements Specification' in column 'Linked test cases' we have result always for last platform in each test case (result on PLAT2 for TEST1, result on PLAT3 for TEST2).
Additional InformationSorry if my english isn't correct but it's not my main language, but i did my best to be understand :P
TagsNo tags attached.
Database (MySQL,Postgres,etc)Postgres
BrowserMozilla Firefox
PHP Version
TestCaseID
QA Team - Task Workflow StatusREADY FOR TESTING
Attached Filesjpg file icon TPLAN-A.jpg [^] (199,333 bytes) 2014-12-11 08:57

- Relationships
related to 0006834closedfman Req. Based Report - add possibility to filter on specific BUILD 
child of 0006609closedfman Availables hot-fixes for 1.9.12 & How To get full fixed package from gitorious 

-  Notes
(0022340)
fman (administrator)
2014-12-10 19:28
edited on: 2014-12-10 19:30

Please provide better details in order to reproduce same scenario.
The more schematic the better, and also some screenshot this time may be can be useful.
Expected details are similar to this

1. install latest code from gitorious (branch 1.9)
2. create Test project: TPROJ-01
3. create Test Suite TS-100
4. create TC-1A,TC-1B,TC-1C
5. Create req spec RSPEC-01
6. Create Req REQ-01,REQ-02
7. create Test Plan TPLAN-A
8. Create build 1.0
9. Create platform ...

and so on

without this level of details unfortunately no action will be possible.

Issue subject is to generic that is useless
(I will change it)
Category could be better (changed)

(0022343)
Arteliar (reporter)
2014-12-11 08:56

I did all of your steps and:

9. Created platforms: PLAT-01, PLAT-02, PLAT-03
10. Added platforms to TPLAN-A
11. Added test cases to TPLAN-A:
  - TC-1A on PLAT-01, PLAT-02, PLAT-03
  - TC-1B on PLAT-01, PLAT-02
  - TC-1C on PLAT-01
12. Added coverage:
  - REQ-01: TC-1A, TC-1B, TC-1C
  - REQ-02: TC-1A, TC-1B
13. Executed tests:
  - TC-1A on PLAT-01: Passed
  - TC-1A on PLAT-02: Failed
  - TC-1A on PLAT-03: Passed
  - TC-1B on PLAT-01: Passed
  - TC-1B on PLAT-02: Failed
  - TC-1C on PLAT-01: Passed

And in 'Requirements based Report' with 'Any' platforms still have a problem. I attached ScrnSht of this problem. Test TC-1A failed on PLAT-02, so on my mind it should be Failed at all. But it have Passed status, because last platform that i added is PLAT-03, there TC-1A was possitive.

Maybe it's my fault, because im new one here, but i don't know what should i fix.
(0022348)
fman (administrator)
2014-12-11 19:38
edited on: 2014-12-11 21:37

>>And in 'Requirements based Report' with 'Any' platforms still have a problem. >>I attached ScrnSht of this problem. Test TC-1A failed on PLAT-02, so on my >>mind it should be Failed at all.
I'm sure we need to explain better how report works, but anyway you are not explaining the logic that you have followed to get your expected result.

Playing a little bit you can understand that report consider LATEST EXECUTION on Any platform.
And because latest exec was OK on platform PLAT-03 you get this result.
This is what seems from your explanation, but image shows that you have not
executed on the order detailed.
Latest execution was not on PLAT-03, but PLAT-02.


JUST CHECKED: This is expected behaviour BUT NOT RESPECTED


>> But it have Passed status, because last platform that i added is PLAT-03,
>> there TC-1A was possitive.
Is not the order of platform adding , but of execution

most important thing
Issue has to be report from the beginning with maximun level of detail in order to allow me to use my limited time to do effective work.

If you need more personalized support, do not hesitate to ask for a payed support quotation

(0022350)
fman (administrator)
2014-12-11 21:22
edited on: 2014-12-11 21:23

I've created a first fix, but IMHO this report needs to be reviewed and better documented.
Why ?

1. There is no option to choose an specific BUILD.
   Test Case exec result is computed considere LATEST EXECUTION IGNORING BUILD.

2. In addition of concept of ANY PLATFORM (that need to be explained anyway.
ANY Means IGNORE PLATFORM on logic), probably the concept of ALL.
What will be meaning of ALL?
That TEST CASE EXEC RESULT need to be computed using LATEST EXEC on EVERY PLATFORM, then combined them.
In this way PASSED will be setted ONLY is PASSED on EACH PLATFORM.

Unfortunately, probably time will not be available before 1.9.13 release,
to create a better version of this report.

(0022351)
fman (administrator)
2014-12-11 21:32

https://gitorious.org/testlink-ga/testlink-code/commits/2f86363c64a115ad2cc07ae030fbbf1c414fe13c [^]
(0022353)
Arteliar (reporter)
2014-12-12 06:45

I don't know how should it work, but on my logic whole test should be failed if on any platform test was failed. Blocked status should be setted, when there is no failed tests and there is at least one Blocked status. And as You said Passed only, when Passed on each platform.
For me it should work like that.
(0022354)
fman (administrator)
2014-12-12 09:15

>> I don't know how should it work, but on my logic whole test should be failed >> if on any platform test was failed. Blocked status should be setted, when
>> there is no failed tests and there is at least one Blocked status. And as You >> said Passed only, when Passed on each platform.
>> For me it should work like that.
As you see we loop a lot before having info.
Then again explain as much as possible
(0022356)
Arteliar (reporter)
2014-12-12 12:35
edited on: 2014-12-12 12:49

Your fix is working just as you said. Status for test in 'Linked test cases' is same as on last executed case, regardless of the platform.

In my opinion it should work a bit different. I will show my train of thought on few cases, where we have one test TC-1A executed on 3 platforms(PLAT-01, PLAT-02, PLAT-03), regardless of the execution order:

I example:
 - PLAT-01: Passed
 - PLAT-02: Failed
 - PLAT-03: Passed

Result in 'Linked test cases' for this test: Failed

II example:
 - PLAT-01: Passed
 - PLAT-02: Blocked
 - PLAT-03: Failed

Result in 'Linked test cases' for this test: Failed

III example:
 - PLAT-01: Passed
 - PLAT-02: Blocked
 - PLAT-03: Passed

Result in 'Linked test cases' for this test: Blocked

IV example:
 - PLAT-01: Passed
 - PLAT-02: Passed
 - PLAT-03: Passed

Result in 'Linked test cases' for this test: Passed


As i said, in my opinion, it shouldn't depend on execution order.
Steps of course are same as above and this 4 cases are just combinations of 13th step - 'Executed tests' and the result is for TC-1A on REQ-01.

Hope i wrote it clear enough.

(0022358)
fman (administrator)
2014-12-13 10:09
edited on: 2014-12-13 10:15

Dear user:
The kind of details you have provided is latest note is the level of detail that is always needed, and this is what expect any time I add the note '... please provide detailed steps to reproduce as indicated on forum ... '.
Thanks for this help

Working on this issue, I've found that when used with platforms, details on report where not clear enough.

I'm documenting how things will work and will be displayed in this google doc:

https://docs.google.com/document/d/1YAA9IuYci-nw9ggnXkUcSXKfUmFqyIEiSa7p_elA3Ok/edit?usp=sharing [^]

Since long time the algorithm used to set req status is detailed (and is configurable to certain extent) on config.inc.php


// Order of test cases status in this array, is used to undestand
// to what status set requirement in the requirements report.
// Standard algorithm, present in getReqCoverage(), is:
//
// if at least one of Test Cases linked to Requirement has status FAILED
// Requirement Coverage Status = FAILED
// else if at least one of Test Cases linked to Requirement has status BLOCKED
// Requirement Coverage Status = BLOCKED
// else if ALL Test Cases linked to Requirement has status NOT RUN
// Requirement Coverage Status = NOT RUN
// else if ALL Test Cases linked to Requirement has status PASSED
// Requirement Coverage Status = PASSED
// else
// Requirement Coverage Status = Partially Passed
//
// This logic is implemented using following config parameter
$tlCfg->req_cfg->coverageStatusAlgorithm['checkOrder']=array('atLeastOne','all');
$tlCfg->req_cfg->coverageStatusAlgorithm['checkType']['atLeastOne']=array('failed','blocked');
$tlCfg->req_cfg->coverageStatusAlgorithm['checkType']['all']=array('passed');


Thanks again for your help.
Remember that building testlink is not an easy task, and that supporting this work buying support, or custom development is very important.


- Issue History
Date Modified Username Field Change
2014-12-10 11:51 Arteliar New Issue
2014-12-10 19:28 fman Note Added: 0022340
2014-12-10 19:28 fman QA Team - Task Workflow Status => TBD
2014-12-10 19:28 fman Priority high => normal
2014-12-10 19:29 fman Note Edited: 0022340 View Revisions
2014-12-10 19:30 fman Category Reports => Reports - Req. Based Report
2014-12-10 19:30 fman Summary Problem with more than one platform for test case => Req Based Report - Problem with more than one platform for test case
2014-12-10 19:30 fman Note Edited: 0022340 View Revisions
2014-12-10 19:30 fman Assigned To => fman
2014-12-10 19:30 fman Status new => feedback
2014-12-11 08:56 Arteliar Note Added: 0022343
2014-12-11 08:56 Arteliar Status feedback => assigned
2014-12-11 08:57 Arteliar File Added: TPLAN-A.jpg
2014-12-11 19:38 fman Note Added: 0022348
2014-12-11 19:43 fman Note Edited: 0022348 View Revisions
2014-12-11 19:44 fman Note Edited: 0022348 View Revisions
2014-12-11 19:44 fman Status assigned => closed
2014-12-11 19:44 fman Resolution open => no change required
2014-12-11 20:29 fman Note Edited: 0022348 View Revisions
2014-12-11 20:30 fman Status closed => assigned
2014-12-11 20:30 fman Resolution no change required => reopened
2014-12-11 21:17 fman Note Edited: 0022348 View Revisions
2014-12-11 21:22 fman Note Added: 0022350
2014-12-11 21:23 fman Note Edited: 0022350 View Revisions
2014-12-11 21:23 fman Note View State: 0022350: public
2014-12-11 21:27 fman Summary Req Based Report - Problem with more than one platform for test case => Req Based Report - When more than one platform exists, execution status is always on SAME platform (last created)
2014-12-11 21:32 fman Note Added: 0022351
2014-12-11 21:32 fman Relationship added child of 0006609
2014-12-11 21:33 fman Status assigned => feedback
2014-12-11 21:36 fman Note Edited: 0022348 View Revisions
2014-12-11 21:36 fman Note Edited: 0022348 View Revisions
2014-12-11 21:37 fman Note Edited: 0022348 View Revisions
2014-12-11 21:37 fman Note Edited: 0022348 View Revisions
2014-12-12 06:45 Arteliar Note Added: 0022353
2014-12-12 06:45 Arteliar Status feedback => assigned
2014-12-12 09:15 fman Note Added: 0022354
2014-12-12 12:35 Arteliar Note Added: 0022356
2014-12-12 12:42 Arteliar Note Edited: 0022356 View Revisions
2014-12-12 12:49 Arteliar Note Edited: 0022356 View Revisions
2014-12-13 10:06 fman Relationship added related to 0006834
2014-12-13 10:09 fman Note Added: 0022358
2014-12-13 10:15 fman Note Edited: 0022358 View Revisions
2014-12-13 11:26 fman QA Team - Task Workflow Status TBD => READY FOR TESTING
2014-12-13 11:26 fman Status assigned => resolved
2014-12-13 11:26 fman Fixed in Version => 1.9.13 (2015 #1)
2014-12-13 11:26 fman Resolution reopened => fixed
2014-12-31 14:22 fman Status resolved => closed



Copyright © 2000 - 2018 MantisBT Team
Powered by Mantis Bugtracker