MantisBT - TestLink
View Issue Details
0005003TestLinkReportspublic2012-05-04 17:142012-09-01 19:59
DanielB 
fman 
normalminoralways
closedfixed 
CentOS5.8
1.9.3 (2011 Q3 - bug fixing) 
1.9.4 (2012 Q3 - bug fixing) 
mysql
firefox
2.3
TBD
0005003: General Test Plan Metrics pass/fail/blocked numbers include unassigned test cases
Sorry if this is a duplicate. I've searched the issue list for a while and have read related issues 0004695 and 0004734, but this particular issue doesn't appear to be mentioned anywhere that I can see.

In the General Test Plan Metrics page, under overall build status, the number of passed/failed and blocked test cases includes unassigned test cases. This can lead to greater than 100% of tests being complete, since the total number of test cases just includes assigned ones.
1) Create a new test project, test plan, test suite and build
2) Make two test cases
3) assign one of them to a user
4) pass both of them
5) view general test plan metrics

It will show that there is one assigned test case, but that 2 have passed, and so 200% have passed. The completed column at the end shows 100% so I assume it only includes assigned test cases, but the other columns include unassigned ones. Regardless of whether you only measure assigned or not, I think the metrics should be consistent in order to have meaningful numbers.
No tags attached.
duplicate of 0004989closed fman Reports - Overall Build Status - refactoring and final business logic. 
png Screenshot at 2012-05-04 13:00:14.png (48,957) 2012-05-04 17:14
http://mantis.testlink.org/file_download.php?file_id=2670&type=bug
png
Issue History
2012-05-04 17:14DanielBNew Issue
2012-05-04 17:14DanielBFile Added: Screenshot at 2012-05-04 13:00:14.png
2012-05-04 18:50fmanNote Added: 0016639
2012-05-04 18:55DanielBNote Added: 0016640
2012-05-04 20:01fmanRelationship addedduplicate of 0004989
2012-07-28 15:08fmanTask Workflow Status => TBD
2012-07-28 15:08fmanStatusnew => resolved
2012-07-28 15:08fmanFixed in Version => 1.9.4 (2012 Q3 - bug fixing)
2012-07-28 15:08fmanResolutionopen => fixed
2012-07-28 15:08fmanAssigned To => fman
2012-09-01 19:59fmanNote Added: 0017317
2012-09-01 19:59fmanStatusresolved => closed

Notes
(0016639)
fman   
2012-05-04 18:50   
>> Regardless of whether you only measure assigned or not, I think the metrics
>> should be consistent in order to have meaningful numbers.

unfortunatelly this sentence is to generic, that do not help.
Please make the effort of explaining clearly.
When reporting an issue or feature request, use same level of detail you will use on your regular work.


We are planning to do continue on 1.9.4 with same logic:

On General test plan metrics ONLY PIECE of data that will consider TESTER ASSIGNMENT will be Overall Builds Section.

All other section that summarize data, will consider Testcase WITH and WITHOUT Tester assignment.

This kind of summarize vision (by platform, but top level suites, etc), seems to provide figures that create LESS confusion, when calculated as we have done till today.

If possible we will try to add an option on GUI, to allow user following choice:

1. CONSIDER on ALL METRICS only Test cases that HAS TESTER ASSIGNMENT
(DEFAULT NO)
(0016640)
DanielB   
2012-05-04 18:55   
The issue is that the metric is not only considering cases with tester assignment. In the steps to reproduce that I listed, one test that is assigned will pass and one test that is unassigned will pass, however the general test plan metrics will show two passes even though only one assigned test has passed. It is correctly showing the total number as only including assigned, meaning that 2/1 test cases have passed, which is nonsensical and useless.

I'm not talking about consistency between sections, I'm talking about consistency within the overall builds section itself.
(0017317)
fman   
2012-09-01 19:59   
1.9.4 released