MantisBT - TestLink
View Issue Details
0005272TestLinkReportspublic2012-10-06 22:002013-05-26 20:06
denjolras 
fman 
normalminoralways
closedfixed 
1.9.4 (2012 Q3 - bug fixing) 
1.9.5 (2012 Q4 - bug fixing) 
MySQL
READY FOR TESTING
0005272: General Test Plan Metrics / Overall Build Status / Not Run is not for build but global.
In General Test Plan Metrics,
The Overall Build Status show a list of the build.
for each we can see the Assigned, Passed, Failed, Block and Not RUN

The "Not Run" is not for the build, but the global value (ie never run)

Of course, given this bug, the addition of Passed+Failed+Blocked/Not run does not = Assigned.
I use platform, have multiple build.
I have an extra status "w" ( Passed with warning )
Version 1.9.0.4 with all fixes.
you can see that the addition of f+p+w+b+n is not = qty

mysql> /* Class:tlTestPlanMetrics - Method: getExecCountersByBuildExecStatus UNION WITH ALL CLAUSE */ SELECT build_id,status, count(0) AS exec_qty FROM (/* Class:tlTestPlanMetrics - Method: getExecCountersByBuildExecStatus sqlUnionAB - executions */ SELECT UA.build_id, TPTCV.tcversion_id, TPTCV.platform_id, COALESCE(E.status,'n') AS status FROM testplan_tcversions TPTCV JOIN user_assignments UA ON UA.feature_id = TPTCV.id AND UA.build_id IN (757,782,793,802,808) AND UA.type = 1 /* GO FOR Absolute LATEST exec ID ON BUILD and PLATFORM */ JOIN ( SELECT EE.tcversion_id,EE.testplan_id,EE.platform_id,EE.build_id, MAX(EE.id) AS id FROM executions EE WHERE EE.testplan_id=112983 AND EE.build_id IN (757,782,793,802,808) GROUP BY EE.tcversion_id,EE.testplan_id,EE.platform_id,EE.build_id ) AS LEBBP ON LEBBP.testplan_id = TPTCV.testplan_id AND LEBBP.platform_id = TPTCV.platform_id AND LEBBP.tcversion_id = TPTCV.tcversion_id AND LEBBP.testplan_id = 112983 /* Get execution status WRITTEN on DB */ JOIN executions E ON E.id = LEBBP.id AND E.build_id = UA.build_id WHERE TPTCV.testplan_id=112983 AND UA.build_id IN (757,782,793,802,808) UNION ALL /* Class:tlTestPlanMetrics - Method: getExecCountersByBuildExecStatus sqlUnionBB - NOT RUN */ SELECT UA.build_id, TPTCV.tcversion_id, TPTCV.platform_id, COALESCE(E.status,'n') AS status FROM testplan_tcversions TPTCV JOIN user_assignments UA ON UA.feature_id = TPTCV.id AND UA.build_id IN (757,782,793,802,808) AND UA.type = 1 /* Get REALLY NOT RUN => BOTH LE.id AND E.id ON LEFT OUTER see WHERE */ LEFT OUTER JOIN ( SELECT EE.tcversion_id,EE.testplan_id,EE.platform_id,EE.build_id, MAX(EE.id) AS id FROM executions EE WHERE EE.testplan_id=112983 AND EE.build_id IN (757,782,793,802,808) GROUP BY EE.tcversion_id,EE.testplan_id,EE.platform_id,EE.build_id ) AS LEBBP ON LEBBP.testplan_id = TPTCV.testplan_id AND LEBBP.platform_id = TPTCV.platform_id AND LEBBP.tcversion_id = TPTCV.tcversion_id AND LEBBP.testplan_id = 112983 LEFT OUTER JOIN executions E ON E.tcversion_id = TPTCV.tcversion_id AND E.testplan_id = TPTCV.testplan_id AND E.platform_id = TPTCV.platform_id /* FILTER BUILDS in set on target test plan (not alway can be applied) */ WHERE TPTCV.testplan_id=112983 AND UA.build_id IN (757,782,793,802,808) /* Get REALLY NOT RUN => BOTH LE.id AND E.id NULL */ AND E.id IS NULL AND LEBBP.id IS NULL ) AS SQBU GROUP BY build_id,status ;
+----------+--------+----------+
| build_id | status | exec_qty |
+----------+--------+----------+
| 757 | f | 1 |
| 757 | n | 11 |
| 757 | p | 46 |
| 782 | b | 7 |
| 782 | f | 24 |
| 782 | n | 11 |
| 782 | p | 584 |
| 782 | w | 28 |
| 793 | b | 1 |
| 793 | f | 3 |
| 793 | n | 11 |
| 793 | p | 25 |
| 793 | w | 1 |
| 802 | b | 21 |
| 802 | f | 3 |
| 802 | n | 11 |
| 802 | p | 201 |
| 802 | w | 14 |
| 808 | b | 4 |
| 808 | n | 14 |
| 808 | p | 22 |
+----------+--------+----------+
21 rows in set (0.27 sec)

mysql> SELECT COUNT(0) AS qty, UA.build_id FROM user_assignments UA JOIN testplan_tcversions TPTCV ON TPTCV.id = UA.feature_id WHERE UA. build_id IN ( 757,782,793,802,808 ) AND UA.type = 1 GROUP BY build_id ;
+-----+----------+
| qty | build_id |
+-----+----------+
| 850 | 757 |
| 850 | 782 |
| 265 | 793 |
| 256 | 802 |
| 80 | 808 |
+-----+----------+
5 rows in set (0.01 sec)
TO BE FIXED on 2.0
related to 0005353closed fman Not run count on General Test Plan Metrics is wrong 
child of 0005193closed fman Availables hot-fixes for 1.9.4 & How To get full fixed package from gitorious 
xml testproject-deep.xml (3,073) 2012-10-08 05:36
http://mantis.testlink.org/file_download.php?file_id=2894&type=bug
jpg Not run.jpg (89,559) 2012-10-08 05:37
http://mantis.testlink.org/file_download.php?file_id=2895&type=bug
jpg

jpg extra_prints.jpg (144,207) 2012-10-09 05:13
http://mantis.testlink.org/file_download.php?file_id=2898&type=bug
jpg

jpg events.jpg (177,538) 2012-10-09 05:13
http://mantis.testlink.org/file_download.php?file_id=2899&type=bug
jpg
Issue History
2012-10-06 22:00denjolrasNew Issue
2012-10-06 22:17denjolrasNote Added: 0017619
2012-10-07 07:41fmanNote Added: 0017620
2012-10-08 05:34sujataverma3Note Added: 0017621
2012-10-08 05:36sujataverma3File Added: testproject-deep.xml
2012-10-08 05:37sujataverma3File Added: Not run.jpg
2012-10-08 09:02fmanNote Added: 0017622
2012-10-08 19:15fmanNote Added: 0017626
2012-10-08 19:16fmanProduct Version => 1.9.4 (2012 Q3 - bug fixing)
2012-10-08 19:16fmanFixed in Version => 1.9.5 (2012 Q4 - bug fixing)
2012-10-08 19:17fmanNote Added: 0017627
2012-10-08 19:17fmanTag Attached: TO BE FIXED on 2.0
2012-10-08 19:18fmanRelationship addedchild of 0005193
2012-10-08 19:45fmanAssigned To => fman
2012-10-08 19:45fmanStatusnew => feedback
2012-10-08 20:55fmanTask Workflow StatusTBD => READY FOR TESTING
2012-10-09 05:13sujataverma3Note Added: 0017630
2012-10-09 05:13sujataverma3File Added: extra_prints.jpg
2012-10-09 05:13sujataverma3File Added: events.jpg
2012-10-09 18:13fmanNote Added: 0017638
2012-10-10 04:14sujataverma3Note Added: 0017643
2012-10-10 04:40sujataverma3Note Added: 0017644
2012-10-10 05:56fmanNote Added: 0017645
2012-10-10 08:02sujataverma3Note Added: 0017647
2012-10-12 18:45fmanStatusfeedback => resolved
2012-10-12 18:45fmanResolutionopen => fixed
2012-11-19 15:42fmanRelationship addedrelated to 0005353
2013-05-26 20:06fmanStatusresolved => closed

Notes
(0017619)
denjolras   
2012-10-06 22:17   
as a dirty workaround, I have modify my getOverallBuildStatusForRender
to recompute the not run:

just after
$rf = &$renderObj->info[$buildID]['details'];

I've added :
                                // My workaound
                                $metrics['with_tester'][$buildID]['n']['exec_qty']=$metrics['total'][$buildID]['qty'];
                                foreach($code_verbose as $statusCode => $statusVerbose)
                                {
                                        $rf[$statusVerbose] = array('qty' => 0, 'percentage' => 0);
                                        $rf[$statusVerbose]['qty'] = $metrics['with_tester'][$buildID][$statusCode]['exec_qty'];

                                        $metrics['with_tester'][$buildID]['n']['exec_qty'] -= $statusVerbose == 'not_run' ? 0 : $rf[$statusVerbose]['qty'];
                                }
(0017620)
fman   
2012-10-07 07:41   
@denjolras
It's ok if you provide a workaround, but is absolutely more useful for us if you can provided a very detailed steps to reproduce, in this way

1. create test project
2. create test suite A, B, C
3. create following test cases ...

Then is very important that you export as much as possible elements on our XML format, this way we can re-create your environment and retest.

Another detail that is useful whilw reporting, please do not create kilometric notes as you have done with the query, put all this info in a text file or something like that and upload.
This make browsing the issue easier.
(0017621)
sujataverma3   
2012-10-08 05:34   
I tried to reproduce the bug, and was able to reproduce.

Steps to reproduce :

1) Create test project A

2) Create Test plan A

3) Create 2 builds -> 501122, 502233

3) Create 4 test cases in Test suite A

4) Assign all 4 test cases for both Builds.

5) Go to General test plan metrics

 -> Over all build status shows -> 4 not run for both the build -> correct

6) Now execute -> test cases for one build 501122-> 1 Passed, 1 failed, 1 blocked

7) Go to general test plan metrics

 -> overall build status shows
 
    not run = 1 for both the builds -> wrong behavior. (As mentioned in the bug it is taking "never run" , and is not on per build basis)

Please see the attached screenshot for more clarity.

I will add this test case in demo QA repository.
(0017622)
fman   
2012-10-08 09:02   
@sujataverma3
Thanks a lot!!!
(0017626)
fman   
2012-10-08 19:15   
issue is generated by missing clauses regarding UA.build_id
The code was in place but was commented, with some concern.
I've going to commit.
@sujataverma3
I would appreciate if we can do a set of extensive (as much as your time allows) test.
Same for denjolras
    
we need to test with:
NO PLATFORM - Multiple builds => this has been tested.
NO PLATFORM - ONE BUIDL
ONE PLATFORM - Multiple builds
ONE PLATFORM - ONE BUILD
(0017627)
fman   
2012-10-08 19:17   
TestLink 1.9.5
http://gitorious.org/testlink-ga/testlink-code/commit/c5a033e6c35d3b73ef7ba9ce49af9ae23d3e89e0 [^]
(0017630)
sujataverma3   
2012-10-09 05:13   
Issue is not observed for No Platform- 2 builds, but some extra messages are displayed when we click on General test plan metrics. I am attaching the event log and screenshot.
(0017638)
fman   
2012-10-09 18:13   
@sujataverma3
ok will remove debug info
would you mind to test other scenarios ?
same request to reporter
(0017643)
sujataverma3   
2012-10-10 04:14   
No platform - 1 build -> No issue

No Platform -> multiple builds ( 4 builds) -> no issues

1 Platform -> 1 build -> No issue

1 Platform -> multiple builds ( some issue ) -> trying to reproduce , will update
(0017644)
sujataverma3   
2012-10-10 04:40   
Ok, there is another issue and which is independent of platforms, will file bug separately.

For this issue

1 Platform -> 3 builds -> properly "not run" status updated

2 Platform -> 1 build -> no issue

2 platform -> 2 build -> over all build status shows correct
(0017645)
fman   
2012-10-10 05:56   
OK, last check. does this means I can set this issue to resolved ?
(0017647)
sujataverma3   
2012-10-10 08:02   
Yes, for all combination,I did not see the issue and "Not run" status is shown properly on build basis.