Mantis Bugtracker          
testlink.org

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0004023TestLinkTest Executepublic2010-11-16 17:292011-01-22 15:11
Reporterpaulewog 
Assigned Toasimon 
PriorityhighSeveritymajorReproducibilityalways
StatusclosedResolutionfixed 
Platformx86OSSLESOS Version11.1
Product Version1.9 (Prague) 
Fixed in Version1.9.1 (bug fixing) 
Summary0004023: Incorrect filtering of tests in exec tree
DescriptionWhen filtering tests in the execution tree menu, we are seeing some very strange results. For example: when selecting "NOT RUN" on "ALL BUILDS" (to see what tests have not been run yet), it ends up displaying 2 testcases when it should display 30.

Even stranger, if I choose "Passed" testcases on "ANY BUILD," it shows 26 not-run and 2 passed. If I choose "Failed" on "ANY BUILD" (there aren't any failed executions) it shows 2 NOT-RUN.

I've been looking around in some of the php classes, but I haven't seen anything that I could add some debugging messages in to see how it is selecting the testcases.

We saw this just yesterday while running the RC version, and also today while running the GA version. I don't recall seeing it in the beta. We did do an unsupported upgrade from 1.9 beta (I believe it was beta 2 or so) to 1.9 RC and then to GA... were there table changes between beta/RC/GA that may be causing the problem?
TagsSQL
Database (MySQL,Postgres,etc)MySQL
Browser
PHP Version
TestCaseID
QA Team - Task Workflow Status
Attached Filespng file icon test_exec_tree.png [^] (41,534 bytes) 2010-11-16 17:59


png file icon aix5.3-64b_executions.png [^] (123,707 bytes) 2010-11-16 18:00


? file icon treeMenu.inc.php [^] (85,456 bytes) 2010-12-14 19:23
? file icon testplan.class.php [^] (156,352 bytes) 2010-12-14 19:23

- Relationships

-  Notes
(0012591)
paulewog (reporter)
2010-11-16 17:31

One additional note - the test reports appear to find all the correct executions. For example, the Metrics Dashboard displays correctly.
(0012592)
Julian (reporter)
2010-11-16 17:54
edited on: 2010-11-16 17:55

please explain how you expect those filters are working. attach screenshot of yout test result matrix.

were there table changes between beta/RC/GA that may be causing the problem?
- you know the answer right ?

(0012593)
paulewog (reporter)
2010-11-16 17:58

Table changes - I just checked a fresh install of 1.9 GA and compared the table to mine. There was one difference - build_id for user_assignments defaults to 0 instead of defaulting to NULL. I changed this and updated all user_assignment build_id values of NULL to 0.

I'll use just one aspect of the filter: NOT RUN on ALL BUILDS. I expect this to show me a tree containing all the testcases that have not been run on any of the builds.

I will upload two screenshots; one of the number of testcases run against AIX 5.3 64-bit and one of the displayed tree menu for NOT RUN on ALL BUILDS.
(0012594)
paulewog (reporter)
2010-11-16 18:20

Another thing I have noticed looking at my database... since we have many platforms, many of the testcases are being run against multiple platforms. I checked out one testcase that *should* show up as NOT RUN in the execution tree but is not. That particular testcase has been run against another platform already. The ones that DO show up as NOT RUN - only two of them - have *not* been run on any other platform.

Perhaps certain filtering options do not take executed-on-a-different-platform into account, or something like that? ...
(0012596)
paulewog (reporter)
2010-11-16 18:40

I found the spot in the testplan class that actually runs the SQL and ran it in phpmyadmin against the database. It accurately grabs all 31 test for the AIX 5.3 64-bit platform, so it looks like it is a filtering issue, not a SQL issue...
(0012597)
paulewog (reporter)
2010-11-16 18:46
edited on: 2010-11-16 18:48

It gets the correct linked versions. By the time it gets to the below code in treeMenu.inc.php code, $keys contains only two testcases, when it should contain 29... so the error (I suppose it still could be an environment issue on my part) seems to be between line 971 and 1074 in treeMenu.inc.php (or called methods...):

$keys = array_keys($tplan_tcases);

edit: actually, it happens between lines 989 and 1010.

(0012599)
paulewog (reporter)
2010-11-16 19:17

... more looking through. It looks like filter_by_same_status_for_all_builds(...) is returning some funniness.


if( !is_null($buildSet) ) {
   $tcase_build_set = $tplan_mgr->get_same_status_for_build_set($tplan_id, array_keys($buildSet),$filters->{$status});

That particular function call ends up returning a $tcase_build_set that only overlaps with the provided $tcase_set with two testcases. Those are the two testcases that end up getting shown in the execution pane.

Looking at the get_same_status_for_build_set passed arguments ... it doesn't look like it is passing any information about platform? Is it perhaps only checking for ALL testcases, regardless of platform, with the given status? It returns a variety of testcases, but as I said, only two of them actually overlap with the testcase set to be run against the given AIX 5.3 platform...
(0012600)
paulewog (reporter)
2010-11-16 19:48

So, it looks like the SQL is selecting distinct testcase version ID's, matching them to executions...

But the executions are not filtered by platform. This means if I ran a testcase against AIX 6.1, that execution is joined (and thus has been "run").

If I run the SQL against my database, I end up getting the results that show up in the test execution pane. The SQL, at the moment, looks like this:


SELECT distinct T.tcversion_id,E.build_id,NH.parent_id AS tcase_id FROM IT_testplan_tcversions T JOIN IT_nodes_hierarchy NH ON T.tcversion_id=NH.id AND NH.node_type_id=4 LEFT OUTER JOIN IT_executions E ON T.tcversion_id = E.tcversion_id AND T.testplan_id=E.testplan_id AND E.build_id IN (32,33,38,35,36,37,39) WHERE T.testplan_id=2356 AND E.build_id IS NULL

If I change it to look like this (platform "31" is AIX 5.3 64-bit), it works:
SELECT DISTINCT T.tcversion_id,T.platform_id,E.build_id,NH.parent_id AS tcase_id FROM IT_testplan_tcversions T JOIN IT_nodes_hierarchy NH ON T.tcversion_id=NH.id AND NH.node_type_id=4 LEFT OUTER JOIN IT_executions E ON T.tcversion_id = E.tcversion_id AND T.testplan_id=E.testplan_id AND E.platform_id=31 AND E.build_id IN (32,33,38,35,36,37,39) WHERE T.platform_id=31 AND T.testplan_id=2356 AND E.build_id IS NULL

It also works to take out T.platform_id=31 in the WHERE clause, but it helped limit the amount of data returned while I was testing the query.
(0012604)
paulewog (reporter)
2010-11-16 20:17
edited on: 2010-11-16 21:40

I think I have hacked together a fix. It's not pretty, since it just sticks an extra optional argument in the testplan manager methods, but it looks like it has at least resolved the bug for us locally. Here's what I did, using get_same_status_for_build_set method as an example:

* Added $platformid=NULL to function arguments
- function get_same_status_for_build_set($id,$buildSet,$status)
+ function get_same_status_for_build_set($id,$buildSet,$status,$platformid=NULL)

* Added the following lines:
$status_in = implode("',", (array)$status);
+ $tcversionPlatformString = ""; $executionPlatformString = "";
+ if($platformid) { $tcversionPlatformString = "AND T.platform_id=$platformid"; $executionPlatformString = "AND E.platform_id=$platformid"; }

* Changed the SQL string concatenation to include the above:
- " LEFT OUTER JOIN {$this->tables['executions']} E ON T.tcversion_id = E.tcversion_id " .
+ " LEFT OUTER JOIN {$this->tables['executions']} E ON T.tcversion_id = E.tcversion_id $executionPlatformString " .

- " WHERE T.testplan_id={$id} AND E.build_id IS NULL";
+ " WHERE T.testplan_id={$id} AND E.build_id IS NULL $tcversionPlatformString";

** Cannot verify correct results, as we do not have any matches for something other than NOT RUN on ALL Builds **
- " WHERE E.build_id IN ({$build_in}) " .
+ " WHERE E.build_id IN ({$build_in}) $executionPlatformString " .

* Changed the call in treeMenu.inc.php:
- $tcase_build_set = $tplan_mgr->get_same_status_for_build_set($tplan_id,array_keys($buildSet),$filters->{$status});
+ $tcase_build_set = $tplan_mgr->get_same_status_for_build_set($tplan_id,array_keys($buildSet),$filters->{$status},$filters->setting_platform);

I did something similar to this function, as well:
function get_status_for_any_build($id,$buildSet,$status,$platformid=NULL)

Which appears to have fixed other similar issues (selecting "Passed on ANY Build" did not work correctly).

(0012971)
bizob28 (reporter)
2010-12-14 18:06

Just experienced this today. How does your "hack" affect performance of using the UI?
(0012972)
paulewog (reporter)
2010-12-14 18:19

Hello - I did not notice any performance differences.
(0012973)
bizob28 (reporter)
2010-12-14 18:22

can you attach your changed files? I tried manually doing this and it didn't work
(0012974)
paulewog (reporter)
2010-12-14 19:18

Sure, I can attach the files, one moment...
(0012975)
paulewog (reporter)
2010-12-14 19:22

I believe it is just two files, both in [testlink home]/lib/functions/: treeMenu.inc.php and testplan.class.php. I realize now I didn't ever put where the changes I made were, oops. I'll attach both files.
(0012976)
bizob28 (reporter)
2010-12-14 19:26

Yea i figured out where the changes were but I think I fat fingered something, or your solution doesn't fix my issue. I'll give these files a shot, thanks
(0012977)
bizob28 (reporter)
2010-12-14 19:29

Yep, must have fat fingered something. Your fix appears to be working but i'll provide feedback as we use it. Thank you, I was getting heat from my testers
(0012978)
paulewog (reporter)
2010-12-14 19:34

Sure, no problem. I was getting a bit frustrated myself, since it majorly impacted how useful the Platforms feature was ... and that was the main reason we upgraded from 1.8.
(0012979)
fman (administrator)
2010-12-14 20:25

>> I was getting a bit frustrated myself, since ...
Explain what does this means ?
Frustrated with a tool like TestLink ?
(0012982)
paulewog (reporter)
2010-12-14 20:32

No, not frustrated with TL - frustrated with the bug. :)

TestLink works very well for us; we were really looking forward to the Platforms feature. Unfortunately, this particular bug made the platforms feature really difficult to use when executing tests. It was frustrating because the feature would be very helpful, but the bug was preventing us from really using it.
(0012984)
amitkhullar (reporter)
2010-12-15 04:13

m not sure whats the issue
but it works fine for me on the current cvs , I have though fixed 2 issues related to custom field filters in the tree, in case you want to apply the fix 3995/4024
(0012991)
fman (administrator)
2010-12-15 09:52

Reminder sent to: Julian

do you think Andreas can check if fix is OK ?
(0013000)
asimon (developer)
2010-12-15 15:15

@paulewog: Thanks for your help and the patches you sent.
I just committed the fixes for branch 1.9 and head, so your fixes will be in the next TestLink release.
(0013380)
fman (administrator)
2011-01-22 15:11

1.9.1 Released

- Issue History
Date Modified Username Field Change
2010-11-16 17:29 paulewog New Issue
2010-11-16 17:31 paulewog Note Added: 0012591
2010-11-16 17:54 Julian Note Added: 0012592
2010-11-16 17:55 Julian Note Edited: 0012592 View Revisions
2010-11-16 17:58 paulewog Note Added: 0012593
2010-11-16 17:59 paulewog File Added: test_exec_tree.png
2010-11-16 18:00 paulewog File Added: aix5.3-64b_executions.png
2010-11-16 18:20 paulewog Note Added: 0012594
2010-11-16 18:40 paulewog Note Added: 0012596
2010-11-16 18:46 paulewog Note Added: 0012597
2010-11-16 18:47 paulewog Note Edited: 0012597 View Revisions
2010-11-16 18:48 paulewog Note Edited: 0012597 View Revisions
2010-11-16 19:02 paulewog Note Added: 0012598
2010-11-16 19:05 paulewog Note Deleted: 0012598
2010-11-16 19:17 paulewog Note Added: 0012599
2010-11-16 19:48 paulewog Note Added: 0012600
2010-11-16 20:17 paulewog Note Added: 0012604
2010-11-16 20:18 paulewog Tag Attached: SQL
2010-11-16 21:40 paulewog Note Edited: 0012604 View Revisions
2010-12-14 18:06 bizob28 Note Added: 0012971
2010-12-14 18:19 paulewog Note Added: 0012972
2010-12-14 18:22 bizob28 Note Added: 0012973
2010-12-14 19:18 paulewog Note Added: 0012974
2010-12-14 19:22 paulewog Note Added: 0012975
2010-12-14 19:23 paulewog File Added: treeMenu.inc.php
2010-12-14 19:23 paulewog File Added: testplan.class.php
2010-12-14 19:26 bizob28 Note Added: 0012976
2010-12-14 19:29 bizob28 Note Added: 0012977
2010-12-14 19:34 paulewog Note Added: 0012978
2010-12-14 20:25 fman Note Added: 0012979
2010-12-14 20:32 paulewog Note Added: 0012982
2010-12-15 04:13 amitkhullar Note Added: 0012984
2010-12-15 09:52 fman Note Added: 0012991
2010-12-15 15:15 asimon Note Added: 0013000
2010-12-15 15:15 asimon Status new => resolved
2010-12-15 15:15 asimon Fixed in Version => 1.9.1 (bug fixing)
2010-12-15 15:15 asimon Resolution open => fixed
2010-12-15 15:15 asimon Assigned To => asimon
2011-01-22 15:11 fman Note Added: 0013380
2011-01-22 15:11 fman Status resolved => closed



Copyright © 2000 - 2020 MantisBT Team
Powered by Mantis Bugtracker