Mantis Bugtracker          
testlink.org

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0004451TestLinkReportspublic2011-05-02 17:512011-07-02 13:49
Reporterfrl 
Assigned ToJulian 
PrioritynormalSeverityminorReproducibilityalways
StatusclosedResolutionfixed 
PlatformTeslinkOSWindows & Linux-DebianOS Version1.9.1
Product Version1.9.2 (2011 Q2 - bug fixing) 
Fixed in Version1.9.3 (2011 Q3 - bug fixing) 
Summary0004451: Test Result Matrix may indicate invalid test case version when test case is not run for a build
DescriptionTest result matrix report does not take into account the current TC version on builds for which TC is not run, when the same TC has been executed on another build.

If TC was not executed for any build, the version is correctly displayed. (current version assigned to the test plan)

(see attached hardcopy images)
Steps To Reproduce1 - Define a test case (version 1), a test plan with 2 builds (build1 and build 2)
2 - Assign the version 1 of TC to the test plan
3 - Execute the TC for build1 (result passed for example)
4 - Build report Test Result Matrix (check result => passed[V1] for build1 and notrun [v1] for build2)
5 - Create a new version (2) of the TC and assign this version to the Test Plan
6 - Build report Test Result Matrix (check result => passed[V1] for build1 and notrun [v1] for build2) => now for build2, we should have notrun[v2]
TagsNo tags attached.
Database (MySQL,Postgres,etc)mySQL 5.x
Browserany (IE,8 - Firefox 3.6.15)
PHP Version5.2
TestCaseID
QA Team - Task Workflow Status
Attached Filesjpg file icon testresultmatrix.jpg [^] (89,913 bytes) 2011-05-02 17:51

- Relationships
child of 0004337closedfman Availables Fixes for 1.9.2 (Prague) 

-  Notes
(0014791)
fman (administrator)
2011-05-04 21:36

Reminder sent to: Julian

@Julian:
hmm, may be I'm wrong, but I think we have fixed this. Do you remember something ?

thanks
(0014793)
Julian (reporter)
2011-05-05 06:54

I was able to reproduce.

will give a quick look.
(0014794)
Julian (reporter)
2011-05-05 07:18

resultsTC.php:

line 156:
$linkedTCVersion = $tcase['version'];
-> this is the version of the test case from a previous build column where a result was set

Example:
build1: P [v1]
build2: N -> v1 according to build 1
build3: N -> v1 according to build 1
build4: F [v2]

line 265:
// If no execution was found => not run
if( $resultsForBuild === null )
{
[...]
$resultsForBuildText .= sprintf($versionTag,$linkedTCVersion);

How about this case:

build1: passed [v1]
build2: not run [v?]
build3: failed [v2]
build4: not run [v?]

AND test case is now linked to test plan in version 3

Let me know which version you expect to be shown for build 2 and build 4.
(0014807)
Julian (reporter)
2011-05-06 14:03

@frl: your feedback is required...
(0014822)
Julian (reporter)
2011-05-10 13:59

Without user feedback i will not act.
I do not like the behaviour when users only report an issue but do not participate in further investigations.
(0014830)
frl (reporter)
2011-05-12 00:07

Sorry for the delay of my response (I just get your request for feedback)

The given example raises several questions (more than I initially thought),
which could lead to full versioning of test plans (like reqs and test cases)
to save history of test case assignment and associated build with test plan
versions. This is only way to give correct value in all cases.
May be a nice feature for future TL versions, but seems out of scope here.

As testplan_tcversions.tcversion_id is overwritten when assigning another TC version to TP, I did not find a workaround to get and use timestamps and retrieve the correct version for each build.


Main expectation here is to get a "confidence level" on result given by TC execution on a build, in the case of TC version update (new TC version assigned to the TP)

If you execute a TC on a build on which the TC was never executed, TL will set result for the version now linked to TP => This should be the version indicated
=> Therefore NotRun[Vx] means that Vx will be the TC version
which will determine the result for this build (according to current TP definition)


In your last example if I execute now the TC, without any change on TC version assignment on TP :
I would get F,P or B [v3] once execution is saved (even for build2 !!!).

This is quite strange, I agree, but without an history of TC version assignment for each build, I do not find more accurate result.

Moreover if TC has been executed for all builds => we cannot see in the report that a new TC version is now linked to the TP

----------

Another (simple) possibility could be to add a separate column with the current TC version linked to TP, and remove the displayed version for build on which test was not run, giving a resyult like this one (with your example)

TestSuite TestCase Version build1 build2 build3 build4
my_test_suite my_test_case [v3] P[v1] N F[v2] N

=> Seems to be the best solution (avoiding all confusions)
(0014833)
Julian (reporter)
2011-05-12 06:30

Thank you.

As you already mentioned test plan versioning is out of our scope.

I personally prefer the simple solution of just removing version from not run test cases as i think this information is not very valuable.
(0014835)
Julian (reporter)
2011-05-12 06:54

Removed version tag for not run test cases.

Branch 1.9:
http://gitorious.org/testlink-ga/testlink-code/commit/543c58b5574f4dd5bc492a22689ad11c3e0695cb [^]

Master:
http://gitorious.org/testlink-ga/testlink-code/commit/b29f2bc0657b262033f567ded6d9e5c5cc2a8862 [^]
(0015453)
fman (administrator)
2011-07-02 13:49

1.9.3 released

- Issue History
Date Modified Username Field Change
2011-05-02 17:51 frl New Issue
2011-05-02 17:51 frl File Added: testresultmatrix.jpg
2011-05-04 21:36 fman Note Added: 0014791
2011-05-05 06:54 Julian Note Added: 0014793
2011-05-05 07:18 Julian Note Added: 0014794
2011-05-05 07:18 Julian Assigned To => Julian
2011-05-05 07:18 Julian Status new => feedback
2011-05-06 14:03 Julian Note Added: 0014807
2011-05-10 13:59 Julian Note Added: 0014822
2011-05-12 00:07 frl Note Added: 0014830
2011-05-12 00:07 frl Status feedback => assigned
2011-05-12 06:30 Julian Note Added: 0014833
2011-05-12 06:54 Julian Note Added: 0014835
2011-05-12 06:54 Julian Status assigned => resolved
2011-05-12 06:54 Julian Fixed in Version => 1.9.3 (2011 Q3 - bug fixing)
2011-05-12 06:54 Julian Resolution open => fixed
2011-05-12 13:06 Julian Relationship added child of 0004337
2011-07-02 13:49 fman Note Added: 0015453
2011-07-02 13:49 fman Status resolved => closed



Copyright © 2000 - 2018 MantisBT Team
Powered by Mantis Bugtracker