MantisBT - TestLink
View Issue Details
0004451TestLinkReportspublic2011-05-02 17:512011-07-02 13:49
frl 
Julian 
normalminoralways
closedfixed 
TeslinkWindows & Linux-Debian1.9.1
1.9.2 (2011 Q2 - bug fixing) 
1.9.3 (2011 Q3 - bug fixing) 
mySQL 5.x
any (IE,8 - Firefox 3.6.15)
5.2
0004451: Test Result Matrix may indicate invalid test case version when test case is not run for a build
Test result matrix report does not take into account the current TC version on builds for which TC is not run, when the same TC has been executed on another build.

If TC was not executed for any build, the version is correctly displayed. (current version assigned to the test plan)

(see attached hardcopy images)
1 - Define a test case (version 1), a test plan with 2 builds (build1 and build 2)
2 - Assign the version 1 of TC to the test plan
3 - Execute the TC for build1 (result passed for example)
4 - Build report Test Result Matrix (check result => passed[V1] for build1 and notrun [v1] for build2)
5 - Create a new version (2) of the TC and assign this version to the Test Plan
6 - Build report Test Result Matrix (check result => passed[V1] for build1 and notrun [v1] for build2) => now for build2, we should have notrun[v2]
No tags attached.
child of 0004337closed fman Availables Fixes for 1.9.2 (Prague) 
jpg testresultmatrix.jpg (89,913) 2011-05-02 17:51
http://mantis.testlink.org/file_download.php?file_id=2308&type=bug
jpg
Issue History
2011-05-02 17:51frlNew Issue
2011-05-02 17:51frlFile Added: testresultmatrix.jpg
2011-05-04 21:36fmanNote Added: 0014791
2011-05-05 06:54JulianNote Added: 0014793
2011-05-05 07:18JulianNote Added: 0014794
2011-05-05 07:18JulianAssigned To => Julian
2011-05-05 07:18JulianStatusnew => feedback
2011-05-06 14:03JulianNote Added: 0014807
2011-05-10 13:59JulianNote Added: 0014822
2011-05-12 00:07frlNote Added: 0014830
2011-05-12 00:07frlStatusfeedback => assigned
2011-05-12 06:30JulianNote Added: 0014833
2011-05-12 06:54JulianNote Added: 0014835
2011-05-12 06:54JulianStatusassigned => resolved
2011-05-12 06:54JulianFixed in Version => 1.9.3 (2011 Q3 - bug fixing)
2011-05-12 06:54JulianResolutionopen => fixed
2011-05-12 13:06JulianRelationship addedchild of 0004337
2011-07-02 13:49fmanNote Added: 0015453
2011-07-02 13:49fmanStatusresolved => closed

Notes
(0014791)
fman   
2011-05-04 21:36   
Reminder sent to: Julian

@Julian:
hmm, may be I'm wrong, but I think we have fixed this. Do you remember something ?

thanks
(0014793)
Julian   
2011-05-05 06:54   
I was able to reproduce.

will give a quick look.
(0014794)
Julian   
2011-05-05 07:18   
resultsTC.php:

line 156:
$linkedTCVersion = $tcase['version'];
-> this is the version of the test case from a previous build column where a result was set

Example:
build1: P [v1]
build2: N -> v1 according to build 1
build3: N -> v1 according to build 1
build4: F [v2]

line 265:
// If no execution was found => not run
if( $resultsForBuild === null )
{
[...]
$resultsForBuildText .= sprintf($versionTag,$linkedTCVersion);

How about this case:

build1: passed [v1]
build2: not run [v?]
build3: failed [v2]
build4: not run [v?]

AND test case is now linked to test plan in version 3

Let me know which version you expect to be shown for build 2 and build 4.
(0014807)
Julian   
2011-05-06 14:03   
@frl: your feedback is required...
(0014822)
Julian   
2011-05-10 13:59   
Without user feedback i will not act.
I do not like the behaviour when users only report an issue but do not participate in further investigations.
(0014830)
frl   
2011-05-12 00:07   
Sorry for the delay of my response (I just get your request for feedback)

The given example raises several questions (more than I initially thought),
which could lead to full versioning of test plans (like reqs and test cases)
to save history of test case assignment and associated build with test plan
versions. This is only way to give correct value in all cases.
May be a nice feature for future TL versions, but seems out of scope here.

As testplan_tcversions.tcversion_id is overwritten when assigning another TC version to TP, I did not find a workaround to get and use timestamps and retrieve the correct version for each build.


Main expectation here is to get a "confidence level" on result given by TC execution on a build, in the case of TC version update (new TC version assigned to the TP)

If you execute a TC on a build on which the TC was never executed, TL will set result for the version now linked to TP => This should be the version indicated
=> Therefore NotRun[Vx] means that Vx will be the TC version
which will determine the result for this build (according to current TP definition)


In your last example if I execute now the TC, without any change on TC version assignment on TP :
I would get F,P or B [v3] once execution is saved (even for build2 !!!).

This is quite strange, I agree, but without an history of TC version assignment for each build, I do not find more accurate result.

Moreover if TC has been executed for all builds => we cannot see in the report that a new TC version is now linked to the TP

----------

Another (simple) possibility could be to add a separate column with the current TC version linked to TP, and remove the displayed version for build on which test was not run, giving a resyult like this one (with your example)

TestSuite TestCase Version build1 build2 build3 build4
my_test_suite my_test_case [v3] P[v1] N F[v2] N

=> Seems to be the best solution (avoiding all confusions)
(0014833)
Julian   
2011-05-12 06:30   
Thank you.

As you already mentioned test plan versioning is out of our scope.

I personally prefer the simple solution of just removing version from not run test cases as i think this information is not very valuable.
(0014835)
Julian   
2011-05-12 06:54   
Removed version tag for not run test cases.

Branch 1.9:
http://gitorious.org/testlink-ga/testlink-code/commit/543c58b5574f4dd5bc492a22689ad11c3e0695cb [^]

Master:
http://gitorious.org/testlink-ga/testlink-code/commit/b29f2bc0657b262033f567ded6d9e5c5cc2a8862 [^]
(0015453)
fman   
2011-07-02 13:49   
1.9.3 released