Mantis Bugtracker          
testlink.org

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0002692TestLinkTest Executepublic2009-07-01 15:292010-05-01 20:33
Reporterandrigog 
Assigned Tofman 
PrioritynormalSeveritymajorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version1.8.2 
Fixed in Version1.9 Beta 2 
Summary0002692: Not possible to find TCs based on last result (e.g. cannot find failed TCs that need re-run)
DescriptionHere is a scenario for the problem.

1- Create a Test Plan with 5 TCs

2- Create Build 1 for test plan

3- Execute TCs for Build 1, with these results
   . TC1 = Passed
   . TC2 = Passed
   . TC3 = Failed
   . TC4 = Blocked
   . TC5 = Failed

4- Create Build 2 for test plan

5- Execute TCs for Build 2, with these results:
   . TC1 = Not Run
   . TC2 = Not Run
   . TC3 = Not Run
   . TC4 = Failed
   . TC5 = Blocked

5- Create Build 3

6- Step purpose: run TCs which have failed in their last execution. It is not possible! One might try configuring the filter like this:
   . Build = Build 3
   . Result on ALL previous builds = Failed

However, no test cases will be shown since the filter will check for TCs which failed in Build 1 AND Build 2. If you look at the TC results above, no TC fits that case. TC5 was failed in Build 1, but became Blocked when testing against Build 2. And TC4 was Blocked, and became Failed.

The point here is that the filter works correctly, but its use case is not optimal. It is very often the case that testers are given many builds (say, once a week), and there is no time to re-run all test cases for every build. Rather, test execution takes place progressively across several builds. And a very common scenario is to re-run what failed or blocked, or run what was not yet run.

So, my suggestion is that the filter in Test Execution be modified so that testers can easily find test cases in the test plan whose last execution (in any build) was of a given status. That will be much easier for them to know what they need to re-run, or run for the first time, in order to finaly reach 100% test coverage after many builds.

Additional InformationRelated issue: 2455
TagsNo tags attached.
Database (MySQL,Postgres,etc)
Browser
PHP Version
TestCaseID
QA Team - Task Workflow Status
Attached Files

- Relationships
related to 0002455closedasimon Improvement of execution and test case execution assignment filters 

-  Notes
(0007420)
andrigog (reporter)
2009-07-01 15:33

By the way, is there any workaround which could make it possible to execute the use case described in step 6 in 1.8.2 or 1.8.3?
(0007422)
tomt (reporter)
2009-07-01 22:39
edited on: 2009-07-01 22:40

I have also seen this. "Last Result" is a very important feature for us that seems to have been taken out. From my entry in the 1.8 Forum.

I just migrated to from 1.7.5 to 1.8.3.

I can't seem to review the "last result" of tests excuted in one of several builds. That is, I would like to see the result of last execution of test cases that may have been run in one of several prior builds.

Any help is appreciated!

Tom

I have the following settings in custom_config.inc.php:

// TRUE -> the whole execution history for the choosen build will be showed
// FALSE -> just last execution for the choosen build will be showed [STANDARD BEHAVIOUR]
$tlCfg->exec_cfg->history_on = TRUE;

//$tlCfg->exec_cfg->show_testsuite_contents = ENABLED;

// TRUE -> test case VERY LAST (i.e. in any build) execution status will be displayed
// FALSE -> only last result on current build. [STANDARD BEHAVIOUR]
$tlCfg->exec_cfg->show_last_exec_any_build = TRUE;

// TRUE -> History for all builds will be shown
// FALSE -> Only history of the current build will be shown [STANDARD BEHAVIOUR]
$tlCfg->exec_cfg->show_history_all_builds = TRUE;

(0007560)
tgrimm (reporter)
2009-07-15 03:22
edited on: 2009-07-15 06:01

I know this is not a forum (and this may look like a message) but the intention is that I would like to escalate this ticket.

I too have custom_config.inc.php configured as Tom describes and there doesn't appear to be any way currently to perform filtering based on last execution result status.

IMPACT: This makes it extremely difficult to review what testing is outstanding while in execution mode. Selecting each test case individually to review the last status result is not a scalable solution.

We are performing an evaluation of testlink against some real test work flows and this deficiency is hurting badly - enough to prevent deployment.

Is there perhaps an undocumented flag needed now?

[Edit]
Is there any information on how this hangs together? I would be more than willing to take a look at addressing this.

(0007562)
amitkhullar (reporter)
2009-07-15 12:36

@andrigog: When u have a TC failed in build 1 and it has not been run against build2 then y do u change its status to blocked, dont run it/update the status in TL, and you will get its status as Failed when u run the report Failed Test Cases then you can know which test cases are there which failed and which have to be executed in the forthcoming releases.

Also you have reports like Failed / Blocked and Not Run Test cases to identify such test cases.
(0007566)
eitrance (reporter)
2009-07-15 17:33

amitkhullar:
your mentioned method is hardly as convenient as the previous functionality.
going back and forth from "execute" to "reports" is getting harder and harder
as your TP getting bigger with many TC in it.

from usability perspective there is a definite decreased this way.
(0007572)
tgrimm (reporter)
2009-07-16 02:43
edited on: 2009-07-16 02:47

I believe this problem is sourced by the overloading of the "filters and settings" build field.

I think the filtering for displaying the tree should not be used to specify what build we are executing against - they should be separate.

For example: The filtering of the tree should be allowed to be based on one of these selections:
   1. Last Execution any build
         Shows overall result of the test regimen since beginning
   2. Last Execution current active build (default - what happens today)
         Shows what the results are, and progress on the current build
   3. Last Execution on a specific build
         Shows what the results were historically on a specific build
   4. Last Execution on a set of builds (multi-select)
         Shows what has been done over a set of builds - e.g. say you want to ensure specific coverage over 5 last builds.

The default selection option needs to be configurable (saved in a cookie) so it doesn't need to be reset every time you raise the execution page.

BUT the build used for committing an execution result update needs to be selected separately. It should default to the last build created and also allow the selection of a specific build (as it is today).

I agree with eitrance: moving between reports and execution is not optimal. Operations that are performed frequently (and nothing in testlink is more frequently used than execution) should not need to be on separate pages, tabs or browsers.

NOTE: I believe this can be achieved by providing two build selection lists:
  1. Filter (multi-select): Looks the same as the build selector today but has an "Any" entry (provides the "Last Execution on any build") and is multi-selectable. This only controls what is displayed in the tree.

  2. Execution Build (single-select): Looks identical to the build selector today. This controls what build the execution result is committed against.

(0007579)
fman (administrator)
2009-07-16 17:41

>> BUT the build used for committing an execution result update needs to be
>> selected separately.
Please explain how ?
 
>>It should default to the last build created and also allow the selection of a >> specific build (as it is today).
Today build filter works this way, because combo is populated with build with MAX(internal build id).
(0007581)
fman (administrator)
2009-07-16 17:43

First simple implementation started.
(0007588)
tgrimm (reporter)
2009-07-16 23:33

Thanks fman.

>> Please explain how ?

I'm not sure I understand your question. I'm not familiar with the underlying architecture so I don't know how easy/difficult it would be to separate tree filtering from commit selection of builds. I suggested using two build selection lists - one for filtering the tree, the other for the build commit selection.

I think I might have missed the essence of your question.
(0007599)
fman (administrator)
2009-07-17 15:41

@tgrimm:
I've made a mistake, due to bad reading, please forget question.
here some points about your request:

>> I believe this problem is sourced by the overloading of the "filters and settings" build field.
>>
>> I think the filtering for displaying the tree should not be used to specify
>> what build we are executing against - they should be separate.
>>
Let me explain, how i think works has to work:
Tester choose on what build he/she wants to work and tree will show figures about executions
on SELECTED BUILD FOR EXECUTION, with test cases colored based on last execution status on THIS build.

Then user may be wants to do more deep filter (reduce tree size) based on last execution status on OTHER BUILDS,
but after filtering, tree will continue to show status on SELECTED BUILD FOR EXECUTION.
If we do not work on this way we will find a little bit of problems with :
- counters on tree.
- test case colouring.

How test cases will be colored when you choose:
1. Last Execution any build ?

Displaying corresponing BUILD on tree for each test cases will mess (IMHO) tree,
and best thing will be IMHO having access to Test Case Matrix results from execution page
on a new window, if user wants to have this info.

>> For example: The filtering of the tree should be allowed to be based on one of these selections:
>> 1. Last Execution any build
>> Shows overall result of the test regimen since beginning
OK, first implementation done, but using ONLY PREVIOUS BUILDS.
Any must include BUILD on what execution will be done ?

>> 2. Last Execution current active build (default - what happens today)
>> Shows what the results are, and progress on the current build
Filter that already exists and IMHO must be filter that will used in combination with other
build filters (filters 1,3,4)

>> 3. Last Execution on a specific build
>> Shows what the results were historically on a specific build
OK. can be done, but is present today because only thing user has to do is set a differentt build.

>> 4. Last Execution on a set of builds (multi-select)
>> Shows what has been done over a set of builds - e.g. say you want to ensure specific coverage over 5 last builds.
>>
This is implemented today but using a fixed set, PREVIOUS BUILDS.

>> The default selection option needs to be configurable (saved in a cookie) so it doesn't need to be reset every time you raise the execution page.
>>
>> BUT the build used for committing an execution result update needs to be selected separately.
>> It should default to the last build created and also allow the selection of a specific build (as it is today).
>>
>> I agree with eitrance: moving between reports and execution is not optimal.
>> Operations that are performed frequently (and nothing in testlink is more frequently used than execution)
>> should not need to be on separate pages, tabs or browsers.
>>
>> NOTE: I believe this can be achieved by providing two build selection lists:
>> 1. Filter (multi-select): Looks the same as the build selector today but has an "Any" entry (provides the "Last Execution on any build")
>> and is multi-selectable. This only controls what is displayed in the tree.
>>
>> 2. Execution Build (single-select): Looks identical to the build selector today.
>> This controls what build the execution result is committed against.


------------------------------
I will post a patch in a few days
(0007600)
fman (administrator)
2009-07-17 15:43

Reminder sent to: mhavlat

would you mind to give a look, to discuss re-implementation of settings and filters on execution ?
(0007601)
mhavlat (reporter)
2009-07-17 20:55

We are in agreement.
Filter 1. and 2. are really handy and should be supported.
Filter 3. and 4. should not be available. 3. means just to change the current build. 4. is really specific - user can use Query Metrics

Tree colouring is always related to the current build. Tree colouring cannot be dependent to filtering. That is correct.
There was specific option before TL 1.7 version, that define colours meaning (result on the current build || overall last result). It could be reimplemented as extra option. (not in 1.8)
(0007604)
tgrimm (reporter)
2009-07-18 00:25
edited on: 2009-07-18 00:39

Thanks fman and mhavlat.

Sounds good for now. The changes you are making will definitely help with sorting out our testing issues.

It's complicated to know what filters are most frequently used - everyone seems to have different work flows for testing.

Perhaps we should create a set of execution use scenarios and prioritize them for frequency of use. i.e. Get some sort of voting input on who uses/requires what.

Any changes to help enhance these filters (e.g. colours, counts, etc.) should be secondary but would definitely help improve usability.

(0007975)
qabe (reporter)
2009-09-16 23:56
edited on: 2010-02-15 02:37

Agreed, thanks for the improvement.

I am QA Manager and this is becoming a test management problem for our team usage.

I support the need for the requirements mentionned by tgrimm in it's note (0007572), all 4 filter behaviors are also necessary for our usage, and having to go review the result matrix seems to me like a step we could avoid. I have received complains from testers that this was not user friendly, or practical.



I also think 2 builds combos would help, (1) one to set the current execution build, (2) one to set a filter to view only desired test cases as per their previous statuses.

Filter (1) for build, like today, to set where the execution is done.
Filter (2) for what test cases to display on the tree.

For filter (2), I also support the need for the proposed 4 filters, but some were partially implemented, please consider the following filter conditions which are usefull.

1. Last Execution any build
IMPORTANT: previous builds to be filters with an OR condition, not an AND condition.

4. Last Execution on a set of builds (multi-select)
Sometimes we don't want ALL PREVIOUS BUILD, maybe just offering this combo in advanced mode would permit to do a multi-select?

Is there a patch for the partial implementation?
Is there a plan for the final implementation?

Thanks for your time,
Best regards,
QAbe

(0008997)
asimon (developer)
2010-02-12 17:22

Added relation to 2455. The code I attached to issue 2455 should solve this issue too.

- Issue History
Date Modified Username Field Change
2009-07-01 15:29 andrigog New Issue
2009-07-01 15:33 andrigog Note Added: 0007420
2009-07-01 22:39 tomt Note Added: 0007422
2009-07-01 22:40 tomt Note Edited: 0007422
2009-07-15 03:22 tgrimm Note Added: 0007560
2009-07-15 06:01 tgrimm Note Edited: 0007560
2009-07-15 12:36 amitkhullar Note Added: 0007562
2009-07-15 17:33 eitrance Note Added: 0007566
2009-07-16 02:43 tgrimm Note Added: 0007572
2009-07-16 02:47 tgrimm Note Edited: 0007572
2009-07-16 17:41 fman Note Added: 0007579
2009-07-16 17:43 fman Note Added: 0007581
2009-07-16 17:43 fman Status new => work in progress
2009-07-16 17:44 fman Status work in progress => assigned
2009-07-16 17:44 fman Assigned To => fman
2009-07-16 17:44 fman Status assigned => work in progress
2009-07-16 23:33 tgrimm Note Added: 0007588
2009-07-17 15:41 fman Note Added: 0007599
2009-07-17 15:43 fman Note Added: 0007600
2009-07-17 20:55 mhavlat Note Added: 0007601
2009-07-18 00:25 tgrimm Note Added: 0007604
2009-07-18 00:39 tgrimm Note Edited: 0007604
2009-09-16 23:56 qabe Note Added: 0007975
2010-02-12 17:20 asimon Relationship added related to 0002455
2010-02-12 17:22 asimon Note Added: 0008997
2010-02-15 02:37 fman Note Edited: 0007975
2010-02-15 02:38 fman Status work in progress => assigned
2010-02-15 02:38 fman Status assigned => resolved
2010-02-15 02:38 fman Fixed in Version => 1.9 Beta 2
2010-02-15 02:38 fman Resolution open => fixed
2010-05-01 20:33 fman Status resolved => closed



Copyright © 2000 - 2020 MantisBT Team
Powered by Mantis Bugtracker