MantisBT - TestLink
View Issue Details
0001310TestLinkTest Specificationpublic2008-01-22 15:282014-07-28 20:33
francisl 
fman 
highfeature requestN/A
closedfixed 
 
1.9.11 (2014 Q2 - bug fixing) 
READY FOR TESTING
0001310: [multiple steps] - Multiple test steps and expected results
1. Usually, test case consists of multiple test steps. Each test step associated with an expected results. It would be useful if TL could support this characteristics of test case.

2. If point 1 above is implemented, it would be useful if a test case can refer to another test case which acts as a container of some common test steps. This is similar to the concept of php include files or subprograms.
Attachment
related to 0000507closed fman [multiple steps] - Test Specification: Separete steps and results 
related to 0000422closed fman [multiple steps] - possibility to split execution steps 
parent of 0000718closed fman New way to add expected results 
parent of 0004318closed fman Requirements mapping to Test Case step 
has duplicate 0002555closed mhavlat Test Steps and the expected Results Do not match up 
has duplicate 0004986closed fman Report ( RUN : Failed , OK, ) directly in the Teststeps and not only for the whole Testcase 
child of 0006313closed fman Availables hot-fixes for 1.9.10 & How To get full fixed package from gitorious 
? Steps_Results (116,551) 2009-04-23 22:44
http://mantis.testlink.org/file_download.php?file_id=999&type=bug
Issue History
2008-01-22 15:28francislNew Issue
2008-01-22 15:58fmanNote Added: 0002996
2008-01-23 14:22francislNote Added: 0003021
2008-01-23 14:22francislNote Edited: 0003021
2008-01-23 15:26fmanNote Added: 0003024
2008-01-23 15:29fmanRelationship addedrelated to 0000507
2008-01-23 15:29fmanRelationship addedrelated to 0000422
2008-01-23 15:54francislNote Added: 0003027
2008-01-23 15:59francislNote Edited: 0003027
2008-01-23 15:59francislNote Edited: 0003027
2008-01-23 16:29fmanNote Added: 0003028
2008-01-23 16:38fmanStatusnew => assigned
2008-01-23 16:38fmanAssigned To => fman
2008-01-23 16:38fmanNote Added: 0003029
2008-01-29 18:20olivier_houdasNote Added: 0003120
2008-01-29 18:36fmanNote Added: 0003121
2008-01-30 07:40francislNote Added: 0003147
2008-01-30 15:45fmanNote Added: 0003148
2008-01-30 16:04fmanNote Added: 0003149
2008-01-30 21:02fmanSummaryMultiple test steps and expected results => [multiple steps] - Multiple test steps and expected results
2008-02-07 17:43olivier_houdasNote Added: 0003203
2008-02-15 21:42francislNote Added: 0003296
2008-06-19 03:50jorgecNote Added: 0003711
2008-06-26 01:59talameNote Added: 0003727
2008-06-26 15:39mhavlatNote Added: 0003729
2008-06-26 15:39mhavlatPrioritynormal => high
2008-06-26 15:39mhavlatProjectionnone => redesign
2008-06-26 15:39mhavlatETAnone => < 1 month
2008-06-27 21:41francislNote Added: 0003741
2009-04-02 21:46ssinghNote Added: 0006178
2009-04-08 16:57fmanNote Deleted: 0006178
2009-04-23 22:43ssinghTag Attached: Attachment
2009-04-23 22:44ssinghFile Added: Steps_Results
2009-04-23 22:46ssinghNote Added: 0006554
2009-04-24 14:45francislNote Added: 0006561
2009-05-14 14:50mhavlatRelationship addedparent of 0000718
2009-05-31 01:36mhavlatRelationship addedhas duplicate 0002555
2009-09-06 01:06bighuxTag Attached: test
2009-09-06 01:07bighuxTag Detached: test
2010-04-07 11:53tarun3kumarNote Added: 0009664
2010-04-07 20:35fmanNote Added: 0009665
2011-03-12 10:45JulianRelationship addedparent of 0004318
2012-04-28 17:08fmanRelationship addedhas duplicate 0004986
2014-05-24 08:33fmanQA Team - Task Workflow Status => READY FOR TESTING
2014-05-24 08:33fmanNote Added: 0021004
2014-05-24 08:33fmanStatusassigned => resolved
2014-05-24 08:33fmanFixed in Version => 1.9.11 (2014 Q2 - bug fixing)
2014-05-24 08:33fmanResolutionopen => fixed
2014-05-24 10:35fmanRelationship addedchild of 0006313
2014-07-28 20:33fmanStatusresolved => closed

Notes
(0002996)
fman   
2008-01-22 15:58   
1. TL architecture will not allow TestCase to be a TestCase container
2. Please next time look for similar issues before reporting.
3. Today you have a workaroung -> create tables inside steps and expected results.
4. ISSUE is:
    Any step will have it's result during execution ?
    if anwser is yes, Which will be the algorithm to compute TestCase exec status using steps status ?

Need deep discussion
(0003021)
francisl   
2008-01-23 14:22   
1. Sorry, I don't understand this point. Need more clarification. Please mail me.
2. I have searched Mantis but found not similar issue. Need to improve my searching skill.
3. Interesting workaround. However, it is not different to use number list to associated steps and expected results.
4. Yes, each step could have its execution result. The rules I used to determine final results are: a) test failed if any of the steps failed; b) test result overridden if justifications supplied by tester.

I am happy to further discuss this request as I would like to see TL works better for testing.

(0003024)
fman   
2008-01-23 15:26   
1. TL architecture will not allow TestCase to be a TestCase container: means
if you analise Database schema (or read old doc in document repository on CVS) you will find that TC can contain just TCversions.

We will not change this.
(0003027)
francisl   
2008-01-23 15:54   
(edited on: 2008-01-23 15:59)
Thank you for the explanation. So, I assume it refers to item 2 in my suggested features.

1. The request is based on good faith and my experience.
2. Version of TC is certainly important for configuration control.
3. No matter we like it or not, the fact is that it is very common to have a set of steps repeating in many test cases.
4. Tools should facilitate works instead of bending work to fit a tool.
5. Reading database schema is rare while evaluating a test tool.

By the way, it is also a good idea to able to group steps as suggested in the other request.

(0003028)
fman   
2008-01-23 16:29   
>> So, I assume it refers to item 2 in my suggested features.
yes

>> 1. The request is based on good faith and my experience.
no doubts.
>> 2. Version of TC is certainly important for configuration control.
>> 3. No matter we like it or not, the fact is that it is very common to have a set of steps repeating in many test cases.
OK, then you can COPY common steps on new testcases and add new one.


>> 4. Tools should facilitate works instead of bending work to fit a tool.
I do not fully agree, because sometime is not possible.
who can THE VISION when trying to develop a product/solution ?

>> 5. Reading database schema is rare while evaluating a test tool.
I'm just explaining why can not be done.


Need more detailed info (examples, ecc) in order to try to evaluate a solution.
(not in the short time)

regards
(0003029)
fman   
2008-01-23 16:38   
TARGET RELEASE: 1.9
(0003120)
olivier_houdas   
2008-01-29 18:20   
If the ability to sort the TC execution order inside a Testplan is given, wouldn't this be a good way to address the issue ?
(0003121)
fman   
2008-01-29 18:36   
>> If the ability to sort the TC execution order inside a Testplan is given,
>> wouldn't this be a good way to address the issue ?

1. can not be done (in short/mid time)
   due to:
   a. lot of changes at DB level
   b. not as simple as seems due to testcase link to testcase implementation.

2. will be just a workaround, (used once for a customer)
   where you consider Test Suite ad object that will have status.

3. if we want steps inside testcase, we need to implement it as clean as possible
(0003147)
francisl   
2008-01-30 07:40   
> who can THE VISION when trying to develop a product/solution ?
I have a vision but I don't want to force it on other people's project. :-)

> I'm just explaining why can not be done.
Sorry, I misunderstood your meaning. :-(

> If the ability to sort the TC execution order inside a Testplan is given, wouldn't this be a good way to address the issue ?
Don't understand "to sort the TC execution order". Need clarification.
(0003148)
fman   
2008-01-30 15:45   
1. I'm wrong: test cases execution order = test case order inside test suite.

2.
>>I do not fully agree, because sometime is not possible.
>>who can THE VISION when trying to develop a product/solution ?
wrong english:
who can have THE right VISION when trying to develop a product/solution ?
(0003149)
fman   
2008-01-30 16:04   
@ olivier_houdas :

I've think over your idea.
I will have some documentation during next week
(0003203)
olivier_houdas   
2008-02-07 17:43   
Well, I am new to Testlink, and I just discovered that you can order TCs and Testsuites.
So in fact, I found the solution to what I needed.

Regarding the present MANTIS item, isn't it the solution :
- A test case with multiple steps should in fact just be a sub-test suite;
- Inside the sub-test suite, you can create as many TCs as there are steps;
- If you forgot one step, you just need to add it at the end, then click on the Reorder children button, which is available when you click on the sub test suite item in the tree.

francisl, isn't it what you were actually looking for ?
(0003296)
francisl   
2008-02-15 21:42   
Hello, olivier_houdas,

Before I answer your question, I would like to clarify the relationships of these terms.
1. A test suite (test set) consist of multiple tests and/or another test suite;
2. A test consists of a test case and test data.
3. A test case consists of multiple test steps and expected results.

Based on TestLink's sample project, it satisfies point 1 above. Certainly, you can use test suite to document a test and use test case to document a test step. But, this does not improve the usability of the tool.
(0003711)
jorgec   
2008-06-19 03:50   
joc - 06/18/2008:
Definitely we need multi-steps within same test case as other TMF are supporting now i.e.~SalomeTMF, aqManager, etc.
(0003727)
talame   
2008-06-26 01:59   
There are two ways Test Case forms, the top-down approach and the side-by-side approach.
Top-dowjn Approach:

Steps:
Step 1.
Step 2.
Step n.
Expected Result:
(expected result)

Side-by-Side approach:
Steps: Expected Result:
----- ----------------
1.Enter Username, type: XYZ 1. Username displayed
2.Enter Password, type: abc 2. Password masked.
3.Click on the Submit button 3. The <TATA> screen is displayed.

Which ever approach is dependented on your needs. I switch between both.
(0003729)
mhavlat   
2008-06-26 15:39   
I can confirm that it is in scope of TestLink 1.9.
My view:

REQ 1
User can choose TC type: htmltext steps/expctedResults, simpletext steps/expctedResults and step by step.

REQ 2 Step by step design
User can add/modify/delete rows that contain two simpletext areas for steps and expected results data. User can changes order of steps.
We can implement basic html tags for the data according resources. FCKEditor will have special menu definition in this case.

REQ 3 Step by step results
Test case execution will define overall TC result and results for particular steps. The next four statuses are applicable for steps only: not run (default), pass, fail and blocked. Timestamp and a user_id is recorded for each step.

User can fill test results and test note for particular steps. User can fill test results for all steps by one click. User can modify overall test results independently on steps. TestLink automatically modify overall test result if
- all step PASSED
- result FAILED is added to any step
- result BLOCKED is added to any step and overall test result is Not run.

REQ 4 Performance
We should consider to show just one Test case in execution page (instead of list) to compass performance problems. All other pages should show just overall status.

REQ 5 Rendering
Test case content rendering should be extracted to extra templates per TC type.


Database changes consideration
There is required a new TC attribute tc_type.
The simplest solution to store steps are using field steps to store all structure. The similar way is applicable to steps test results with the filed 'note'.
Robust solution means to introduce two new tables tc_steps and tc_steps_result.


Is somebody able to compare this draft with other tools? Do I miss something?
(0003741)
francisl   
2008-06-27 21:41   
Below are my suggestions and comments:

1. REQ 3: Step by step results

1.1 Since it is rare to have steps executed by separate testers, there is no need to record timestamp and user Id for each step. Just record timestamp and user Id for each TC execution should be adequate.

1.2 I think the following rules should be used to determine the overall TC execution result:
     a) If all steps passed, TC execution result should be "Passed".
     b) If any step failed, TC execution should be terminated and results as "Failed".
     c) If any step blocked, TC execution should be terminated and results as "Blocked".
     d) If execution result is failed and justification is provided, results should be overwritten as "Passed".

1.3 It would be helpful if creator of a TC execution record is allowed to update actual results and notes. Tester may find typos in their record and need to correct them after the record is saved.

2. REQ 4: Performance

2.1 TC execution report should present results of last execution as default. However, user should be allowed to present all execution results of a TC.

3. Database changes

3.1 There may be situation that one step associated with multiple expected results. However, by allowing to use numbered list in the expected results should be able to resolve this.

3.2 It would be great if common/shared steps can be defined like a subroutine and called by TC's. For example, steps to login a system may be defined as common/shared steps. It can be called by various TC's to form a complete TC. This feature is available from TestDirector.
(0006554)
ssingh   
2009-04-23 22:46   
See the step and results screen shot given in attached file Steps_Results - which will be perfect scenario to have
(0006561)
francisl   
2009-04-24 14:45   
Regarding REQ 3, I would like to repeat my suggestion:

The overall TC execution result should be:
     a) If all steps passed, TC execution result should be "Passed".
     b) If any step failed, TC execution should be terminated and results as "Failed".
     c) If any step blocked, TC execution should be terminated and results as "Blocked".
     d) If execution result is failed and justification is provided, results should be overwritten as "Passed".

Occasionally, there is a need for scenario (d).
(0009664)
tarun3kumar   
2010-04-07 11:53   
Having ability to capture results for different steps of test case is must.
I am yet to dig in to Reporting capabilities of TestLink but I believe that it should be reflected in Test Reports also.
I completely agree with francisl last comment on this issue.
This issue was fist reported on -> 2008-01-22 07:28 and is still in Open Status.
Is there any deadline towards fixing this?

~ T
(0009665)
fman   
2010-04-07 20:35   
>> This issue was fist reported on -> 2008-01-22 07:28
>> and is still in Open Status.
And will not completely fixed on 1.9 => result at step level will not be available due to amount of rework to be done.
(0021004)
fman   
2014-05-24 08:33   
https://gitorious.org/testlink-ga/testlink-code/commit/480f91743d81e7d6efdef0dfa182eaaa13ecc979 [^]