Mantis Bugtracker          
testlink.org

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0001310TestLinkTest Specificationpublic2008-01-22 15:282014-07-28 20:33
Reporterfrancisl 
Assigned Tofman 
PriorityhighSeverityfeature requestReproducibilityN/A
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version 
Fixed in Version1.9.11 (2014 Q2 - bug fixing) 
Summary0001310: [multiple steps] - Multiple test steps and expected results
Description1. Usually, test case consists of multiple test steps. Each test step associated with an expected results. It would be useful if TL could support this characteristics of test case.

2. If point 1 above is implemented, it would be useful if a test case can refer to another test case which acts as a container of some common test steps. This is similar to the concept of php include files or subprograms.
TagsAttachment
Database (MySQL,Postgres,etc)
Browser
PHP Version
TestCaseID
QA Team - Task Workflow StatusREADY FOR TESTING
Attached Files? file icon Steps_Results [^] (116,551 bytes) 2009-04-23 22:44 [Show Content]

- Relationships
related to 0000507closedfman [multiple steps] - Test Specification: Separete steps and results 
related to 0000422closedfman [multiple steps] - possibility to split execution steps 
parent of 0000718closedfman New way to add expected results 
parent of 0004318closedfman Requirements mapping to Test Case step 
has duplicate 0002555closedmhavlat Test Steps and the expected Results Do not match up 
has duplicate 0004986closedfman Report ( RUN : Failed , OK, ) directly in the Teststeps and not only for the whole Testcase 
child of 0006313closedfman Availables hot-fixes for 1.9.10 & How To get full fixed package from gitorious 

-  Notes
(0002996)
fman (administrator)
2008-01-22 15:58

1. TL architecture will not allow TestCase to be a TestCase container
2. Please next time look for similar issues before reporting.
3. Today you have a workaroung -> create tables inside steps and expected results.
4. ISSUE is:
    Any step will have it's result during execution ?
    if anwser is yes, Which will be the algorithm to compute TestCase exec status using steps status ?

Need deep discussion
(0003021)
francisl (reporter)
2008-01-23 14:22
edited on: 2008-01-23 14:22

1. Sorry, I don't understand this point. Need more clarification. Please mail me.
2. I have searched Mantis but found not similar issue. Need to improve my searching skill.
3. Interesting workaround. However, it is not different to use number list to associated steps and expected results.
4. Yes, each step could have its execution result. The rules I used to determine final results are: a) test failed if any of the steps failed; b) test result overridden if justifications supplied by tester.

I am happy to further discuss this request as I would like to see TL works better for testing.

(0003024)
fman (administrator)
2008-01-23 15:26

1. TL architecture will not allow TestCase to be a TestCase container: means
if you analise Database schema (or read old doc in document repository on CVS) you will find that TC can contain just TCversions.

We will not change this.
(0003027)
francisl (reporter)
2008-01-23 15:54
edited on: 2008-01-23 15:59

Thank you for the explanation. So, I assume it refers to item 2 in my suggested features.

1. The request is based on good faith and my experience.
2. Version of TC is certainly important for configuration control.
3. No matter we like it or not, the fact is that it is very common to have a set of steps repeating in many test cases.
4. Tools should facilitate works instead of bending work to fit a tool.
5. Reading database schema is rare while evaluating a test tool.

By the way, it is also a good idea to able to group steps as suggested in the other request.

(0003028)
fman (administrator)
2008-01-23 16:29

>> So, I assume it refers to item 2 in my suggested features.
yes

>> 1. The request is based on good faith and my experience.
no doubts.
>> 2. Version of TC is certainly important for configuration control.
>> 3. No matter we like it or not, the fact is that it is very common to have a set of steps repeating in many test cases.
OK, then you can COPY common steps on new testcases and add new one.


>> 4. Tools should facilitate works instead of bending work to fit a tool.
I do not fully agree, because sometime is not possible.
who can THE VISION when trying to develop a product/solution ?

>> 5. Reading database schema is rare while evaluating a test tool.
I'm just explaining why can not be done.


Need more detailed info (examples, ecc) in order to try to evaluate a solution.
(not in the short time)

regards
(0003029)
fman (administrator)
2008-01-23 16:38

TARGET RELEASE: 1.9
(0003120)
olivier_houdas (reporter)
2008-01-29 18:20

If the ability to sort the TC execution order inside a Testplan is given, wouldn't this be a good way to address the issue ?
(0003121)
fman (administrator)
2008-01-29 18:36

>> If the ability to sort the TC execution order inside a Testplan is given,
>> wouldn't this be a good way to address the issue ?

1. can not be done (in short/mid time)
   due to:
   a. lot of changes at DB level
   b. not as simple as seems due to testcase link to testcase implementation.

2. will be just a workaround, (used once for a customer)
   where you consider Test Suite ad object that will have status.

3. if we want steps inside testcase, we need to implement it as clean as possible
(0003147)
francisl (reporter)
2008-01-30 07:40

> who can THE VISION when trying to develop a product/solution ?
I have a vision but I don't want to force it on other people's project. :-)

> I'm just explaining why can not be done.
Sorry, I misunderstood your meaning. :-(

> If the ability to sort the TC execution order inside a Testplan is given, wouldn't this be a good way to address the issue ?
Don't understand "to sort the TC execution order". Need clarification.
(0003148)
fman (administrator)
2008-01-30 15:45

1. I'm wrong: test cases execution order = test case order inside test suite.

2.
>>I do not fully agree, because sometime is not possible.
>>who can THE VISION when trying to develop a product/solution ?
wrong english:
who can have THE right VISION when trying to develop a product/solution ?
(0003149)
fman (administrator)
2008-01-30 16:04

@ olivier_houdas :

I've think over your idea.
I will have some documentation during next week
(0003203)
olivier_houdas (reporter)
2008-02-07 17:43

Well, I am new to Testlink, and I just discovered that you can order TCs and Testsuites.
So in fact, I found the solution to what I needed.

Regarding the present MANTIS item, isn't it the solution :
- A test case with multiple steps should in fact just be a sub-test suite;
- Inside the sub-test suite, you can create as many TCs as there are steps;
- If you forgot one step, you just need to add it at the end, then click on the Reorder children button, which is available when you click on the sub test suite item in the tree.

francisl, isn't it what you were actually looking for ?
(0003296)
francisl (reporter)
2008-02-15 21:42

Hello, olivier_houdas,

Before I answer your question, I would like to clarify the relationships of these terms.
1. A test suite (test set) consist of multiple tests and/or another test suite;
2. A test consists of a test case and test data.
3. A test case consists of multiple test steps and expected results.

Based on TestLink's sample project, it satisfies point 1 above. Certainly, you can use test suite to document a test and use test case to document a test step. But, this does not improve the usability of the tool.
(0003711)
jorgec (reporter)
2008-06-19 03:50

joc - 06/18/2008:
Definitely we need multi-steps within same test case as other TMF are supporting now i.e.~SalomeTMF, aqManager, etc.
(0003727)
talame (reporter)
2008-06-26 01:59

There are two ways Test Case forms, the top-down approach and the side-by-side approach.
Top-dowjn Approach:

Steps:
Step 1.
Step 2.
Step n.
Expected Result:
(expected result)

Side-by-Side approach:
Steps: Expected Result:
----- ----------------
1.Enter Username, type: XYZ 1. Username displayed
2.Enter Password, type: abc 2. Password masked.
3.Click on the Submit button 3. The <TATA> screen is displayed.

Which ever approach is dependented on your needs. I switch between both.
(0003729)
mhavlat (reporter)
2008-06-26 15:39

I can confirm that it is in scope of TestLink 1.9.
My view:

REQ 1
User can choose TC type: htmltext steps/expctedResults, simpletext steps/expctedResults and step by step.

REQ 2 Step by step design
User can add/modify/delete rows that contain two simpletext areas for steps and expected results data. User can changes order of steps.
We can implement basic html tags for the data according resources. FCKEditor will have special menu definition in this case.

REQ 3 Step by step results
Test case execution will define overall TC result and results for particular steps. The next four statuses are applicable for steps only: not run (default), pass, fail and blocked. Timestamp and a user_id is recorded for each step.

User can fill test results and test note for particular steps. User can fill test results for all steps by one click. User can modify overall test results independently on steps. TestLink automatically modify overall test result if
- all step PASSED
- result FAILED is added to any step
- result BLOCKED is added to any step and overall test result is Not run.

REQ 4 Performance
We should consider to show just one Test case in execution page (instead of list) to compass performance problems. All other pages should show just overall status.

REQ 5 Rendering
Test case content rendering should be extracted to extra templates per TC type.


Database changes consideration
There is required a new TC attribute tc_type.
The simplest solution to store steps are using field steps to store all structure. The similar way is applicable to steps test results with the filed 'note'.
Robust solution means to introduce two new tables tc_steps and tc_steps_result.


Is somebody able to compare this draft with other tools? Do I miss something?
(0003741)
francisl (reporter)
2008-06-27 21:41

Below are my suggestions and comments:

1. REQ 3: Step by step results

1.1 Since it is rare to have steps executed by separate testers, there is no need to record timestamp and user Id for each step. Just record timestamp and user Id for each TC execution should be adequate.

1.2 I think the following rules should be used to determine the overall TC execution result:
     a) If all steps passed, TC execution result should be "Passed".
     b) If any step failed, TC execution should be terminated and results as "Failed".
     c) If any step blocked, TC execution should be terminated and results as "Blocked".
     d) If execution result is failed and justification is provided, results should be overwritten as "Passed".

1.3 It would be helpful if creator of a TC execution record is allowed to update actual results and notes. Tester may find typos in their record and need to correct them after the record is saved.

2. REQ 4: Performance

2.1 TC execution report should present results of last execution as default. However, user should be allowed to present all execution results of a TC.

3. Database changes

3.1 There may be situation that one step associated with multiple expected results. However, by allowing to use numbered list in the expected results should be able to resolve this.

3.2 It would be great if common/shared steps can be defined like a subroutine and called by TC's. For example, steps to login a system may be defined as common/shared steps. It can be called by various TC's to form a complete TC. This feature is available from TestDirector.
(0006554)
ssingh (reporter)
2009-04-23 22:46

See the step and results screen shot given in attached file Steps_Results - which will be perfect scenario to have
(0006561)
francisl (reporter)
2009-04-24 14:45

Regarding REQ 3, I would like to repeat my suggestion:

The overall TC execution result should be:
     a) If all steps passed, TC execution result should be "Passed".
     b) If any step failed, TC execution should be terminated and results as "Failed".
     c) If any step blocked, TC execution should be terminated and results as "Blocked".
     d) If execution result is failed and justification is provided, results should be overwritten as "Passed".

Occasionally, there is a need for scenario (d).
(0009664)
tarun3kumar (reporter)
2010-04-07 11:53

Having ability to capture results for different steps of test case is must.
I am yet to dig in to Reporting capabilities of TestLink but I believe that it should be reflected in Test Reports also.
I completely agree with francisl last comment on this issue.
This issue was fist reported on -> 2008-01-22 07:28 and is still in Open Status.
Is there any deadline towards fixing this?

~ T
(0009665)
fman (administrator)
2010-04-07 20:35

>> This issue was fist reported on -> 2008-01-22 07:28
>> and is still in Open Status.
And will not completely fixed on 1.9 => result at step level will not be available due to amount of rework to be done.
(0021004)
fman (administrator)
2014-05-24 08:33

https://gitorious.org/testlink-ga/testlink-code/commit/480f91743d81e7d6efdef0dfa182eaaa13ecc979 [^]

- Issue History
Date Modified Username Field Change
2008-01-22 15:28 francisl New Issue
2008-01-22 15:58 fman Note Added: 0002996
2008-01-23 14:22 francisl Note Added: 0003021
2008-01-23 14:22 francisl Note Edited: 0003021
2008-01-23 15:26 fman Note Added: 0003024
2008-01-23 15:29 fman Relationship added related to 0000507
2008-01-23 15:29 fman Relationship added related to 0000422
2008-01-23 15:54 francisl Note Added: 0003027
2008-01-23 15:59 francisl Note Edited: 0003027
2008-01-23 15:59 francisl Note Edited: 0003027
2008-01-23 16:29 fman Note Added: 0003028
2008-01-23 16:38 fman Status new => assigned
2008-01-23 16:38 fman Assigned To => fman
2008-01-23 16:38 fman Note Added: 0003029
2008-01-29 18:20 olivier_houdas Note Added: 0003120
2008-01-29 18:36 fman Note Added: 0003121
2008-01-30 07:40 francisl Note Added: 0003147
2008-01-30 15:45 fman Note Added: 0003148
2008-01-30 16:04 fman Note Added: 0003149
2008-01-30 21:02 fman Summary Multiple test steps and expected results => [multiple steps] - Multiple test steps and expected results
2008-02-07 17:43 olivier_houdas Note Added: 0003203
2008-02-15 21:42 francisl Note Added: 0003296
2008-06-19 03:50 jorgec Note Added: 0003711
2008-06-26 01:59 talame Note Added: 0003727
2008-06-26 15:39 mhavlat Note Added: 0003729
2008-06-26 15:39 mhavlat Priority normal => high
2008-06-26 15:39 mhavlat Projection none => redesign
2008-06-26 15:39 mhavlat ETA none => < 1 month
2008-06-27 21:41 francisl Note Added: 0003741
2009-04-02 21:46 ssingh Note Added: 0006178
2009-04-08 16:57 fman Note Deleted: 0006178
2009-04-23 22:43 ssingh Tag Attached: Attachment
2009-04-23 22:44 ssingh File Added: Steps_Results
2009-04-23 22:46 ssingh Note Added: 0006554
2009-04-24 14:45 francisl Note Added: 0006561
2009-05-14 14:50 mhavlat Relationship added parent of 0000718
2009-05-31 01:36 mhavlat Relationship added has duplicate 0002555
2009-09-06 01:06 bighux Tag Attached: test
2009-09-06 01:07 bighux Tag Detached: test
2010-04-07 11:53 tarun3kumar Note Added: 0009664
2010-04-07 20:35 fman Note Added: 0009665
2011-03-12 10:45 Julian Relationship added parent of 0004318
2012-04-28 17:08 fman Relationship added has duplicate 0004986
2014-05-24 08:33 fman QA Team - Task Workflow Status => READY FOR TESTING
2014-05-24 08:33 fman Note Added: 0021004
2014-05-24 08:33 fman Status assigned => resolved
2014-05-24 08:33 fman Fixed in Version => 1.9.11 (2014 Q2 - bug fixing)
2014-05-24 08:33 fman Resolution open => fixed
2014-05-24 10:35 fman Relationship added child of 0006313
2014-07-28 20:33 fman Status resolved => closed



Copyright © 2000 - 2018 MantisBT Team
Powered by Mantis Bugtracker