0001310: [multiple steps] - Multiple test steps and expected results

(0002996)
fman (administrator)
2008-01-22 15:58

1. TL architecture will not allow TestCase to be a TestCase container
2. Please next time look for similar issues before reporting.
3. Today you have a workaroung -> create tables inside steps and expected results.
4. ISSUE is:
Any step will have it's result during execution ?
if anwser is yes, Which will be the algorithm to compute TestCase exec status using steps status ?

Need deep discussion

(0003021)
francisl (reporter)
2008-01-23 14:22
edited on: 2008-01-23 14:22

1. Sorry, I don't understand this point. Need more clarification. Please mail me.
2. I have searched Mantis but found not similar issue. Need to improve my searching skill.
3. Interesting workaround. However, it is not different to use number list to associated steps and expected results.
4. Yes, each step could have its execution result. The rules I used to determine final results are: a) test failed if any of the steps failed; b) test result overridden if justifications supplied by tester.

I am happy to further discuss this request as I would like to see TL works better for testing.

(0003024)
fman (administrator)
2008-01-23 15:26

1. TL architecture will not allow TestCase to be a TestCase container: means
if you analise Database schema (or read old doc in document repository on CVS) you will find that TC can contain just TCversions.

We will not change this.

(0003027)
francisl (reporter)
2008-01-23 15:54
edited on: 2008-01-23 15:59

Thank you for the explanation. So, I assume it refers to item 2 in my suggested features.

1. The request is based on good faith and my experience.
2. Version of TC is certainly important for configuration control.
3. No matter we like it or not, the fact is that it is very common to have a set of steps repeating in many test cases.
4. Tools should facilitate works instead of bending work to fit a tool.
5. Reading database schema is rare while evaluating a test tool.

By the way, it is also a good idea to able to group steps as suggested in the other request.

(0003028)
fman (administrator)
2008-01-23 16:29

>> So, I assume it refers to item 2 in my suggested features.
yes

>> 1. The request is based on good faith and my experience.
no doubts.
>> 2. Version of TC is certainly important for configuration control.
>> 3. No matter we like it or not, the fact is that it is very common to have a set of steps repeating in many test cases.
OK, then you can COPY common steps on new testcases and add new one.

>> 4. Tools should facilitate works instead of bending work to fit a tool.
I do not fully agree, because sometime is not possible.
who can THE VISION when trying to develop a product/solution ?

>> 5. Reading database schema is rare while evaluating a test tool.
I'm just explaining why can not be done.

Need more detailed info (examples, ecc) in order to try to evaluate a solution.
(not in the short time)

regards

(0003029)
fman (administrator)
2008-01-23 16:38

TARGET RELEASE: 1.9

(0003120)
olivier_houdas (reporter)
2008-01-29 18:20

If the ability to sort the TC execution order inside a Testplan is given, wouldn't this be a good way to address the issue ?

(0003121)
fman (administrator)
2008-01-29 18:36

>> If the ability to sort the TC execution order inside a Testplan is given,
>> wouldn't this be a good way to address the issue ?

1. can not be done (in short/mid time)
   due to:
   a. lot of changes at DB level
   b. not as simple as seems due to testcase link to testcase implementation.

2. will be just a workaround, (used once for a customer)
   where you consider Test Suite ad object that will have status.

3. if we want steps inside testcase, we need to implement it as clean as possible

(0003147)
francisl (reporter)
2008-01-30 07:40

> who can THE VISION when trying to develop a product/solution ?
I have a vision but I don't want to force it on other people's project. :-)

> I'm just explaining why can not be done.
Sorry, I misunderstood your meaning. :-(

> If the ability to sort the TC execution order inside a Testplan is given, wouldn't this be a good way to address the issue ?
Don't understand "to sort the TC execution order". Need clarification.

(0003148)
fman (administrator)
2008-01-30 15:45

1. I'm wrong: test cases execution order = test case order inside test suite.

2.
>>I do not fully agree, because sometime is not possible.
>>who can THE VISION when trying to develop a product/solution ?
wrong english:
who can have THE right VISION when trying to develop a product/solution ?

(0003149)
fman (administrator)
2008-01-30 16:04

@ olivier_houdas :

I've think over your idea.
I will have some documentation during next week

(0003203)
olivier_houdas (reporter)
2008-02-07 17:43

Well, I am new to Testlink, and I just discovered that you can order TCs and Testsuites.
So in fact, I found the solution to what I needed.

Regarding the present MANTIS item, isn't it the solution :
- A test case with multiple steps should in fact just be a sub-test suite;
- Inside the sub-test suite, you can create as many TCs as there are steps;
- If you forgot one step, you just need to add it at the end, then click on the Reorder children button, which is available when you click on the sub test suite item in the tree.

francisl, isn't it what you were actually looking for ?

(0003296)
francisl (reporter)
2008-02-15 21:42

Hello, olivier_houdas,

Before I answer your question, I would like to clarify the relationships of these terms.
1. A test suite (test set) consist of multiple tests and/or another test suite;
2. A test consists of a test case and test data.
3. A test case consists of multiple test steps and expected results.

Based on TestLink's sample project, it satisfies point 1 above. Certainly, you can use test suite to document a test and use test case to document a test step. But, this does not improve the usability of the tool.

(0003711)
jorgec (reporter)
2008-06-19 03:50

joc - 06/18/2008:
Definitely we need multi-steps within same test case as other TMF are supporting now i.e.~SalomeTMF, aqManager, etc.

(0003727)
talame (reporter)
2008-06-26 01:59

There are two ways Test Case forms, the top-down approach and the side-by-side approach.
Top-dowjn Approach:

Steps:
Step 1.
Step 2.
Step n.
Expected Result:
(expected result)

Side-by-Side approach:
Steps: Expected Result:
----- ----------------
1.Enter Username, type: XYZ 1. Username displayed
2.Enter Password, type: abc 2. Password masked.
3.Click on the Submit button 3. The <TATA> screen is displayed.

Which ever approach is dependented on your needs. I switch between both.

(0003729)
mhavlat (reporter)
2008-06-26 15:39

I can confirm that it is in scope of TestLink 1.9.
My view:

REQ 1
User can choose TC type: htmltext steps/expctedResults, simpletext steps/expctedResults and step by step.

REQ 2 Step by step design
User can add/modify/delete rows that contain two simpletext areas for steps and expected results data. User can changes order of steps.
We can implement basic html tags for the data according resources. FCKEditor will have special menu definition in this case.

REQ 3 Step by step results
Test case execution will define overall TC result and results for particular steps. The next four statuses are applicable for steps only: not run (default), pass, fail and blocked. Timestamp and a user_id is recorded for each step.

User can fill test results and test note for particular steps. User can fill test results for all steps by one click. User can modify overall test results independently on steps. TestLink automatically modify overall test result if
- all step PASSED
- result FAILED is added to any step
- result BLOCKED is added to any step and overall test result is Not run.

REQ 4 Performance
We should consider to show just one Test case in execution page (instead of list) to compass performance problems. All other pages should show just overall status.

REQ 5 Rendering
Test case content rendering should be extracted to extra templates per TC type.

Database changes consideration
There is required a new TC attribute tc_type.
The simplest solution to store steps are using field steps to store all structure. The similar way is applicable to steps test results with the filed 'note'.
Robust solution means to introduce two new tables tc_steps and tc_steps_result.

Is somebody able to compare this draft with other tools? Do I miss something?

(0003741)
francisl (reporter)
2008-06-27 21:41

Below are my suggestions and comments:

1. REQ 3: Step by step results

1.1 Since it is rare to have steps executed by separate testers, there is no need to record timestamp and user Id for each step. Just record timestamp and user Id for each TC execution should be adequate.

1.2 I think the following rules should be used to determine the overall TC execution result:
     a) If all steps passed, TC execution result should be "Passed".
     b) If any step failed, TC execution should be terminated and results as "Failed".
     c) If any step blocked, TC execution should be terminated and results as "Blocked".
     d) If execution result is failed and justification is provided, results should be overwritten as "Passed".

1.3 It would be helpful if creator of a TC execution record is allowed to update actual results and notes. Tester may find typos in their record and need to correct them after the record is saved.

2. REQ 4: Performance

2.1 TC execution report should present results of last execution as default. However, user should be allowed to present all execution results of a TC.

3. Database changes

3.1 There may be situation that one step associated with multiple expected results. However, by allowing to use numbered list in the expected results should be able to resolve this.

3.2 It would be great if common/shared steps can be defined like a subroutine and called by TC's. For example, steps to login a system may be defined as common/shared steps. It can be called by various TC's to form a complete TC. This feature is available from TestDirector.

(0006554)
ssingh (reporter)
2009-04-23 22:46

See the step and results screen shot given in attached file Steps_Results - which will be perfect scenario to have

(0006561)
francisl (reporter)
2009-04-24 14:45

Regarding REQ 3, I would like to repeat my suggestion:

The overall TC execution result should be:
     a) If all steps passed, TC execution result should be "Passed".
     b) If any step failed, TC execution should be terminated and results as "Failed".
     c) If any step blocked, TC execution should be terminated and results as "Blocked".
     d) If execution result is failed and justification is provided, results should be overwritten as "Passed".

Occasionally, there is a need for scenario (d).

(0009664)
tarun3kumar (reporter)
2010-04-07 11:53

Having ability to capture results for different steps of test case is must.
I am yet to dig in to Reporting capabilities of TestLink but I believe that it should be reflected in Test Reports also.
I completely agree with francisl last comment on this issue.
This issue was fist reported on -> 2008-01-22 07:28 and is still in Open Status.
Is there any deadline towards fixing this?

~ T

(0009665)
fman (administrator)
2010-04-07 20:35

>> This issue was fist reported on -> 2008-01-22 07:28
>> and is still in Open Status.
And will not completely fixed on 1.9 => result at step level will not be available due to amount of rework to be done.

(0021004)
fman (administrator)
2014-05-24 08:33

https://gitorious.org/testlink-ga/testlink-code/commit/480f91743d81e7d6efdef0dfa182eaaa13ecc979 [^]

Issue History
Date Modified	Username	Field	Change
2008-01-22 15:28	francisl	New Issue
2008-01-22 15:58	fman	Note Added: 0002996
2008-01-23 14:22	francisl	Note Added: 0003021
2008-01-23 14:22	francisl	Note Edited: 0003021
2008-01-23 15:26	fman	Note Added: 0003024
2008-01-23 15:29	fman	Relationship added	related to 0000507
2008-01-23 15:29	fman	Relationship added	related to 0000422
2008-01-23 15:54	francisl	Note Added: 0003027
2008-01-23 15:59	francisl	Note Edited: 0003027
2008-01-23 15:59	francisl	Note Edited: 0003027
2008-01-23 16:29	fman	Note Added: 0003028
2008-01-23 16:38	fman	Status	new => assigned
2008-01-23 16:38	fman	Assigned To	=> fman
2008-01-23 16:38	fman	Note Added: 0003029
2008-01-29 18:20	olivier_houdas	Note Added: 0003120
2008-01-29 18:36	fman	Note Added: 0003121
2008-01-30 07:40	francisl	Note Added: 0003147
2008-01-30 15:45	fman	Note Added: 0003148
2008-01-30 16:04	fman	Note Added: 0003149
2008-01-30 21:02	fman	Summary	Multiple test steps and expected results => [multiple steps] - Multiple test steps and expected results
2008-02-07 17:43	olivier_houdas	Note Added: 0003203
2008-02-15 21:42	francisl	Note Added: 0003296
2008-06-19 03:50	jorgec	Note Added: 0003711
2008-06-26 01:59	talame	Note Added: 0003727
2008-06-26 15:39	mhavlat	Note Added: 0003729
2008-06-26 15:39	mhavlat	Priority	normal => high
2008-06-26 15:39	mhavlat	Projection	none => redesign
2008-06-26 15:39	mhavlat	ETA	none => < 1 month
2008-06-27 21:41	francisl	Note Added: 0003741
2009-04-02 21:46	ssingh	Note Added: 0006178
2009-04-08 16:57	fman	Note Deleted: 0006178
2009-04-23 22:43	ssingh	Tag Attached: Attachment
2009-04-23 22:44	ssingh	File Added: Steps_Results
2009-04-23 22:46	ssingh	Note Added: 0006554
2009-04-24 14:45	francisl	Note Added: 0006561
2009-05-14 14:50	mhavlat	Relationship added	parent of 0000718
2009-05-31 01:36	mhavlat	Relationship added	has duplicate 0002555
2009-09-06 01:06	bighux	Tag Attached: test
2009-09-06 01:07	bighux	Tag Detached: test
2010-04-07 11:53	tarun3kumar	Note Added: 0009664
2010-04-07 20:35	fman	Note Added: 0009665
2011-03-12 10:45	Julian	Relationship added	parent of 0004318
2012-04-28 17:08	fman	Relationship added	has duplicate 0004986
2014-05-24 08:33	fman	QA Team - Task Workflow Status	=> READY FOR TESTING
2014-05-24 08:33	fman	Note Added: 0021004
2014-05-24 08:33	fman	Status	assigned => resolved
2014-05-24 08:33	fman	Fixed in Version	=> 1.9.11 (2014 Q2 - bug fixing)
2014-05-24 08:33	fman	Resolution	open => fixed
2014-05-24 10:35	fman	Relationship added	child of 0006313
2014-07-28 20:33	fman	Status	resolved => closed

Notes
(0002996) fman (administrator) 2008-01-22 15:58	1. TL architecture will not allow TestCase to be a TestCase container 2. Please next time look for similar issues before reporting. 3. Today you have a workaroung -> create tables inside steps and expected results. 4. ISSUE is: Any step will have it's result during execution ? if anwser is yes, Which will be the algorithm to compute TestCase exec status using steps status ? Need deep discussion

(0003021) francisl (reporter) 2008-01-23 14:22 edited on: 2008-01-23 14:22	1. Sorry, I don't understand this point. Need more clarification. Please mail me. 2. I have searched Mantis but found not similar issue. Need to improve my searching skill. 3. Interesting workaround. However, it is not different to use number list to associated steps and expected results. 4. Yes, each step could have its execution result. The rules I used to determine final results are: a) test failed if any of the steps failed; b) test result overridden if justifications supplied by tester. I am happy to further discuss this request as I would like to see TL works better for testing.

(0003024) fman (administrator) 2008-01-23 15:26	1. TL architecture will not allow TestCase to be a TestCase container: means if you analise Database schema (or read old doc in document repository on CVS) you will find that TC can contain just TCversions. We will not change this.

(0003027) francisl (reporter) 2008-01-23 15:54 edited on: 2008-01-23 15:59	Thank you for the explanation. So, I assume it refers to item 2 in my suggested features. 1. The request is based on good faith and my experience. 2. Version of TC is certainly important for configuration control. 3. No matter we like it or not, the fact is that it is very common to have a set of steps repeating in many test cases. 4. Tools should facilitate works instead of bending work to fit a tool. 5. Reading database schema is rare while evaluating a test tool. By the way, it is also a good idea to able to group steps as suggested in the other request.

(0003028) fman (administrator) 2008-01-23 16:29	>> So, I assume it refers to item 2 in my suggested features. yes >> 1. The request is based on good faith and my experience. no doubts. >> 2. Version of TC is certainly important for configuration control. >> 3. No matter we like it or not, the fact is that it is very common to have a set of steps repeating in many test cases. OK, then you can COPY common steps on new testcases and add new one. >> 4. Tools should facilitate works instead of bending work to fit a tool. I do not fully agree, because sometime is not possible. who can THE VISION when trying to develop a product/solution ? >> 5. Reading database schema is rare while evaluating a test tool. I'm just explaining why can not be done. Need more detailed info (examples, ecc) in order to try to evaluate a solution. (not in the short time) regards

(0003029) fman (administrator) 2008-01-23 16:38	TARGET RELEASE: 1.9

(0003120) olivier_houdas (reporter) 2008-01-29 18:20	If the ability to sort the TC execution order inside a Testplan is given, wouldn't this be a good way to address the issue ?

(0003121) fman (administrator) 2008-01-29 18:36	>> If the ability to sort the TC execution order inside a Testplan is given, >> wouldn't this be a good way to address the issue ? 1. can not be done (in short/mid time) due to: a. lot of changes at DB level b. not as simple as seems due to testcase link to testcase implementation. 2. will be just a workaround, (used once for a customer) where you consider Test Suite ad object that will have status. 3. if we want steps inside testcase, we need to implement it as clean as possible

(0003147) francisl (reporter) 2008-01-30 07:40	> who can THE VISION when trying to develop a product/solution ? I have a vision but I don't want to force it on other people's project. :-) > I'm just explaining why can not be done. Sorry, I misunderstood your meaning. :-( > If the ability to sort the TC execution order inside a Testplan is given, wouldn't this be a good way to address the issue ? Don't understand "to sort the TC execution order". Need clarification.

(0003148) fman (administrator) 2008-01-30 15:45	1. I'm wrong: test cases execution order = test case order inside test suite. 2. >>I do not fully agree, because sometime is not possible. >>who can THE VISION when trying to develop a product/solution ? wrong english: who can have THE right VISION when trying to develop a product/solution ?

(0003149) fman (administrator) 2008-01-30 16:04	@ olivier_houdas : I've think over your idea. I will have some documentation during next week

(0003203) olivier_houdas (reporter) 2008-02-07 17:43	Well, I am new to Testlink, and I just discovered that you can order TCs and Testsuites. So in fact, I found the solution to what I needed. Regarding the present MANTIS item, isn't it the solution : - A test case with multiple steps should in fact just be a sub-test suite; - Inside the sub-test suite, you can create as many TCs as there are steps; - If you forgot one step, you just need to add it at the end, then click on the Reorder children button, which is available when you click on the sub test suite item in the tree. francisl, isn't it what you were actually looking for ?

(0003296) francisl (reporter) 2008-02-15 21:42	Hello, olivier_houdas, Before I answer your question, I would like to clarify the relationships of these terms. 1. A test suite (test set) consist of multiple tests and/or another test suite; 2. A test consists of a test case and test data. 3. A test case consists of multiple test steps and expected results. Based on TestLink's sample project, it satisfies point 1 above. Certainly, you can use test suite to document a test and use test case to document a test step. But, this does not improve the usability of the tool.

(0003711) jorgec (reporter) 2008-06-19 03:50	joc - 06/18/2008: Definitely we need multi-steps within same test case as other TMF are supporting now i.e.~SalomeTMF, aqManager, etc.

(0003727) talame (reporter) 2008-06-26 01:59	There are two ways Test Case forms, the top-down approach and the side-by-side approach. Top-dowjn Approach: Steps: Step 1. Step 2. Step n. Expected Result: (expected result) Side-by-Side approach: Steps: Expected Result: ----- ---------------- 1.Enter Username, type: XYZ 1. Username displayed 2.Enter Password, type: abc 2. Password masked. 3.Click on the Submit button 3. The <TATA> screen is displayed. Which ever approach is dependented on your needs. I switch between both.

(0003729) mhavlat (reporter) 2008-06-26 15:39	I can confirm that it is in scope of TestLink 1.9. My view: REQ 1 User can choose TC type: htmltext steps/expctedResults, simpletext steps/expctedResults and step by step. REQ 2 Step by step design User can add/modify/delete rows that contain two simpletext areas for steps and expected results data. User can changes order of steps. We can implement basic html tags for the data according resources. FCKEditor will have special menu definition in this case. REQ 3 Step by step results Test case execution will define overall TC result and results for particular steps. The next four statuses are applicable for steps only: not run (default), pass, fail and blocked. Timestamp and a user_id is recorded for each step. User can fill test results and test note for particular steps. User can fill test results for all steps by one click. User can modify overall test results independently on steps. TestLink automatically modify overall test result if - all step PASSED - result FAILED is added to any step - result BLOCKED is added to any step and overall test result is Not run. REQ 4 Performance We should consider to show just one Test case in execution page (instead of list) to compass performance problems. All other pages should show just overall status. REQ 5 Rendering Test case content rendering should be extracted to extra templates per TC type. Database changes consideration There is required a new TC attribute tc_type. The simplest solution to store steps are using field steps to store all structure. The similar way is applicable to steps test results with the filed 'note'. Robust solution means to introduce two new tables tc_steps and tc_steps_result. Is somebody able to compare this draft with other tools? Do I miss something?

(0003741) francisl (reporter) 2008-06-27 21:41	Below are my suggestions and comments: 1. REQ 3: Step by step results 1.1 Since it is rare to have steps executed by separate testers, there is no need to record timestamp and user Id for each step. Just record timestamp and user Id for each TC execution should be adequate. 1.2 I think the following rules should be used to determine the overall TC execution result: a) If all steps passed, TC execution result should be "Passed". b) If any step failed, TC execution should be terminated and results as "Failed". c) If any step blocked, TC execution should be terminated and results as "Blocked". d) If execution result is failed and justification is provided, results should be overwritten as "Passed". 1.3 It would be helpful if creator of a TC execution record is allowed to update actual results and notes. Tester may find typos in their record and need to correct them after the record is saved. 2. REQ 4: Performance 2.1 TC execution report should present results of last execution as default. However, user should be allowed to present all execution results of a TC. 3. Database changes 3.1 There may be situation that one step associated with multiple expected results. However, by allowing to use numbered list in the expected results should be able to resolve this. 3.2 It would be great if common/shared steps can be defined like a subroutine and called by TC's. For example, steps to login a system may be defined as common/shared steps. It can be called by various TC's to form a complete TC. This feature is available from TestDirector.

(0006554) ssingh (reporter) 2009-04-23 22:46	See the step and results screen shot given in attached file Steps_Results - which will be perfect scenario to have

(0006561) francisl (reporter) 2009-04-24 14:45	Regarding REQ 3, I would like to repeat my suggestion: The overall TC execution result should be: a) If all steps passed, TC execution result should be "Passed". b) If any step failed, TC execution should be terminated and results as "Failed". c) If any step blocked, TC execution should be terminated and results as "Blocked". d) If execution result is failed and justification is provided, results should be overwritten as "Passed". Occasionally, there is a need for scenario (d).

(0009664) tarun3kumar (reporter) 2010-04-07 11:53	Having ability to capture results for different steps of test case is must. I am yet to dig in to Reporting capabilities of TestLink but I believe that it should be reflected in Test Reports also. I completely agree with francisl last comment on this issue. This issue was fist reported on -> 2008-01-22 07:28 and is still in Open Status. Is there any deadline towards fixing this? ~ T

(0009665) fman (administrator) 2010-04-07 20:35	>> This issue was fist reported on -> 2008-01-22 07:28 >> and is still in Open Status. And will not completely fixed on 1.9 => result at step level will not be available due to amount of rework to be done.

(0021004) fman (administrator) 2014-05-24 08:33	https://gitorious.org/testlink-ga/testlink-code/commit/480f91743d81e7d6efdef0dfa182eaaa13ecc979 [^]