Mantis Bugtracker 

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0002642TestLinkReportspublic2009-06-19 19:042010-05-01 20:32
Assigned Tofman 
PrioritynormalSeverityfeature requestReproducibilityhave not tried
PlatformOSOS Version
Product Version1.8.3 
Fixed in Version1.9 Beta 3 
Summary0002642: Platforms feature
DescriptionUse case:
User need to test one TC in several platforms (Linux, mikrosoft, Solaris) or with several interfaces or HW types. Report should include metrics overall and each particular CF value (as defined during test execution).

Only one CF (list type) could be possible to use for the purpose.

Additional InformationNeed to clarify by users:
1. What should be a content of the report?
2. How TestLink recognizes which CF should be used for the reporting?
TagsNo tags attached.
Database (MySQL,Postgres,etc)
PHP Version
QA Team - Task Workflow Status
Attached Files

- Relationships
has duplicate 0002793closedmhavlat Support for a testplan on mutiple configurations. 

-  Notes
tomtom (reporter)
2009-06-19 21:33

-- following [^]

I think it is a very big deal with a lot of rewrite...
Not only the reports but also the forms should be able to support duplication of TC based on a custom field.

This is what I'd like to be able to do :
* define a list of configuration, using a custom field or something else

* specify somewhere in the test plan and in the test requirement for which configuration/platform/interface the test has to be run. (one or many)

* Coverage, results and matrix reports should be aware of these configurations. ie, the requirement is not tested/passed until the test has been run on all configuration.
for instance, "not run test case"-report should include the configurations for which the test has not been run.

* When executing test, the form should propose execution for all the variant of the test (1 variant => 1 configuration). Assignation should be available for every variant...

Actually, what I tried is
Creating a "list" Custom Field with 3 values.
* "Assign requirement" => can not assign test cases based on this custom field.
* "Test Plan" => idem
* "Test reports", generaly only the last execution is used, whatever the content of the CF is.

Maybe a simpler solution is to be able to create multiples variant of a test case based on a custom field ? I don't know testlink enough to say.

But even then, I feel like it's a very complex problem and this solution won't solve everything...

Anyway, thank for your interest.
mhavlat (reporter)
2009-06-19 22:20

@tomtom, it looks like nearly all test plan features should be affected. In this case I don't see too much difference from using of several Test Plans (for each platform one).

I mean you could look for a "Master report" that will collect data from several Test Plans. For example "Failed TCs" will have several lists according to number of selected Test Plans. Does it make a sense?

Note that my both ideas are not too expensive to opposite to your requirement. Simple solution has real chance to be developed in reasonable time. Complex requests are often stored in this tracker for years.
In addition we should care about performance of requests. Queries are already difficult.
Eloff (reporter)
2009-06-30 16:35

This pretty close to what we want. I've seen more requests for this feature.

We use different testplans for different hardware today.

it feels like custom fields should be able to do this with some modifications. I don't have 100% knowledge of the inner workings of CF ... yet

I will also look into how CF_EXEC_TIME is made. Maybe it's possible to add custom logic in a similar way. (CF_PLATFORM?)

@mhavlat, I like your idea of a "master report". Maybe it is easier to implement this issue in presentation layer instead of cf?
mhavlat (reporter)
2009-07-02 22:08

Draft of design for discussion: [^]
mhavlat (reporter)
2009-07-02 23:11

Draft of implementation idea "platforms association"

why not use CF? We need to assign no, one or more to one TC.

User can define platforms related to project and can enable/disable record for using.
table platforms(platform_id, project_id, name[255], description[text], availability[0,1])
Question: should not be platform related to Test Plan instead of Project?

User can assign no, one more platforms to TC. No platform means that TC is supposed to test one times in a Test Plan.

table testcase_platforms(testcase_id, project_id, platform_id)

Execution page allows set result for particular platforms.
Execution navigator allows to according filter platforms.
General metrics shows metrics per platform.
Test Report print test results for all platforms valid for TC.
mhavlat (reporter)
2009-07-04 07:25
edited on: 2009-07-04 07:27

I think that we also can extend keywords instead of a new kind of object in my last suggestion.

I consider that platform should be related to Test Plan. Because user should test 5 platforms in one Test Plan and only one platform in the second Test Plan.

Implementation draft:
User define which keywords will behave as platforms a the Test Plan.
Selection is written to a new table platforms(keyword_id, testplan_id). Maybe as CF related to Test Plan ... not clear if it possible.

Execution page offers save more results with respect to number of defined platforms.
Such execution result save keyword_id into a new field of execution table. Otherwise null is default value.


Eloff (reporter)
2009-07-07 18:52

How about assigning TCs to testers? For example I'd like to do all tests on one platform, or maybe one test on all platforms. Is this possible with keywords solution?

"Execution page offers save more results with respect to number of defined platforms."
So you think of executing one test on all platforms at once? I'd like to have one item in the tree view for each execution I'm assigned to

Am I right you want to use testplan_id, tcversion_id, keyword_id in executions to indicate what has been executed?
Eloff (reporter)
2009-07-08 14:22

I felt I just couldn't do nothing so I tried out creating a master testplan. I managed to get execute view to work (showing one subtree for every platform).

However in add TCs to test plan there are some more trouble. Mostly with grabbing and structuring data from db. Functions as gen_spec_view are pretty big and hard for me to refactor. So you could say I maybe hit a wall :(

I have approx one month left in this project so we need things to speed up if I should be able to implement this feature.

Martin, I'm going to dig a bit deeper in your other proposals now
fman (administrator)
2009-07-10 23:54

Google document has not enough information to understand what is feature requested => IMHO a more in deep analisys is needed
Eloff (reporter)
2009-07-11 04:35

@fman: I added some more info to the google doc to be more clear.
Can you please be more specific what's missing.

Btw the google doc was more ment to be used for discussion and braing storming and not to be a complete solution. If it was complete I would have coded it already :)
fman (administrator)
2009-07-13 21:37

>> If it was complete I would have coded it already :)
IMHO with few data picture is incomplete then brainstroming can not be very useful.
IMHO on all this discussion we have tried to adapt a request to how TL is done today, and this is a mistake. I know you have time constrains but to reach a good solution we need time, and everybody must understand this.

IMHO we need to review approach to this problem in a new way, i.e.:
let's think we are staring development of TL then we are free to create a solution without constrains. After we have our "best solution" then we can start adding constrains (i.e. struggling with actual TL architecture).
what do you think ?
fman (administrator)
2009-07-15 00:34

More ideas or confusion:

P1 - Proposal based on Configuration for Test Plans.

1. new TL item configuration ( not a good name but wider that platform)
   a configuration will be used at UI level as a HTML select with multiselection always enabled.
   One possible approach
   Two tables needed:
   Master Configuration Data
   Detail Configuration Data (variations /config_item ??)
   Your proposal (with different names and minor change to be system wide)
   configurations(id, name[255], description[text], active[0,1])
   Link Table
   testplan_configuration (testplan_id, configuration_id)

2. item definition will be system wide, same way that custom fields are system wide,
   to allow reuse.
3. Firts Question:
   configuration will be linked to Test Projects or to Test Plans ?
   From discussion seems that will be linked to test plans.
   This is more flexible that linking to test projects ?
Issue/Design decission:
    a test plan can be linked one or zero configuration.

4. where configuration will have effect ?
   if a test plan has a configuration, then during testcase assignment will be asked
   to what item(s) of configuration must be assigned.
   User MUST choose one or more variation or system will not allow him/her to link testcase version.
   This way user will be able to create in a single shoot some sort of several
   sub-testplans (not a good name)
   To do this a new field must be added to testplan_tcversions table
   CREATE TABLE testplan_tcversions (
  `id` int(10) unsigned NOT NULL auto_increment,
  `testplan_id` int(10) unsigned NOT NULL default '0',
  `tcversion_id` int(10) unsigned NOT NULL default '0',
  `variation_id` int(10) unsigned NOT NULL default '0', <<<<<<< NEW FIELD
  `node_order` int(10) unsigned NOT NULL default '1',
  `urgency` smallint(5) NOT NULL default '2',
  `author_id` int(10) unsigned default NULL,
  `creation_ts` datetime NOT NULL default '0000-00-00 00:00:00',
   PRIMARY KEY (`id`),
   UNIQUE KEY `tp_tcversion` (testplan_id,tcversion_id,variation_id) <<<< KEY CHANGED

   Then suppose we have:
   Test Project StarTrek
   And we want to run test present on Test Suite:
   with a configuration with 3 variations:
   Ethernet (ID 1), WiFi (ID 2), GPRS (ID 3)
   And we want do following combination Test Cases / Configurations
             |Ethernet | WiFi | GPRS |
   Test Case |---------|------|------|
   TC1 | X | X | |
   TC2 | | X | X |
   TC3 | X | X | X |
   This will generate on testplan_tcversions
   testplan_id, tcversion_id, variation_id
        1 1 1 -> TC1 / Ethernet
        1 1 2 -> TC1 / WiFi
        1 2 2 -> TC2 / WiFi
        1 2 2 -> TC2 / GPRS
        1 3 1 -> TC3 / Ethernet
        1 3 2 -> TC3 / WiFi
        1 3 3 -> TC3 / GPRS
5. At execution time a new filter will be displayed on left frame (that will affect tree
   creation), Configuration Items.
   Due to potential performance issues, IMHO is better to display just ONE TREE at
   a time instead of displaying (as suggested on google document) a firts level
   with all configuration variations.
   May be we can resolve perfomance issues if we can refactor tree creation logic
   in order to create all tree under any variation ONLY on user click.
   Remember that right now on execution we need to create WHOLE test plan tree,
   and we can not use progressive (on request creation).
   With our example we will be able to have 3 different trees.
   When user executes test case he/she does not need to indicate what variation is used
   because is know by system.

6. On reports, must be analized if test plan has a configuration linked, and then all
   algorithm will display results grouped by variation.
7. Other impacts:
   - Also statistic data related to BUILD will be affected by Configuration variation.
   - Some Limits to impose:
     if a test plan is not empty configuration can not be added (linked to test plan)
     if a test plan is not empty and have a configuration link between config and testplan
     can not be removed ?
     Please think about other types of these situations that always have impacts
     on statistics and reports.

8. Assigning test case to testers
   Complexity is not low at User Inteface level, if we want to able to assign different
   variation to different testers (is this feture a Must to have ?).
   We have two choices:
   1. work as done on execution adding a filter with configuration variation
   2. add on every test case a combo with available variations (more processing
      time when creating user interface)

P2 - Proposal based on Custom Fields With special meaning

     this approach has impacts only on reports code, but also offer a different
     approach at user interface (less intuitive?)
     How this can work?
     create a custom field named CF_SPEC_CONFIG that will be use during test case spec
     to choose target configuration variations that has to be used while
     running test case.
     DRAWBACK 1: you can not have different configuration for different test plans
     because once a custom field is linked to a test project will be displayed on ALL test cases

     A second custom field named CF_EXEC_CONFIG with same definition of CF_SPEC_CONFIG
     need to be created. This will be CF displayed and manageable during execution.
     Some logic must be developed to limit set of values that user can choose.
     An example will be much better that lot of words.
     To reproduce first example
     we create CF_SPEC_CONFIG
     label: Target Comm. method
     Type: multiselection list
     Values: 'Ethernet,WiFi,GPRS'
     Enable on: Test Spec
     Display on Test Execution: yes
     Available for: test cases
     we create CF_EXEC_CONFIG
     Label: Comm. method
     Type: list
     Values: 'Ethernet,WiFi,GPRS'
     Enable on: Test Execution
     Available for: test cases

     DRAWBACK 2: duplicated work
     When creating TC1 user will select on Target Comm. method: Ethernet,WiFi
     When creating TC2 user will select on Target Comm. method: WiFi,GPRS
     When creating TC2 user will select on Target Comm. method: Ethernet,WiFi,GPRS
     When going to execute TC1, on Comm. method user will see Ethernet,WiFi,GPRS.
     Not Good, really he/she will be have available just Ethernet,WiFi.
     To do this new code need to be developed, and some hacking needed, because
     there is not an attribute on Cfields table that allow express some kind
     of relationship between Custom Fields.
     Alternative use strings with special meaning on possible values fields.
     Anyway this means some changes on custom field class is needed.
     User has to record an execution result for each variation, i.e.
     user do test of TC1 with Ethernet [FAILED] (choose this value on Comm. method) and save it
     user do test of TC1 with WiFi [PASSED] (choose this value on Comm. method) and save it

     If he/she enable 'Show complete execution history' will see both results.
     On tree TC1 will be colored as PASSED (last exec result).
     DRAWBACK 3: there is no way to understand from tree amount of test cases runned for
                 every variation.
     Reports logic must be changed (in a similar way to done with CF_EXEC_TIME), to
     group results in different way if CF_EXEC_CONFIG exists.
     Thinks this solution is not good, and must be discarded.
tomtom (reporter)
2009-07-15 02:03

This is just a thought. There is another feature request that comes up regularly: Having test case separated from test steps, test data... [^]
Wich I believe is quite standard in validation tools.

I think I read somewhere this will be implemented in testlink but I cant remember where... :D

Then if such thing is implemented,
Actual test case may becomes a "test scenario" which links to
- test steps
- test data
- test configuration...

Writing tests for different configuration means the user only need to copy the test scenario to a new one and assign it to another configuration (via Custom field or whatever).

If test steps are stored in a shared table, updating a test step will update it for every test scenario linked to it. (which was the point of my previous post)

The good point is that as far I know from testlink, It should only affect database and logic under the testcase level.

The bad point is that it is a bigger surgery to TL... with rewrite of every form/report that display test cases and creating logic/GUI for test step.

Again, thank you for your work on TL :) I didn't expect my first post to go that far !
Eloff (reporter)
2009-07-15 15:47

Thanks for your thorough feedback


1. I'm not really sure if configuration is better than platform. But both are better than hardware. (For now I prefer platform because it easier to write)

I don't see what you need two tables for? Do you mean that one configuration/platforms has some settings(details)?

2. Or maybe per project?

3. It will be linked to Test Plans.
This way we can decrease the amount of clutter, by not showing all configs all the time, in later views.

4. Oh, now I see your two table solution. I was thinking of linking 1 or several platforms. Not using a platform would require special logic. If we instead link a "default" in silence this is a none problem.

(Sub-testplans was another idea I had, not relevant to this solution. There is no need for a new term)

The changes to testplan_tc_versions is exactly what I mean. Although I named it platform_id

5. Several trees came to my mind when I was working on "1 platform-1 testplan" idea. Then several trees was an easy way => Not really needed

I think we have some alternatives here:
5.1 Show several trees: One per platform
    |- Platform1
    | |- TestSuite1
    | |- TestCase1
    | |- TestCase2
    |- Platform2
    | |- TestSuite1
    | |- TestCase1
    | |- TestCase2
    |- Platform3
         |- TestSuite1
              |- TestCase1
    Good: All combos visible
    Bad: Performance issue when generating several trees?

5.2 Show all TCs per platform side-by-side in one tree
    |- TestSuite1
         |- TestCase1 - Platform1
         |- TestCase1 - Platform2
         |- TestCase1 - Platform3
         |- TestCase2 - Platform1
         |- TestCase2 - Platform2
    Good: All combos visible
    Bad: Less readable

5.3 Show one tree with every linked TC independent of hardware (like today). Then on the report the user can select platform
    |- TestSuite1
         |- TestCase1
         |- TestCase2
    Good: More forgiving to user. You cannot select the wrong platform from tree. Less clutter in the tree
    Bad: How to show exec status in tree? On tree item can be several platforms.

5.4 Always require a filter by platform, and thus only showing one.
    Good: Shouldn't require a big code change
    Bad: Filtering is not obvious

6. Yep, and maybe filtered.

7. If we always have a default configuration/platform it's possible to add another one (and rename the first).

8. GUI is not really easy to make. The easy solution would be to show one row for every TC/platform. This will unfortunately make the assign view N times larger if having N platforms. The same problem is present when adding TCs to testplan

Assigning on individual platform is not highest priority, but it would be good to see if someone is assigned to the TC (on any platform)

P2 - CF

Yeah, I see how this way would be hard on so many levels
fman (administrator)
2009-07-16 00:34

I will analize new info and give feedback.
Think that with a couple of roeund of reviews, we can define very well what we want and then Elof will be able to start development.
Joakim (reporter)
2009-10-02 18:57

I this feature planned for 1.09?
A lot of our tests are configuration dependent. Linking configurations with test cases will be very useful.
fman (administrator)
2010-02-02 02:36

contribution by Eloff, reviewed and refactored

- Issue History
Date Modified Username Field Change
2009-06-19 19:04 mhavlat New Issue
2009-06-19 21:33 tomtom Note Added: 0007302
2009-06-19 22:20 mhavlat Note Added: 0007303
2009-06-19 22:20 mhavlat Status new => feedback
2009-06-30 16:35 Eloff Note Added: 0007404
2009-07-02 22:08 mhavlat Note Added: 0007432
2009-07-02 23:11 mhavlat Note Added: 0007433
2009-07-04 07:25 mhavlat Note Added: 0007438
2009-07-04 07:27 mhavlat Note Edited: 0007438
2009-07-07 18:52 Eloff Note Added: 0007448
2009-07-08 14:22 Eloff Note Added: 0007450
2009-07-10 23:54 fman Note Added: 0007518
2009-07-11 04:35 Eloff Note Added: 0007521
2009-07-13 21:37 fman Note Added: 0007530
2009-07-15 00:34 fman Note Added: 0007557
2009-07-15 02:03 tomtom Note Added: 0007559
2009-07-15 15:47 Eloff Note Added: 0007565
2009-07-16 00:34 fman Note Added: 0007569
2009-07-17 14:51 mhavlat Assigned To => fman
2009-08-11 19:01 mhavlat Relationship added has duplicate 0002793
2009-10-02 18:57 Joakim Note Added: 0008035
2009-10-02 22:17 fman Summary Reports with respect to a Custom Field (platforms) => Platforms feature
2010-01-11 07:48 mhavlat Status feedback => work in progress
2010-02-02 02:35 fman Status work in progress => assigned
2010-02-02 02:36 fman Note Added: 0008913
2010-02-02 02:36 fman Status assigned => resolved
2010-02-02 02:36 fman Fixed in Version => 1.9 Beta 3 (Next Beta)
2010-02-02 02:36 fman Resolution open => fixed
2010-05-01 20:32 fman Status resolved => closed

Copyright © 2000 - 2020 MantisBT Team
Powered by Mantis Bugtracker