Mantis Bugtracker          
testlink.org

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0002919TestLinkReportspublic2009-10-26 23:472013-05-26 20:09
Reportertahmed 
Assigned Toasimon 
PrioritynormalSeveritymajorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version1.8.4 
Fixed in Version1.9.3 (2011 Q3 - bug fixing) 
Summary0002919: Number of test cases altered after build is executed
DescriptionIf you add additional test cases to a test plan, for which a build has been executed, total number of test cases goes up and metrics become inaccurate.
TagsNo tags attached.
Database (MySQL,Postgres,etc)
BrowserIE, FF
PHP Version
TestCaseID
QA Team - Task Workflow Status
Attached Filespng file icon tc_total.PNG [^] (36,052 bytes) 2010-01-22 23:14


png file icon tc_total2.PNG [^] (38,799 bytes) 2010-01-22 23:14

- Relationships

-  Notes
(0008140)
fman (administrator)
2009-10-27 01:53

this will be fixed on next releases
(0008146)
tahmed (reporter)
2009-10-27 02:04

will be next release or another future release?
(0008166)
fman (administrator)
2009-10-28 21:50

planned on 2.0
(0008167)
Amaradana (reporter)
2009-10-28 21:57

This is just my opnion.

2.0 is long, as metrics are important always, if tool is good and metrics are not correct then how would the mangement think to implement this?

I am not saying that please fix in next version, but i am only asking to consider, metrics related defects as high priority rather than going for other new features.

Please ignore the above message if you still think this is not worth. I only mentioned as this is how metrics plays a major role and decide the quality.
(0008169)
tahmed (reporter)
2009-10-28 22:18

I agree - This is a serious bug, and as a user, I would have liked it fixed sooner than 2.0.
(0008668)
goosetea1 (reporter)
2010-01-13 22:14

I second the motion. As I was forcing my company to use Testlink one of the first questions that I had to answer was this inaccuracy in the results and that's why I still run testlink on my computer and it is not officially installed ^(
(0008671)
Jgaudreau (reporter)
2010-01-13 23:29

I aggree with Amaradana, tahmed and goosetea1.
The metrics is one of the Main features that makes TestLink very interesting for our company. If the Metrics do not work as intended, it would be a big negative point for our communications with external clients.
(0008672)
fman (administrator)
2010-01-13 23:32

We are open to any kind of contribution, then if Amaranda, or tahmed or gooseteal have time to take in charge this issue to develop with our support it would be great.
Right now we are working on new features for 1.9, and on migration from 1.8.x (not a minor thing), then do not have enough development hours to fix it.
Hope you understand.
Meanwhile you can find a workaround to get right metrics.
(0008675)
goosetea1 (reporter)
2010-01-13 23:43

For now as a workaround I create different versions of test plans if the customer is keen to have precise stats or save results before every change in tc numbers and adjust figures manually in excel report.

But it is not a very good solution 'cause one has to reassign all cases for every Test Plan and having a team of young testers under command means adding and removing test cases on the daily (build) basis, which might end in having zillion test plans :)

Anyway, we will wait eagerly for this to be developed. Can't help you unfortunately. Don't know php.
(0008690)
qabe (reporter)
2010-01-15 03:34

1st, I agree metrics are critical information for a serious tool, this jeopardize the fact we were going to use Test Link for a bigger project, since we need to share metrics between multiple partners (companies), we will look like inexperienced fools if we use a buggy tool, QA knows better.

2nd, we are not programmers, we are QA staff, testers or managers; so suggesting us that if we wan't an important bugfix, we should contribute ourselves, is a very bad approach. We already invest time in reporting the issues to the community. It only shows us, when we really need help, that we can't expect any.

3rd, If you want us to use the new features of version 1.9, we must continue using the tool, but if we have too many issues with the tool, we will not see version 1.9 if we have to migrate to some other tool who supports our concerns.

Currently, it looks like the Test Link development is done the same way as a quick development commercial products, features takes priority before stability and bugfixing, I would have expect Test Link priorities to be very close to QA related priorities.

I hope you understand.

This ticket should be escalated, priority wise.
(0008693)
fman (administrator)
2010-01-15 15:45

@qabe:
I really do not want to start here a long discussiion that will make all us waste our time.
Just few words:
we have done our best on last 5 years to support community, and we will continue sometimes making some people happy and sometimes not.
(0008697)
Julian (reporter)
2010-01-15 17:23

i have to comment your 2nd point qabe:

Testlink is non-commercial, that means all developers are volunteers.
Reporting to community takes not much time compared to the time that is necessary to implement a bugfix/feature.
If testlink is the tool of your choice and you want to customize it to your companies needs why not contribute? the money your company has to spend for those contributions will be much less than buying a commercial tool. My experience is that even a commercial tool for lots of money will not fit all your needs and every change must be done by the seller for again lots of money.
Please understand that offending developers or development processes will not improve anything. Opensource projects can only grow (fast) if there is community participation in development.
I ask you to stop further discussion here to not lose track of this issue.
what we are asking for is constructive criticism.
(0008698)
goosetea1 (reporter)
2010-01-15 17:36

Guys (Developers)
We appreciate what you do and we obviously use Testlink because it is good tool. Thank you, honestly.
But I guess what we are trying to say as people who use the tool on daily basis (and this topic was brought up before): we might not need some of the new feature 'cause we have problems in the stats/metrics and as insignificant as it sounds it makes us unable to use testlink on some projects. What I don't understand is why this problem wasn't addressed before and planned for, say, release 1.9. On one hand you want to make Testlink better and develop new features on the other hand you don't seem to listen to people who use the tool and cry over the broken metrics.
I installed 1.9 and I really like it, so much changes that will make working with Testlink even more comfortable, but I imagine telling bosses: Hey, look new version is fantastic! and them saying what about that bug with metrics?
thanks for the understanding and your work.
(0008699)
Julian (reporter)
2010-01-15 17:50
edited on: 2010-01-15 17:51

you have to understand that not every wrong behaviour is just a easy-to-fix bug - this issue seems to be more a result of the way testlink handles the relationship between test cases and test plans. for non-developers those constraints are maybe not that obvious, but if my feeling about this is right the effort to fix this will not be small.
And i again ask to stop general discussion here.
Whether you accept the statement of fman that it is planned for 2.0 or you share your solution for this problem and you will see it in 1.9 or i will have to make this issue invisible for non-developers.

(0008702)
qabe (reporter)
2010-01-16 00:48

ok please fix when you can, it is important to us.

also, how could I know if this problem only is from 1.8.4 and above?

I currently run 1.8.2, if this problem was always there, I could update to 1.8.5, otherwise if the problem is only from 1.8.4, I maybe should stick with 1.8.2 if 1.8.3 doesn't bring me much for my needs.

best regards
qabe
(0008703)
goosetea1 (reporter)
2010-01-16 00:53

it was in 1.8.2
you can upgrade to 1.8.5
(0008704)
Amaradana (reporter)
2010-01-16 15:16
edited on: 2010-01-16 19:31

Guys, Understand knew this was a problem, but in one way or the other this is correct but not accurate, will tell you why

1. When test cases gets added to test plan all the build results, test suites results get changed, the reason, it shows for whole set, WE QA people always look for that build wise and the test suite wise we take results, i cant affend the discussion or defend the discussion, as it depends on the comapny and ur process,
believe me some compaines look for whole set of progress,
Stop blaming development team to fix this we can contribute if we have hands on PHP. being a tester there is no rule that we are not supposed to develop, even i cuztomized the product though i donno PHP.

WorkAROUND : We have QUERY METRICS OK. USE THEM AND WAIT TILL THIS ISSUE FIXED

(0008765)
maxel77 (reporter)
2010-01-22 23:13

..."We have QUERY METRICS OK"
-> Query matrics are not OK

tahmed:
.."If you add additional test cases to a test plan, for which a build has been executed, total number of test cases goes up and metrics become inaccurate"

->It's true

This is a serious error and should be repaired
How to keep statistics when the results are inaccurate

see printscreens
(0008770)
Amaradana (reporter)
2010-01-23 00:25
edited on: 2010-01-23 11:01

Maxel, i knew that this is a serious bug, u always have workaround. Query metrics is the only solution that can solve atleast. but u need to prepare the excel sheet seperately to track all the results and calculate. as i mentioned before believe me most companies follow these reports as well as i mentioned above.( previous + current results = total build status for a release.)

not only this any metrics related defects had to be given priority in related to test case management tool in our terms. I am mentioning again this is a open source tool and you cannot demand some one to fix this with higher priority. my analysis on this big it is not so easy to fix, i think this can be fixed in a step by step. as currently when a test case is tied up to test plan, it is not mapped to the build and test plan id is tied up to the builds i guess...

You have also another work around, say create a test plan for the build, you can solve your problem. and if you still want an aggregate report we have some reports generated check the tracker for this. Its all about the custimization.

Your most welcome if you can contribute on this issue and help us.

(0008772)
fman (administrator)
2010-01-23 03:17

on 1.9 we have added some date fields that we think we can help. example date when test case version has been added to test plan. For build release date and closure date. We will try to use these to fix this problem. Meanwhile more words here regarding importance of fixing it, does not help.
(0015741)
asimon (developer)
2011-08-23 08:50

I think we can call this issue fixed. Because of the way we handle the build assignments in TestLink 1.9.x, this old problem does not exist anymore.

There is no "TC Total" number anymore in these metrics. We only display the number of assigned testcases per build, which does not change for older builds automatically like it was the case before. So metrics do not become inaccurate over time anymore, no matter what happens in later builds.

- Issue History
Date Modified Username Field Change
2009-10-26 23:47 tahmed New Issue
2009-10-26 23:47 tahmed Browser => IE, FF
2009-10-27 01:53 fman Note Added: 0008140
2009-10-27 02:04 tahmed Note Added: 0008146
2009-10-28 03:42 amitkhullar Product Version 1.8 RC 4 => 1.8.4
2009-10-28 21:50 fman Note Added: 0008166
2009-10-28 21:57 Amaradana Note Added: 0008167
2009-10-28 22:18 tahmed Note Added: 0008169
2010-01-13 22:14 goosetea1 Note Added: 0008668
2010-01-13 23:29 Jgaudreau Note Added: 0008671
2010-01-13 23:32 fman Note Added: 0008672
2010-01-13 23:43 goosetea1 Note Added: 0008675
2010-01-15 03:34 qabe Note Added: 0008690
2010-01-15 15:45 fman Note Added: 0008693
2010-01-15 17:23 Julian Note Added: 0008697
2010-01-15 17:36 goosetea1 Note Added: 0008698
2010-01-15 17:50 Julian Note Added: 0008699
2010-01-15 17:50 Julian Note Edited: 0008699
2010-01-15 17:51 Julian Note Edited: 0008699
2010-01-16 00:48 qabe Note Added: 0008702
2010-01-16 00:53 goosetea1 Note Added: 0008703
2010-01-16 15:16 Amaradana Note Added: 0008704
2010-01-16 15:19 Amaradana Note Edited: 0008704
2010-01-16 19:31 Amaradana Note Edited: 0008704
2010-01-22 23:13 maxel77 Note Added: 0008765
2010-01-22 23:14 maxel77 File Added: tc_total.PNG
2010-01-22 23:14 maxel77 File Added: tc_total2.PNG
2010-01-23 00:25 Amaradana Note Added: 0008770
2010-01-23 03:17 fman Note Added: 0008772
2010-01-23 11:01 Amaradana Note Edited: 0008770
2010-06-05 09:49 fman Assigned To => asimon
2010-06-05 09:49 fman Status new => assigned
2011-08-23 08:26 asimon Summary Number fo test cases altered after build is executed => Number of test cases altered after build is executed
2011-08-23 08:50 asimon Note Added: 0015741
2011-08-23 08:50 asimon Status assigned => resolved
2011-08-23 08:50 asimon Fixed in Version => 1.9.3 (2011 Q3 - bug fixing)
2011-08-23 08:50 asimon Resolution open => fixed
2013-05-26 20:09 fman Status resolved => closed



Copyright © 2000 - 2018 MantisBT Team
Powered by Mantis Bugtracker