Mantis Bugtracker          
testlink.org

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0003714TestLinkGeneralpublic2010-08-25 13:422017-10-09 15:23
ReporterJulian 
Assigned To 
PriorityimmediateSeveritycrashReproducibilityalways
StatusassignedResolutionopen 
PlatformOSOS Version
Product Version1.9 Beta 5 
Fixed in Version 
Summary0003714: Testlink not usable anymore after cookie size (request header) exceeds server limit
DescriptionAs we use cookies to store user settings like "lastTestPlanForUserID_X", exttable settings and so on, we create pretty much data. if this data stored in the cookie exceeds server limit testlink cannot be used anymore. cookies then have to be deleted in the browser to get testlink running again.
Steps To Reproduceopen pages containing exttables in different test projects.
Additional InformationMessage from Internet Explorer:

Bad Request
Your browser sent a request that this server could not understand.
Size of a request header field exceeds server limit.

In my case limit seems to be 7400 characters
TagsNo tags attached.
Database (MySQL,Postgres,etc)MySQL
BrowserIE, Firefrox
PHP Version
TestCaseID
QA Team - Task Workflow StatusTBD
Attached Files

- Relationships
parent of 0004613closedJulian cookiePrefix needs to be improved for all trees within testlink 
has duplicate 0005534new Testlink web interface stops working - Size of a request header field exceeds server limit. 

-  Notes
(0010938)
Eloff (reporter)
2010-08-25 14:09
edited on: 2010-08-25 14:11

I've thought of some solutions to this (non-prioritized list):

1. Possible workaround:
In short, save state on server via ajax requests instead of cookies
    + cookie size avoided
    - needs server implementation and db tables
    - feels like over-engineered solution
http://www.sencha.com/forum/showthread.php?24970-Buffering-Http-State-Provider [^]


2. Shared states:
A simpler solution is to not save state for every unique table. For example results by failed/blocked/not_run can use same settings
    - we must guarantee to not exceed cookie limit


3. Compress/strip data:
A cookie for a table is very big (>500 byte). Investigate if it is possible to remove data without losing functionality.
    - we must guarantee to not exceed cookie limit

4. No states:
If no other solution is found we have to remove state from ext tables
    + no problem with cookie limit
    - less functionality (user have to live with our default choice)

(0010939)
Eloff (reporter)
2010-08-25 14:18

This can be solved in apache, but I don't think it should. Some users don't run apache, or may not have access to modify the config.
I'm not sure how browsers handle big cookie-data, so this could cause another limit. Preferably the cookies we set should be below 4k in size
(0010941)
Julian (reporter)
2010-08-25 17:58

this one is release critic.
if db tables are required this issue MUST be resolved before beta6 as beta6 MUST contain final db scheme!
In my Eyes we cannot accept this behaviour for a RC this is why a solution has to be found as soon as possible.
(0010942)
fman (administrator)
2010-08-25 18:30

Being a negative (realistic?) guy, I've always thought that pushing the limit trying to save 'the world' in cookies was an error.
Obviously this comment is of NO HELP.
Problem is we do not have enough force now (at least IMHO) to try to solve this before RC ( if we want to maintain release date on october).

We need to understand, what is really the CRITIC data that HAS TO LIVE in cookie.
Would you mind to explain,why are we saving extjs data on cookie?

regards
(0010943)
fman (administrator)
2010-08-25 18:32

>> 4. No states:
>> If no other solution is found we have to remove state from ext tables
>> + no problem with cookie limit
>> - less functionality (user have to live with our default choice)
if this is the simpler and cheaper solution to achieve our release date then this is the choice I woul use now.
With this we can gain time to device a better solution.

(Time to make google do search for us)
(0010947)
Eloff (reporter)
2010-08-25 23:23

@fman
The cookies does not save any critical data. They store the state of various ext components state, like panel open/closed.

Normally this does not take much space, but for ext tables those cookies are big for some reason. They should store info like visible columns, group by column and which column to sort on. I haven't investigated what data is stored, but whatever it is, it is way to big. When we do this for several tables the amount of data explodes
(0010949)
Julian (reporter)
2010-08-26 07:31

as a first solution i added parameter to exttable class that allows to enable storing of table data - default: no data stored.
Like this it is possible to enable data storage for most important tables if necessary.
(0010954)
Eloff (reporter)
2010-08-26 09:33

I just added Ext.ux.JsonCookieProvider; it works like the normal CookieProvider but encodes the state in a more efficient data type. Preliminary testing seem to use about 50% of default format

@Juslian: Do you think we have to store column width? it would be better to only save grouping and sorting columns, as this would decrease space usage drastically.
(0010955)
Julian (reporter)
2010-08-26 10:02

column width is not that important.

focus on: grouping, sorting, (hidden columns)
(0010956)
Eloff (reporter)
2010-08-26 10:49

I committed some code that only store relevant state. Instead of GridPanel use Ext.ux.SlimGridPanel.

Cookie now uses approx 140 byte/table (improved from original >600 byte)
(0010957)
Julian (reporter)
2010-08-26 11:28

great job.
this will allow to enable store feature by default again in my eyes.

Francisco what is your opinion ?
(0010964)
fman (administrator)
2010-08-27 09:10

Ok, with only one MANDATORY condition:
BEFORE releasing 1.9 beta 6 want to have clear documentation inside code
AND a new chapter for dev guide (or a new document is ok) with all Important things to be know for a developer that wants to use EXT-JS on TL.
   Do not want to rely on:
   - external documentation on web, that can disappear,
   - individual knowledge
(0010972)
Eloff (reporter)
2010-08-28 07:16

We should also remove testproject/plan id:s from the cookies, otherwise this will grow over time till it breaks. This will also make the a report look similar no matter what project/tplan you have active.
(0010981)
Julian (reporter)
2010-08-29 00:11

as we found out in chat this noon:

to remove testproject/plan ids it is required to change column ids from idx0,idx1... to idx_platform, idx_priority.
like this it will be possible to use stored data across testprojects.
This will require some code refactoring. (minimum column initalization will not only require title but also index, ...)

how shall we proceed?
(0011419)
fman (administrator)
2010-09-20 17:58

Reminder sent to: Julian

can we set to resolved?
(0011423)
Julian (reporter)
2010-09-20 18:30

Eloff will tell us if he will make further improvements during this week.
Talked to him this morning.
(0011448)
Eloff (reporter)
2010-09-21 20:20

I have now refactored the column names in tables. Instead of idx0, idx1... the column title is used (kind of: non-valid chars are stripped first to not crash in non-english languages).

Also I had to modify the functionality to load state from cookie. For instance if I sort by platform and then change to a test plan without platforms it should not try to sort on a non-existing column obviously. Now the grid uses the default if the save state is non-applicable.

You may have to delete old cookies for this to work.
(0011449)
Julian (reporter)
2010-09-21 21:18
edited on: 2010-09-22 09:08

table ids have been adjusted to reduce cookie size.
table id is the same no matter which test project or test plan.
this is now possible with the refactoring mentioned by eloff in the comment above.

next step is to decide if we can now store all data.

exttable is used on:
1. reqOverview.php
[reqSearch.php - no data stored]
[reqSpecSearch.php - no data stored]
2. resultsBugs.php
3. metricsDashboard.php
4. testCasesWithCF.php
5. resultsTC.php
6. resultsReqs.php
7. resultsByTesterPerBuild.php
8. testPlanWithCF.php
9. resultsByStatus.php
10. testCasesWithoutTester.php
11. freeTestCases.php
12. tcAssignedToUser.php
[tcSearch.php - no data stored]
-------
12 table states need to be stored in worst case

there are stored about 280-470 bytes per table [280 = ys-tl_table_test_cases_not_assigned_to_any_test_plan, 470 = ys-tl_table_results_reqs]

max bytes till error occurs 7400 bytes -> 15 - 26 table states can be stored [my limit]

max bytes till error occurs 4000 bytes -> 8 - 14 table states can be stored [eloffs limit]

-> we need a decision.

After calling all tables to store state i had a cookie with a size of 4639 Bytes.
every state of a tree (test spec, req spec) takes about 150 Bytes per project

(0011450)
fman (administrator)
2010-09-22 07:09

>> I have now refactored the column names in tables. Instead of idx0, idx1... the >> column title is used (kind of: non-valid chars are stripped first to not crash >> in non-english languages).
1.anyone out there can explain whay this was needed ?
2. why we need to use localized titles instead of label key (i.e. test_status instead of 'Estado de las pruebas').
Please explain

remember that: Knowledge is power => we need to understand choices
(0011451)
Eloff (reporter)
2010-09-22 07:21

@fman:
1. Say you sort by priority in one report (idx3), and then view another table where priority is idx2 => it will not be sorted in priority. if the column names makes sense it is possible to keep a consistent state.

2. Because it requires less work :). Otherwise we need to set a label key for every column all through testink. This is possible but the code will become larger.
(0011452)
Eloff (reporter)
2010-09-22 07:24

Regarding point 2 from above:
To construct the columns of a table we always call lang_get() to localize headers. It is possible to let the table class do this.

Instead of:
$columns[] = array('title' => lang_get('platform'), 'width' => 60);

we can do:
$columns[] = array('title_key' => 'platform', 'width' => 60);

This also gives a non-localized label to use as column id
(0011461)
Julian (reporter)
2010-09-22 13:49
edited on: 2010-09-22 13:49

i did some more researches regarding limit for cookie size:
http://www.filonov.com/blog/2009/11/07/error-400-size-of-a-request-header-field-exceeds-server-limit/ [^]

-> for apache default value = 8190 => with apache it is possible to store data of all 12 tables (ca. 4700 Bytes) + 23 tree states (ca 150 Bytes per tree) before user will encounter this error.

(0012651)
Julian (reporter)
2010-11-18 13:59

for 1.9 we found an acceptable but no perfect solution.

what shall we do for 2.0 ?
i got the feeling that the amount of tables will grow steady.
so for 2.0 we need to store tablestate on database, because all other mentioned solutions contain a compromise.

Erik, what next steps do you recommend?
(0014789)
scmme (reporter)
2011-05-04 16:20

This problem is still present in Testlink 1.9.2 final. We have had multiple instances of users experiencing the problem.
(0014790)
Julian (reporter)
2011-05-04 16:26

feel free to fix. your contribution is welcome.
(0020110)
kenden (reporter)
2013-11-22 10:47

Reproduced on 1.9.9, on Google Chrome I get:
400 Bad Request

Request Header Or Cookie Too Large

(sorry, I don't find a wait to Star the bug or click "this bug also affects me" so I comment)
(0025957)
one-testlink (reporter)
2017-02-02 16:07

Hey guys,

We have the same problem too. Working productively with Apache mod_proxy as reverse proxy, which has a limit on 8190 as mentioned above.

Have you thought about using a localStorage-based implementation? We use that for ExtJs based applications quite successfully... If you're interested I can try to put up a patch.

Nicola
(0025958)
fman (administrator)
2017-02-02 21:48

Any help is welcomed.
please create a repo con github with your implementation and share it
(0025961)
one-testlink (reporter)
2017-02-03 16:44

Following is an excerpt of cookies as an example of a user that could not access the web server anymore because the HTTP request header was too big:

ys-add_remove_tc_tplan_id_18645_ext-comp-1001 a%3As%253A/1
ys-assignReqs_tproject_id_1_ext-comp-1001 a%3As%253A/1
ys-edit_tc_tproject_id_1_ext-comp-1001 a%3As%253A/1/10961/13213/149 (...) /13306
ys-tc_exec_assignment_tplan_id_18645_ext-comp-1001 a%3As%253A/1/10961/13213/14833
ys-test_exec_build_id_71_platform_id_13_ext-comp-1001 a%3As%253A/1/10961 (...) 6262
ys-test_exec_build_id_73_platform_id_13_ext-comp-1001 a%3As%253A/1/10961
ys-test_exec_build_id_73_platform_id_7_ext-comp-1001 a%3As%253A/1/10961/13213/14833
ys-testplan_tplan_id_18645_ext-comp-1001 a%3As%253A/1/10961

The entries with "(...)" were very big.

I investigated the issue and replaced all JavaScript references to the generic Cookie state provider with a localStorage provider. My proposition for a solution can be found here: https://github.com/one-github/testlink-code [^]

Could someone with more knowledge of the code base please review the pushed changes and also assess whether the large cookies mentioned above would be really a thing of the past ie. the functionality does work as intended? Thanks.

Nicola
(0025962)
one-testlink (reporter)
2017-02-03 16:51

I decided against a fallback to the cookie storage for reasons of simplicity and unnecessary bulk. All current browser generations support the HTML5 localStorage API: http://caniuse.com/#search=localstorage [^]
(0026069)
one-testlink (reporter)
2017-03-06 16:18

@fman, is there a target version defined yet for this issue? Could it be taken into 1.9.17? When will 1.9.17 come out? Thanks!
(0026071)
fman (administrator)
2017-03-06 16:46

NO ETA for 1.9.17, but this will be included

- Issue History
Date Modified Username Field Change
2010-08-25 13:42 Julian New Issue
2010-08-25 14:09 Eloff Note Added: 0010938
2010-08-25 14:11 Eloff Note Edited: 0010938 View Revisions
2010-08-25 14:18 Eloff Note Added: 0010939
2010-08-25 17:58 Julian Note Added: 0010941
2010-08-25 17:58 Julian Priority high => immediate
2010-08-25 18:30 fman Note Added: 0010942
2010-08-25 18:32 fman Note Added: 0010943
2010-08-25 23:23 Eloff Note Added: 0010947
2010-08-26 07:31 Julian Note Added: 0010949
2010-08-26 09:33 Eloff Note Added: 0010954
2010-08-26 10:02 Julian Note Added: 0010955
2010-08-26 10:49 Eloff Note Added: 0010956
2010-08-26 11:28 Julian Note Added: 0010957
2010-08-27 09:10 fman Note Added: 0010964
2010-08-27 09:23 Julian Note View State: 0010954: public
2010-08-27 09:23 Julian Note View State: 0010955: public
2010-08-27 09:23 Julian Note View State: 0010956: public
2010-08-27 09:23 Julian Note View State: 0010957: public
2010-08-28 07:16 Eloff Note Added: 0010972
2010-08-29 00:11 Julian Note Added: 0010981
2010-09-20 17:58 fman Note Added: 0011419
2010-09-20 18:30 Julian Note Added: 0011423
2010-09-21 19:25 Eloff Assigned To => Eloff
2010-09-21 19:25 Eloff Status new => assigned
2010-09-21 20:20 Eloff Note Added: 0011448
2010-09-21 21:18 Julian Note Added: 0011449
2010-09-22 07:09 fman Note Added: 0011450
2010-09-22 07:21 Eloff Note Added: 0011451
2010-09-22 07:24 Eloff Note Added: 0011452
2010-09-22 08:55 Julian Note Edited: 0011449 View Revisions
2010-09-22 09:08 Julian Note Edited: 0011449 View Revisions
2010-09-22 13:49 Julian Note Added: 0011461
2010-09-22 13:49 Julian Note Edited: 0011461 View Revisions
2010-11-18 13:59 Julian Note Added: 0012651
2011-05-04 16:20 scmme Note Added: 0014789
2011-05-04 16:26 Julian Note Added: 0014790
2011-06-16 05:30 Julian Relationship added parent of 0004613
2012-08-16 12:34 fman Assigned To Eloff =>
2013-03-17 16:31 fman Relationship added has duplicate 0005534
2013-11-22 10:47 kenden Note Added: 0020110
2013-11-22 13:49 fman QA Team - Task Workflow Status => TBD
2017-02-02 16:07 one-testlink Note Added: 0025957
2017-02-02 21:48 fman Note Added: 0025958
2017-02-03 16:44 one-testlink Note Added: 0025961
2017-02-03 16:51 one-testlink Note Added: 0025962
2017-03-06 16:18 one-testlink Note Added: 0026069
2017-03-06 16:46 fman Note Added: 0026071



Copyright © 2000 - 2017 MantisBT Team
Powered by Mantis Bugtracker