MantisBT - TestLink
View Issue Details
0003714TestLinkGeneralpublic2010-08-25 13:422018-05-08 13:25
Julian 
 
immediatecrashalways
assignedopen 
1.9 Beta 5 
 
MySQL
IE, Firefrox
TBD
0003714: Testlink not usable anymore after cookie size (request header) exceeds server limit
As we use cookies to store user settings like "lastTestPlanForUserID_X", exttable settings and so on, we create pretty much data. if this data stored in the cookie exceeds server limit testlink cannot be used anymore. cookies then have to be deleted in the browser to get testlink running again.
open pages containing exttables in different test projects.
Message from Internet Explorer:

Bad Request
Your browser sent a request that this server could not understand.
Size of a request header field exceeds server limit.

In my case limit seems to be 7400 characters
No tags attached.
parent of 0004613closed Julian cookiePrefix needs to be improved for all trees within testlink 
has duplicate 0005534new  Testlink web interface stops working - Size of a request header field exceeds server limit. 
jpg tl-bug.JPG (58,686) 2018-04-10 12:16
http://mantis.testlink.org/file_download.php?file_id=4907&type=bug
jpg
Issue History
2010-08-25 13:42JulianNew Issue
2010-08-25 14:09EloffNote Added: 0010938
2010-08-25 14:11EloffNote Edited: 0010938bug_revision_view_page.php?bugnote_id=10938#r245
2010-08-25 14:18EloffNote Added: 0010939
2010-08-25 17:58JulianNote Added: 0010941
2010-08-25 17:58JulianPriorityhigh => immediate
2010-08-25 18:30fmanNote Added: 0010942
2010-08-25 18:32fmanNote Added: 0010943
2010-08-25 23:23EloffNote Added: 0010947
2010-08-26 07:31JulianNote Added: 0010949
2010-08-26 09:33EloffNote Added: 0010954
2010-08-26 10:02JulianNote Added: 0010955
2010-08-26 10:49EloffNote Added: 0010956
2010-08-26 11:28JulianNote Added: 0010957
2010-08-27 09:10fmanNote Added: 0010964
2010-08-27 09:23JulianNote View State: 0010954: public
2010-08-27 09:23JulianNote View State: 0010955: public
2010-08-27 09:23JulianNote View State: 0010956: public
2010-08-27 09:23JulianNote View State: 0010957: public
2010-08-28 07:16EloffNote Added: 0010972
2010-08-29 00:11JulianNote Added: 0010981
2010-09-20 17:58fmanNote Added: 0011419
2010-09-20 18:30JulianNote Added: 0011423
2010-09-21 19:25EloffAssigned To => Eloff
2010-09-21 19:25EloffStatusnew => assigned
2010-09-21 20:20EloffNote Added: 0011448
2010-09-21 21:18JulianNote Added: 0011449
2010-09-22 07:09fmanNote Added: 0011450
2010-09-22 07:21EloffNote Added: 0011451
2010-09-22 07:24EloffNote Added: 0011452
2010-09-22 08:55JulianNote Edited: 0011449bug_revision_view_page.php?bugnote_id=11449#r346
2010-09-22 09:08JulianNote Edited: 0011449bug_revision_view_page.php?bugnote_id=11449#r347
2010-09-22 13:49JulianNote Added: 0011461
2010-09-22 13:49JulianNote Edited: 0011461bug_revision_view_page.php?bugnote_id=11461#r349
2010-11-18 13:59JulianNote Added: 0012651
2011-05-04 16:20scmmeNote Added: 0014789
2011-05-04 16:26JulianNote Added: 0014790
2011-06-16 05:30JulianRelationship addedparent of 0004613
2012-08-16 12:34fmanAssigned ToEloff =>
2013-03-17 16:31fmanRelationship addedhas duplicate 0005534
2013-11-22 10:47kendenNote Added: 0020110
2013-11-22 13:49fmanQA Team - Task Workflow Status => TBD
2017-02-02 16:07one-testlinkNote Added: 0025957
2017-02-02 21:48fmanNote Added: 0025958
2017-02-03 16:44one-testlinkNote Added: 0025961
2017-02-03 16:51one-testlinkNote Added: 0025962
2017-03-06 16:18one-testlinkNote Added: 0026069
2017-03-06 16:46fmanNote Added: 0026071
2018-02-22 15:07one-testlinkNote Added: 0027223
2018-04-10 12:15hughkayNote Added: 0027282
2018-04-10 12:16hughkayFile Added: tl-bug.JPG
2018-05-08 11:40stef-lhmNote Added: 0027432
2018-05-08 11:59one-testlinkNote Added: 0027433
2018-05-08 12:06one-testlinkNote Added: 0027434
2018-05-08 12:18hughkayNote Added: 0027435
2018-05-08 12:29stef-lhmNote Added: 0027436
2018-05-08 12:35one-testlinkNote Added: 0027437
2018-05-08 12:41one-testlinkNote Added: 0027438
2018-05-08 13:25stef-lhmNote Added: 0027439

Notes
(0010938)
Eloff   
2010-08-25 14:09   
(edited on: 2010-08-25 14:11)
I've thought of some solutions to this (non-prioritized list):

1. Possible workaround:
In short, save state on server via ajax requests instead of cookies
    + cookie size avoided
    - needs server implementation and db tables
    - feels like over-engineered solution
http://www.sencha.com/forum/showthread.php?24970-Buffering-Http-State-Provider [^]


2. Shared states:
A simpler solution is to not save state for every unique table. For example results by failed/blocked/not_run can use same settings
    - we must guarantee to not exceed cookie limit


3. Compress/strip data:
A cookie for a table is very big (>500 byte). Investigate if it is possible to remove data without losing functionality.
    - we must guarantee to not exceed cookie limit

4. No states:
If no other solution is found we have to remove state from ext tables
    + no problem with cookie limit
    - less functionality (user have to live with our default choice)

(0010939)
Eloff   
2010-08-25 14:18   
This can be solved in apache, but I don't think it should. Some users don't run apache, or may not have access to modify the config.
I'm not sure how browsers handle big cookie-data, so this could cause another limit. Preferably the cookies we set should be below 4k in size
(0010941)
Julian   
2010-08-25 17:58   
this one is release critic.
if db tables are required this issue MUST be resolved before beta6 as beta6 MUST contain final db scheme!
In my Eyes we cannot accept this behaviour for a RC this is why a solution has to be found as soon as possible.
(0010942)
fman   
2010-08-25 18:30   
Being a negative (realistic?) guy, I've always thought that pushing the limit trying to save 'the world' in cookies was an error.
Obviously this comment is of NO HELP.
Problem is we do not have enough force now (at least IMHO) to try to solve this before RC ( if we want to maintain release date on october).

We need to understand, what is really the CRITIC data that HAS TO LIVE in cookie.
Would you mind to explain,why are we saving extjs data on cookie?

regards
(0010943)
fman   
2010-08-25 18:32   
>> 4. No states:
>> If no other solution is found we have to remove state from ext tables
>> + no problem with cookie limit
>> - less functionality (user have to live with our default choice)
if this is the simpler and cheaper solution to achieve our release date then this is the choice I woul use now.
With this we can gain time to device a better solution.

(Time to make google do search for us)
(0010947)
Eloff   
2010-08-25 23:23   
@fman
The cookies does not save any critical data. They store the state of various ext components state, like panel open/closed.

Normally this does not take much space, but for ext tables those cookies are big for some reason. They should store info like visible columns, group by column and which column to sort on. I haven't investigated what data is stored, but whatever it is, it is way to big. When we do this for several tables the amount of data explodes
(0010949)
Julian   
2010-08-26 07:31   
as a first solution i added parameter to exttable class that allows to enable storing of table data - default: no data stored.
Like this it is possible to enable data storage for most important tables if necessary.
(0010954)
Eloff   
2010-08-26 09:33   
I just added Ext.ux.JsonCookieProvider; it works like the normal CookieProvider but encodes the state in a more efficient data type. Preliminary testing seem to use about 50% of default format

@Juslian: Do you think we have to store column width? it would be better to only save grouping and sorting columns, as this would decrease space usage drastically.
(0010955)
Julian   
2010-08-26 10:02   
column width is not that important.

focus on: grouping, sorting, (hidden columns)
(0010956)
Eloff   
2010-08-26 10:49   
I committed some code that only store relevant state. Instead of GridPanel use Ext.ux.SlimGridPanel.

Cookie now uses approx 140 byte/table (improved from original >600 byte)
(0010957)
Julian   
2010-08-26 11:28   
great job.
this will allow to enable store feature by default again in my eyes.

Francisco what is your opinion ?
(0010964)
fman   
2010-08-27 09:10   
Ok, with only one MANDATORY condition:
BEFORE releasing 1.9 beta 6 want to have clear documentation inside code
AND a new chapter for dev guide (or a new document is ok) with all Important things to be know for a developer that wants to use EXT-JS on TL.
   Do not want to rely on:
   - external documentation on web, that can disappear,
   - individual knowledge
(0010972)
Eloff   
2010-08-28 07:16   
We should also remove testproject/plan id:s from the cookies, otherwise this will grow over time till it breaks. This will also make the a report look similar no matter what project/tplan you have active.
(0010981)
Julian   
2010-08-29 00:11   
as we found out in chat this noon:

to remove testproject/plan ids it is required to change column ids from idx0,idx1... to idx_platform, idx_priority.
like this it will be possible to use stored data across testprojects.
This will require some code refactoring. (minimum column initalization will not only require title but also index, ...)

how shall we proceed?
(0011419)
fman   
2010-09-20 17:58   
Reminder sent to: Julian

can we set to resolved?
(0011423)
Julian   
2010-09-20 18:30   
Eloff will tell us if he will make further improvements during this week.
Talked to him this morning.
(0011448)
Eloff   
2010-09-21 20:20   
I have now refactored the column names in tables. Instead of idx0, idx1... the column title is used (kind of: non-valid chars are stripped first to not crash in non-english languages).

Also I had to modify the functionality to load state from cookie. For instance if I sort by platform and then change to a test plan without platforms it should not try to sort on a non-existing column obviously. Now the grid uses the default if the save state is non-applicable.

You may have to delete old cookies for this to work.
(0011449)
Julian   
2010-09-21 21:18   
(edited on: 2010-09-22 09:08)
table ids have been adjusted to reduce cookie size.
table id is the same no matter which test project or test plan.
this is now possible with the refactoring mentioned by eloff in the comment above.

next step is to decide if we can now store all data.

exttable is used on:
1. reqOverview.php
[reqSearch.php - no data stored]
[reqSpecSearch.php - no data stored]
2. resultsBugs.php
3. metricsDashboard.php
4. testCasesWithCF.php
5. resultsTC.php
6. resultsReqs.php
7. resultsByTesterPerBuild.php
8. testPlanWithCF.php
9. resultsByStatus.php
10. testCasesWithoutTester.php
11. freeTestCases.php
12. tcAssignedToUser.php
[tcSearch.php - no data stored]
-------
12 table states need to be stored in worst case

there are stored about 280-470 bytes per table [280 = ys-tl_table_test_cases_not_assigned_to_any_test_plan, 470 = ys-tl_table_results_reqs]

max bytes till error occurs 7400 bytes -> 15 - 26 table states can be stored [my limit]

max bytes till error occurs 4000 bytes -> 8 - 14 table states can be stored [eloffs limit]

-> we need a decision.

After calling all tables to store state i had a cookie with a size of 4639 Bytes.
every state of a tree (test spec, req spec) takes about 150 Bytes per project

(0011450)
fman   
2010-09-22 07:09   
>> I have now refactored the column names in tables. Instead of idx0, idx1... the >> column title is used (kind of: non-valid chars are stripped first to not crash >> in non-english languages).
1.anyone out there can explain whay this was needed ?
2. why we need to use localized titles instead of label key (i.e. test_status instead of 'Estado de las pruebas').
Please explain

remember that: Knowledge is power => we need to understand choices
(0011451)
Eloff   
2010-09-22 07:21   
@fman:
1. Say you sort by priority in one report (idx3), and then view another table where priority is idx2 => it will not be sorted in priority. if the column names makes sense it is possible to keep a consistent state.

2. Because it requires less work :). Otherwise we need to set a label key for every column all through testink. This is possible but the code will become larger.
(0011452)
Eloff   
2010-09-22 07:24   
Regarding point 2 from above:
To construct the columns of a table we always call lang_get() to localize headers. It is possible to let the table class do this.

Instead of:
$columns[] = array('title' => lang_get('platform'), 'width' => 60);

we can do:
$columns[] = array('title_key' => 'platform', 'width' => 60);

This also gives a non-localized label to use as column id
(0011461)
Julian   
2010-09-22 13:49   
i did some more researches regarding limit for cookie size:
http://www.filonov.com/blog/2009/11/07/error-400-size-of-a-request-header-field-exceeds-server-limit/ [^]

-> for apache default value = 8190 => with apache it is possible to store data of all 12 tables (ca. 4700 Bytes) + 23 tree states (ca 150 Bytes per tree) before user will encounter this error.

(0012651)
Julian   
2010-11-18 13:59   
for 1.9 we found an acceptable but no perfect solution.

what shall we do for 2.0 ?
i got the feeling that the amount of tables will grow steady.
so for 2.0 we need to store tablestate on database, because all other mentioned solutions contain a compromise.

Erik, what next steps do you recommend?
(0014789)
scmme   
2011-05-04 16:20   
This problem is still present in Testlink 1.9.2 final. We have had multiple instances of users experiencing the problem.
(0014790)
Julian   
2011-05-04 16:26   
feel free to fix. your contribution is welcome.
(0020110)
kenden   
2013-11-22 10:47   
Reproduced on 1.9.9, on Google Chrome I get:
400 Bad Request

Request Header Or Cookie Too Large

(sorry, I don't find a wait to Star the bug or click "this bug also affects me" so I comment)
(0025957)
one-testlink   
2017-02-02 16:07   
Hey guys,

We have the same problem too. Working productively with Apache mod_proxy as reverse proxy, which has a limit on 8190 as mentioned above.

Have you thought about using a localStorage-based implementation? We use that for ExtJs based applications quite successfully... If you're interested I can try to put up a patch.

Nicola
(0025958)
fman   
2017-02-02 21:48   
Any help is welcomed.
please create a repo con github with your implementation and share it
(0025961)
one-testlink   
2017-02-03 16:44   
Following is an excerpt of cookies as an example of a user that could not access the web server anymore because the HTTP request header was too big:

ys-add_remove_tc_tplan_id_18645_ext-comp-1001 a%3As%253A/1
ys-assignReqs_tproject_id_1_ext-comp-1001 a%3As%253A/1
ys-edit_tc_tproject_id_1_ext-comp-1001 a%3As%253A/1/10961/13213/149 (...) /13306
ys-tc_exec_assignment_tplan_id_18645_ext-comp-1001 a%3As%253A/1/10961/13213/14833
ys-test_exec_build_id_71_platform_id_13_ext-comp-1001 a%3As%253A/1/10961 (...) 6262
ys-test_exec_build_id_73_platform_id_13_ext-comp-1001 a%3As%253A/1/10961
ys-test_exec_build_id_73_platform_id_7_ext-comp-1001 a%3As%253A/1/10961/13213/14833
ys-testplan_tplan_id_18645_ext-comp-1001 a%3As%253A/1/10961

The entries with "(...)" were very big.

I investigated the issue and replaced all JavaScript references to the generic Cookie state provider with a localStorage provider. My proposition for a solution can be found here: https://github.com/one-github/testlink-code [^]

Could someone with more knowledge of the code base please review the pushed changes and also assess whether the large cookies mentioned above would be really a thing of the past ie. the functionality does work as intended? Thanks.

Nicola
(0025962)
one-testlink   
2017-02-03 16:51   
I decided against a fallback to the cookie storage for reasons of simplicity and unnecessary bulk. All current browser generations support the HTML5 localStorage API: http://caniuse.com/#search=localstorage [^]
(0026069)
one-testlink   
2017-03-06 16:18   
@fman, is there a target version defined yet for this issue? Could it be taken into 1.9.17? When will 1.9.17 come out? Thanks!
(0026071)
fman   
2017-03-06 16:46   
NO ETA for 1.9.17, but this will be included
(0027223)
one-testlink   
2018-02-22 15:07   
For your information I just created a pull request on GitHub for a fix for this issue.
(0027282)
hughkay   
2018-04-10 12:15   
@one-testlink:
Using your bugfix, I have encountered some issues:
- the user list table and event table are not rendered any more.
I only see the header info and buttons etc.
The rest of the page is blank (see screenshot attachment).
(0027432)
stef-lhm   
2018-05-08 11:40   
We have the same problem on TL1.9.14 with FF52.x or higher.
Deleting cookies is not a good workarround, especially not, if you are in difficult test executions. Navigation in the tree-menu causes that problem every few minutes. It is a complex tree structure, but i think this should not matter at all.
Is there a solution for this problem or maybe planned in a future TL version?
(0027433)
one-testlink   
2018-05-08 11:59   
@hughkay:
Can you provide the JavaScript console output you get in the browser when loading the pages you have issues with?
(0027434)
one-testlink   
2018-05-08 12:06   
@stef-lhm:
The fix is straight forward, but should be obviously included in the official repository. I have not heard from @fman about my pull request on this issue since February 2017.
(0027435)
hughkay   
2018-05-08 12:18   
@one-testlink:
I've found the issue in my code: it was a typo during initialization of the GridPanel in "inc_ext_table.tpl" file (identified via Javascript console in browser, so thanks for the hint :-) ).
So, your code is fine.
(0027436)
stef-lhm   
2018-05-08 12:29   
Great news, nice to hear this.
i am optimistic that in the next version this very old bug will be fixed.
thanks a lot! Good job!!
(0027437)
one-testlink   
2018-05-08 12:35   
@hughkay: LOL
@stef-lhm: High hopes about that "next version". TL is a one man show. :) I'm afraid you'll have to crack a bottle this August for the ten year anniversary of this ticket.
(0027438)
one-testlink   
2018-05-08 12:41   
@stef-lhm: You can change the allowed HTTP-Header size on the server to mitigate the issue. See this discussion: https://stackoverflow.com/questions/686217/maximum-on-http-header-values [^]
(0027439)
stef-lhm   
2018-05-08 13:25   
i thought of changing the header size too, but i understood that the cookie is getting bigger and bigger. so this will not solve the problem. isnĀ“t it?
for a one man show, it is pretty good stuff!!! Respect! so i stay optimistic!