(cache)There is no simple solution for local storage - Mozilla Hacks

TL;DR: we have to stop advocating localStorage as a great opportunity for storing data as it performs badly. Sadly enough the alternatives are not nearly as supported or simple to implement.

When it comes to web development you will always encounter things that sound too good to be true. Sometimes they are good, and all that stops us from using them is our notion of being conspicuous about *everything* as developers. In a lot of cases, however, they really are not as good as they seem but we only find out after using them for a while that we are actually “doing it wrong”.

One such case is local storage. There is a storage specification (falsely attributed to HTML5 in a lot of examples) with an incredibly simple API that was heralded as the cookie killer when it came out. All you have to do to store content on the user’s machine is to access the navigator.localStorage (or sessionStorage if you don’t need the data to be stored longer than the current browser session):

localStorage.setItem( 'outofsight', 'my data' );
console.log( localStorage.getItem( 'outofsight' ) ); // -> 'my data'

This local storage solution has a few very tempting features for web developers:

It is dead simple
It uses strings for storage instead of complex databases (and you can store more complex data using JSON encoding)
It is well supported by browsers
It is endorsed by a lot of companies (and was heralded as amazing when iPhones came out)

A few known issues with it are that there is no clean way to detect when you reach the limit of local storage and there is no cross-browser way to ask for more space. There are also more obscure issues around sessions and HTTPS, but that is just the tip of the iceberg.

The main issue: terrible performance

LocalStorage also has a lot of drawbacks that aren’t quite documented and certainly not covered as much in “HTML5 tutorials”. Especially performance oriented developers are very much against its use.

When we covered localStorage a few weeks ago using it to store images and files in localStorage it kicked off a massive thread of comments and an even longer internal mailing list thread about the evils of localStorage. The main issues are:

localStorage is synchronous in nature, meaning when it loads it can block the main document from rendering
localStorage does file I/O meaning it writes to your hard drive, which can take long depending on what your system does (indexing, virus scanning…)
On a developer machine these issues can look deceptively minor as the operating system cached these requests – for an end user on the web they could mean a few seconds of waiting during which the web site stalls
In order to appear snappy, web browsers load the data into memory on the first request – which could mean a lot of memory use if lots of tabs do it
localStorage is persistent. If you don’t use a service or never visit a web site again, the data is still loaded when you start the browser

This is covered in detail in a follow-up blog post by Taras Glek of the Mozilla performance team and also by Andrea Giammarchi of Nokia.

In essence this means that a lot of articles saying you can use localStorage for better performance are just wrong.

Alternatives

Of course, browsers always offered ways to store local data, some you probably never heard of as shown by evercookie (I think my fave when it comes to the “evil genius with no real-world use” factor is the force-cached PNG image to be read out in canvas). In the internal discussions there was a massive thrust towards advocating IndexedDB for your solutions instead of localStorage. We then published an article how to store images and files in IndexedDB and found a few issues – most actually related to ease-of-use and user interaction:

IndexedDB is a full-fledged DB that requires all the steps a SQL DB needs to read and write data – there is no simple key/value layer like localStorage available
IndexedDB asks the user for permission to store data which can spook them
The browser support is not at all the same as localStorage, right now IndexedDB is supported in IE10, Firefox and Chrome and there are differences in their implementations
Safari, Opera, iOS, Opera Mobile, Android Browser favour WebSQL instead (which is yet another standard that has been officially deprecated by the W3C)

As always when there are differences in implementation someone will come up with an abstraction layer to work around that. Parashuram Narasimhan does a great job with that – even providing a jQuery plugin. It feels wrong though that we as implementers have to use these. It is the HTML5 video debate of WebM vs. H264 all over again.

Now what?

There is no doubt that the real database solutions and their asynchronous nature are the better option in terms of performance. They are also more matured and don’t have the “shortcut hack” feeling of localStorage. On the other hand they are hard to use in comparison, we already have a lot of solutions out there using localStorage and asking the user to give us access to storing local files is unacceptable for some implementations in terms of UX.

The answer is that there is no simple solution for storing data on the end users’ machines and we should stop advocating localStorage as a performance boost. What we have to find is a solution that makes everybody happy and doesn’t break the current implementations. This might prove hard to work around. Here are some ideas:

Build a polyfill library that overrides the localStorage API and stores the content in IndexedDB/WebSQL instead? This is dirty and doesn’t work around the issue of the user being asked for permission
Implement localStorage in an asynchronous fashion in browsers – actively disregarding the spec? (this could set a dangerous precedent though)
Change the localStorage spec to store asynchronously instead of synchronously? We could also extend it to have a proper getStorageSpace interface and allow for native JSON support
Define a new standard that allows browser vendors to map the new API to the existing supported API that matches the best for the use case?

We need to fix this as it doesn’t make sense to store things locally and sacrifice performance at the same time. This is a great example of how new web standards give us much more power but also make us face issues we didn’t have to deal with before. With more access to the OS, we also have to tread more carefully.

Evangelist for HTML5 and open web. Let's fix this!

124 comments

Pete

We had a good spec with WebSQL and with Mozilla’s decision to introduce IndexDB we now have fragmentation. Since IndexDB is not really an easier but simply a less powerful alternative, you cannot complain about the situation now.
Congrats!

March 5th, 2012 at 03:16
1. Ron
  
  I don’t believe IndexedDB is less powerful. Comparing IndexedDB to WebSQL is just like the discussion taking place on servers regarding NoSQL vs SQL DBs. They each have their strengths and there are certain workloads that perform much better if you choose the appropriate storage mechanism.
  
  Of course, it will be possible to simulate WebSQL-via-IndexedDB just as it is currently possible to simulate IndexedDB-via-WebSQL. I look forward to the vendor-prefixes being dropped so we can feel safe with IndexedDB.
  
  March 5th, 2012 at 03:24
2. Chris Heilmann
  
  “This document was on the W3C Recommendation track but specification work has stopped. The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path.”
  
  I can complain all I want – I am a developer and I want things that are simple and work. WebSQL was not a DB for the web, it was complex and the SQLlite poses a lot of issues. If a write is considered corrupted the DB gets wiped out – not something to rely on. Ask any Mail.app user who lost their archives.
  
  Also, this is about making localStorage perform, IndexedDB is far from having the same implementation numbers.
  
  March 5th, 2012 at 03:33
  1. Pete
    
    Sure you can complain. And I didn’t want to rant here. It’s just a level of frustration here. Actually our company decided to go the WebSQL way for intranets and recommends using Chrome. Since we used Gecko since version 0.x it was a painful decision but that’s life…
    
    March 5th, 2012 at 03:37
    1. Chris Heilmann
      
      Then your company is supporting a dead technology – I wouldn’t be surprised if Chrome removed support in the future seeing the IndexedDB work google is doing. Something to flag up :)
      
      March 5th, 2012 at 03:41
      1. Pete
        
        Well, given that there are a bunch of projects out there unsung WebSQL they cannot simply pull the plug. And as long as Apple heavily supports this on iOS they also cannot remove this within in a host or middle term.
        So, right now, it’s the right technology. But I agree, you never know what will happen in the future.
        
        March 5th, 2012 at 03:46
  2. X
    
    Chris: I’m sorry, but the fact that you find SQL(ite) “complex” and that for you it “poses a alot of issues” doesn’t mean it no appropriate fot client-side web. It just means you need to go learn SQL(ite).
    
    March 13th, 2012 at 02:52
3. Alex
  
  Web SQL Database was quite literally a spec that said you “must implement the SQL dialect supported by Sqlite 3.6.19”. It was not a “good spec” — and it is now an abandoned spec — because no W3 recommendation has ever suggested something that had only a single implementation, much less a specific bugfix version. They’ve gone as far as writing their own web servers and browsers to get around this. Yet none of the vendors who started to implement Web SQL bothered to start an alternative implementation.
  
  A couple years ago, they told everyone quite simply what to do to get it to move forward: “we need multiple independent implementations to proceed along a standardisation path”. Nobody cared enough to do this, so you cannot complain about the situation now.
  
  March 5th, 2012 at 10:12
4. Les Orchard
  
  You’ve got it backward. IndexedDB was introduced because WebSQL wasn’t a good spec.
  
  Unless you meant to say “convenient to use”, which is not the same thing as “good spec”. A good spec should ideally introduce an API which is convenient to use, but it should also not shrug and include a vendor-specific hand-wavy thing like “User agents must implement the SQL dialect supported by Sqlite 3.6.19”
  
  March 5th, 2012 at 14:25
Ron

I’m glad I’m not the only one deeply concerned about the current state of HTML5 storage.

I get annoyed with all the IndexedDB advocates given that neither of the major WebKit browsers support it (although Android 5.0 likely will if Google switches completely over to Chrome). The specification is still a draft and current implementations are still vendor-prefixed: I do not want to be relying on it for production web applications.

Why is it that the major web browsers stopped short of providing a decent quota-increasing APIs? This is a severely myopic decision, one that we will all have to live with for a long time (as mobile phone software tends to have a longer tail than on the desktop).

I filed a bug report here with Mozilla about extending the quota-increasing behaviour to existing storage mechanisms (ie localStorage) and they laughed me out of town. mozIndexedDB already has flexible user-permitted quota-increases, and no-one is interested in fixing localStorage in the meantime. Good luck with your B2G and Web API initiatives with a complete lack of a stable storage API.

I filed a similar bug report with Chromium, and their brand spanking new Storage Quota API does not seem like it will be extended to WebSQL or localStorage anytime soon (although according to white-papers it looks like this was a long-term goal at one point).

As a mobile web application developer for professional enterprise mobility, this is a massive problem for me. I feel like web applications are so very close to being competent replacements for native applications, but for this painfully conspicuous omission.

Sorry about the rant, but I struggle with this problem on a daily basis and I’m angry with the W3C and browser vendors for failing to address it in advance.

March 5th, 2012 at 03:20
1. Jonas Sicking
  
  I know that Google is just as concerned about localStorage performance as we at Mozilla are. That’s likely why they aren’t extending the quota spec to localStorage. (Storing more data there will just make the performance worse).
  
  As for why they’re not extending it to WebSQL, your guess is as good as mine. But I do note that they’re investing heavily in IndexedDB.
  
  March 6th, 2012 at 03:14
Sean Hogan

Why is LocalStorage any slower than hitting the browser’s URL-cache?

Sean

March 5th, 2012 at 03:41
Marcel Jackwerth

A polyfill for localStorage seems impossible to me, since we can’t fake a synchronous API without a second thread. If the localStorage API was asynchronous, it could be emulated with IndexedDB/WebSQL and that by itself would be a reason against its stand-alone existence.

LocalStorage’s (easy) synchronous nature and the fact that async programming in JS is no fun / PITA (compared to Go/C#5) makes it difficult to advocate against it at the moment. Harmony doesn’t even have anything async-related on the roadmap, leaving us with StreamlineJS, TameJS, IcedCoffeeScript, and Dart (with an experimental `async` keyword).

March 5th, 2012 at 03:50
1. Burak Yiğit Kaya
  
  Harmony actually comes with a yield keyword similar to Python which can be used to ease the pain of async programming via libraries like TaskJS if I understood your concern correctly.
  
  March 5th, 2012 at 06:42
Christopher Biscardi

Is LocalStorage meant to be used as a file/blob store? I was always under the impression that it was a simple key/value store used for json-stringified data and the like.
Do these problems still pop up when not storing files?

March 5th, 2012 at 04:11
1. Robert Nyman
  
  Like you say, it is meant like a simple key/value store, but has been used to lots of things – mostly due to lack of other viable options, I think.
  
  Performance-wise, it’s the same issue, no matter what you want to store. You can’t actually store files, per se, but just Data URL as strings.
  
  March 5th, 2012 at 04:21
Remy Sharp

I’m still on the fence as to whether we really need to kill off localoStorage, mostly because IndexedDB hasn’t landed without vendor prefix yet, debugging tools are MIA and it’s hard for devs (accessibility of API design is going to be a factor of success – but agree that a polyfill on top of localStorage pointing to IndexedDB might be an elegant solution).

Anyway, one thing I wanted to clarify (or ask your opinion too) is you’re advocating *against* localStorage, but what about sessionStorage? Due to the way the data persists, it doesn’t lend itself well to storing a ton of data in there, so devs are much more likely to use it for tiny snippets of data. Sure that blocks still – which is why I’d like to get clarification on whether you’re advocating “stop using localStorage” or “stop using WebStorage”.

March 5th, 2012 at 04:24
pd

Thanks for your honest and realistic approach to this topic.

March 5th, 2012 at 04:32
Moldován Eduárd

Hey guys,

There is one thing I do not understand. I am no localStorage guru, but why does the browser load any data while loading the page?
I suppose a possible answer would be that the localStorage engine is initiated during page load. If that is the case, why cannot it be done after page load?
Many other things make the document render slow if done at a bad time.

I am very curious about the answer.

Cheers,
edi

March 5th, 2012 at 04:32
1. Marcel Jackwerth
  
  localStorage has a synchronous API. If the data isn’t loaded upfront, then any get-value call could block the whole UI thread for a few milliseconds (or even seconds). So actually this is a work-around of browser-vendors to compensate the short-comings of the localStorage API (discussed in “The main issue: terrible performance” of this blog post).
  
  March 5th, 2012 at 04:47
  1. Moldován Eduárd
    
    Well, this workaoround is then not actually a workaround. In my point of view, when I load a page and it hangs for seconds, I go away and never come back. If I am working, doing something, on the page, then it hangs for a while, telling me that it is loading something, I accept that, because I asked it to do something for me.
    This might be a very simplified point of view, but might work. What do you think of it?
    
    March 5th, 2012 at 06:01
Jens Arps

I simply fail to understand the whole “Anti-localStorage” movement. And I have not seen arguments yet that convince me to stop advocating localStorage. As I did often before, let me adress the two most mentioned issues with ls:

– Performance: localStorage outperforms _any_ other storage engine, including IDB. See http://jsperf.com/indexeddb-vs-localstorage/2

– Blocking: I just have not seen localStorage really blocking or hanging the UI in the real world yet.

Currently, there just isn’t a feasible alternative to ls. IndexedDB might become one, given that:
– the spec is stable
– all major browser vendors have a plugin-free impl
– there are easy-to-use wrappers for it
– it’s available on mobile devices (well, would at least be nice to have)

So, all we can do now is to encourage ppl to look at IDB, play around with it and get a feeling for the API. We need good and working tutorials and we need working wrappers. Right now, the situation is awkward: most wrappers for IDB just don’t work and most tutorials don’t work. The spec changes ever so often (I’m having a hard time myself keeping IDBWrapper up-to-date), and the new versioning API makes a dynamic use of IDB close to impossible.

When someone asks me what storage engine to use in production right now, I tell them to go for localStorage. Because there is no alternative, and because I still don’t see what’s so bad about it.

March 5th, 2012 at 05:35
1. Jonas Sicking
  
  When you say that you haven’t seen localStorage blocking “in the real world” yet, how many devices have you been looking at? The problem is that it’s heavily dependent on the disk speed of the device, as well as what other IO operations the device is doing at the same time. So just looking at your computer, especially if you have a good one, won’t provide very diversified testing.
  
  Measuring performance for localStorage vs. IDB is very hard. Since localStorage is synchronous browsers try to work around it’s blocking behavior by loading the data beforehand. So just sticking Date.now() calls around a loop which reads from localStorage won’t actually measure the actual IO.
  
  But I definitely agree that IDB isn’t as good of an alternative yet since it’s not as widely implemented. Firefox, Chrome and IE10 has it. Opera is working on an implementation, but so far no word from Apple for Safari.
  
  For mobile, Firefox mobile definitely has it. I think Chrome for Android does too, and I would expect Windows Mobile to get it very soon.
  
  But again, who knows about Apple and iOS. Apple is notoriously tight lipped, but I would suspect that they don’t want to get left behind once all other browsers implement IDB.
  
  March 5th, 2012 at 11:30
  1. Ian Bicking
    
    Perhaps it is more realistic to look at localStorage as an API to in-memory data structures, with asynchronous “real” persistence (i.e., disk access) happening in the background. And from what you describe that’s essentially how it is implemented. Memory access effectively doesn’t block, and because of low quotas that access probably won’t exceed what can be reasonably stored in memory. So in many ways the current status seems reasonable. Also the asynchronous nature of how that data is persisted seems reasonable in a web context where there are never any solid promises that anything will be written atomically or won’t be thrown away.
    
    If you start to enrich the localStorage API (like give good ways to increase the quota) then it seems like it will almost certainly start to cause some of the problems that right now feel theoretical.
    
    March 5th, 2012 at 12:06
  2. Jens Arps
    
    Yep, I’ve been working with quite some mobile devices, including some _very_ crappy ones – ranging from crappy hardware prototypes to very cheap androids. And I did make intensive use of ls there, w/o experiencing any issues.
    
    And regarding performance measurement: It’s not that hard. If you care to look at the tests, you’d see that it’s read and write combined, and uses randomized data. So, yes, I’d say that the results are reliable.
    
    Really, I just want to understand why I should stop advocating ls. The main issues that are most often presented as arguments against it, just don’t apply – at least not for what ls is used for today.
    
    March 6th, 2012 at 02:00
    1. Jonas Sicking
      
      Like I said, the load of the data isn’t always when you’re actually using the API.
      
      In a good localStorage implementation the data remains in memory after the first access, so really the performance hit happens on the first access and then never after, until the user reloads the page.
      
      If you look at the tests you link to, they access localStorage the first time outside of the loop that they are actually measuring. So they aren’t actually even measuring the interesting part for localStorage.
      
      As for the IndexedDB measuring, the first mistake they are doing is to wait for the success callback from the write before they are doing the read. That means that the webpage is just sitting around waiting for a large part of the test for no reason (IndexedDB lets you schedule the read immediately after the write and you’ll get correct data).
      
      Similarly, if you want to read 1000 values from IndexedDB, you wouldn’t read one value, wait for that to get back, then read the next value. However that seems to be exactly what the test is doing. What you’d really do is to make 1000 .get calls, and then wait for all results to come back.
      
      The second problem is that it’s that since IndexedDB is an asynchronous API you can’t simply stick a Date.now() before and after and try to measure throughput that way. Since IndexedDB is asynchronous it means that your page is free to do any other processing, react to user interaction etc, while the loads are happening. That’s not taken into account at all in the test.
      
      Like I said, testing IndexedDB vs. localStorage performance is hard.
      
      Another problem is of course that we’ve not even begun working on performance for IndexedDB, but we’ve had quite a while to work on localStorage by now.
      
      March 6th, 2012 at 03:33
      1. Jens Arps
        
        Thanks for taking the time and looking at the tests!
        
        All of your points are valid. I didn’t plan to test in that detail, but if one wanted to, then – again, you’re right – testing indeed _is_ hard. And, to a certain extent, maybe even unfair. I can’t quite think of how to design a test that is precise and fair enough – but maybe that’s not necessary, as the IDB spec is subject to change and impls are far from being optimized.
        
        I totally see the issues w/ disk I/O that blocks before ls access (the “interesting part”), still I think the claim of “terrible performance” is a bit hard to hold up to. Or I was just lucky that I didn’t run into disk I/O issues in my work.
        
        As the title says, there is no simple solution. But when I have to choose between fetching a JS file over a possibly bad and expensive network or fetching it from ls, I’d rather risk blocking the browser for a few; and I think that’s a reasonable decision (at least until we have alternatives). And when I have to store 1k worth of application settings, I’d also risk the blocking if it spares me having to pull in code for IDB or Sqlite or hitting bandwith with cookies.
        
        When IDB is finally there, I’ll be immediately with you and promote migration.
        
        March 6th, 2012 at 04:58
Benedikt P. [:Mic]

Offtopic suggestion:

What about using something like “Summary:” instead of that (imo) awful “TL;DR:”?

March 5th, 2012 at 05:40
1. Chris Heilmann
  
  Totes
  
  March 5th, 2012 at 05:44
check_ca

Isn’t the HTML5 Filewriter API also an alternative? I naively thought it was designed to store files…

http://www.w3.org/TR/file-writer-api/

March 5th, 2012 at 08:26
1. Jonas Sicking
  
  FileWriter has even less cross-browser support than IDB. Currently it’s *only* implemented in Chrome. At last TPAC, all other browser vendors said that they had no plans to implement it.
  
  March 5th, 2012 at 11:48
  1. check_ca
    
    Thanks for the response. I didn’t know other browser vendors position. However, I am not convinced the FileWriter API is not a possible *future* alternative (which works since months on Chrome).
    
    March 5th, 2012 at 13:08
  2. A.J.
    
    google: FileWriter, etc
    mozilla: DeviceStorageAPI
    microsoft: WinRT
    
    I wonder why there is no single solution, but I am pretty sure it’s not a technical problem. Could it be that there is some strategically implied wasting of my time going on? Has happened to me before, and I didn’t like it.
    
    March 5th, 2012 at 15:34
    1. Jonas Sicking
      
      The DeviceStorage API is intended for quite different use cases than “FileWriter, etc”.
      
      I assume that by “, etc” you mean the rest of the FileSystem API?
      
      If so, the use cases for FileSystem API is mainly to be able to simply store files. I.e. similar to IndexedDB and localStorage, but aimed specifically at a single “data type”, i.e. files.
      
      Personally I think IndexedDB is a better solution for that. Starting with Firefox 11 we will support storing File and Blob objects in IndexedDB, just like any other data type. I.e. you don’t need to use one API for storing strings, numbers, JSON objects and another for storing Files. You can use the same API for all your data.
      
      The DeviceStorage API is intended to give access to the “pictures” and “documents” folders (in a safe way of course). That is something that the FileSystem/FileWriter API does not do.
      
      March 6th, 2012 at 03:04
Rakesh Pai

To put things in perspective, without localStorage, the alternative has been to load data off the server over the network. Undoubtedly, that’s orders of magnitude slower. We have eliminated that performance overhead by going to disk instead of the network.

Also, the optimization mentioned where browsers keep stuff in memory should speed up seek times by orders of magnitude again. The concern of this hogging up memory is a real concern too, but this is obviously capped to a 5mb max. Granted it’ll consume a little more than 5mb when in memory, but it won’t be disproportionately much more.

So, performance is not that big a deal (disk vs. network, memory vs. disk, memory vs. network), and the sync API isn’t a big deal either (loading from memory anyway). So the only real problem is that there’s a 5mb limit.

Or am I missing something?

March 5th, 2012 at 08:51
1. Jonas Sicking
  
  FileWriter has even less cross-browser support than IDB. Currently it’s *only* implemented in Chrome. At last TPAC, all other browser vendors said that they had no plans to implement it.
  
  March 5th, 2012 at 11:33
  1. Jonas Sicking
    
    Hmm.. not sure how that happened, the previous response was intended for another comment.
    
    Anyhow.
    
    Going to the network is indeed almost always slower than going to disk.
    
    The problem is the synchronous API. It’s not true that it’s always loading from memory. We couldn’t afford to load all localStorage data into memory since it can be a lot. Instead what we do is that we try to anticipate when a website will use localStorage and at that point load the data into memory.
    
    But anticipating isn’t foolproof. And sometimes a request to get data from localStorage happens before we’ve managed to load the data into memory. Since the API is synchronous, that means that the page is blocked until we’ve loaded the data.
    
    So while going to the network is slower, it doesn’t result in the page getting blocked (unless you’re using sync XHR, which you shouldn’t do for the same reasons).
    
    I’m of course not arguing that storing data on the server is always better than using localStorage. But I wanted to answer some of the questions you raised in your comment.
    
    March 5th, 2012 at 11:47
    1. Sean Hogan
      
      So if a site uses LocalStorage to save settings (basically a replacement for cookies) and NOT to save files then performance shouldn’t be an issue?
      
      March 5th, 2012 at 15:10
      1. Jonas Sicking
        
        It means that we’ll have to load less data, which certainly will make the first access faster. But it still requires a synchronous load, which means that if the disk is busy doing something else, we’ll have to wait for the disk to finish what it’s doing before the page gets unblocked.
        
        March 6th, 2012 at 01:58
    2. Rakesh Pai
      
      Thanks for the reply, Jonas.
      
      The technique of anticipating if a site will need localStorage seems like a good idea. The race condition between when the browser decides whether the page needs localStorage (again, a local lookup or something) and then loading the data into memory, vs the site starting to use it (usually involving going over the network to load scripts), shouldn’t causse too much of a performance problem, I guess. I might be wrong, of course, but it does seem like there’s room for improvement.
      
      There might be ideas to pick up from in-memory databases that lazily persist to disk, like say MongoDB.
      
      I guess what I’m trying to say is that the API right now is beautiful, and is very easy to use. If the browser can do the heavy-lifting to ensure that the API works fast enough at least in most circumstances, we’re doing pretty well, isn’t it?
      
      March 5th, 2012 at 20:17
      1. Jonas Sicking
        
        I don’t personally have numbers, but do you really want an API which only works “in most circumstances”?
        
        I think most users want something that works reliably, so ultimately that’s what I hope developers want too.
        
        March 6th, 2012 at 02:05
Charles

What about compressing localStorage, maybe an API for it to
a) allow for more storage in localStorage
b) actually speed it up. This happens if CPU is much faster than Disk IO

Also, use access to clean up localStorage.

March 5th, 2012 at 09:05
JulienW

I’m also concerned about security, because every pages on the same origin (ie: scheme/hostname/port) can use the same data in localStorage.

Maybe that’s less relevant now, when quite everyone have its own domain name, but I think it’s still an issue in some shared web hosting services.

March 5th, 2012 at 09:26
1. Jonas Sicking
  
  All pages that are on the same origin can always read each other’s data. If you open an to another page that’s in the same origin, you can reach in to it and access any part of the DOM or the JS objects that it has created.
  
  Effectively the security model of the web is that each origin is its own security domain. It’s unfortunately too late to try to create anything more fine grained than that.
  
  March 6th, 2012 at 02:08
  1. Jonas Sicking
    
    Oops, that should say:
    
    If you open an *iframe* to another page…
    
    March 6th, 2012 at 03:47
  2. JulienW
    
    Actually, the “security model of the web” as you call it is not so ubiquitous.
    
    There is actually a very known exception: cookies. The security model of cookies consist of both domain (which can be any parent domain of the domain of the webpage, btw) and a path. I’m really wondering why this wasn’t used for other security model.
    
    Maybe it’s too late; then it must be known so that applications stay safe.
    
    March 15th, 2012 at 03:04
2. Nealle
  
  Security is a concern for me too, I work for a large FI and we are writing guidelines for developers to use as they begin developing in HTML5, with a particular focus on mobile and right now inspite of all the issues mentioned we will simply ban offline storage as we cannot encrypt the content to a level that would make us feel comfortable that the data is secure in case of device theft or loss.
  
  March 14th, 2012 at 14:00
Kevin

Thanks for writing up this warning. Do you have any data on actual performance statistics? It seems like relative to any sort of web call localstorage is still likely to be very fast… it is just that when you are expecting it to be transparent in a synchronous way that it is poor. Perhaps in conjunction with web workers or some other way to do this asynchronously it could still provide a huge speedup relative to loading from the web.

March 5th, 2012 at 09:31
Andy Fuchs

I completely agree to Jens. Localstorage was a god-send, when there were times we had to battle cookies.
The main advantage is, it’s here TODAY and it WORKS on all modern browsers (mobile or not).
For people who focus on the mobile user (and they’re getting more and more), it is the only safe storage option out there. Everything else is limited.
I abandoned Firefox quite a bit in recent times due to the fact, that there’s a webkit on most mobile machines, supporting common APIs as on the desktop.
From a developer’s pov, it is inevitable that we stop the dictate of single browser vendors (first IE, then MOZ, now Webkit) to kill common and useable APIs.
This would make the web more useable for all of us!

March 5th, 2012 at 09:41
Larry Garfield

You really have to look at these issues in context. Yes, a synchronous API is going to block, but is that still going to be better than a network roundtrip? Sometimes, yes. If you’re storing only a few small bits of data (rather than a few megs), then the memory overhead is minimal.

“Change the localStorage spec to store asynchronously instead of synchronously? We could also extend it to have a proper getStorageSpace interface and allow for native JSON support”

That really sounds like the best option. localStorage has the advantage of the most usable API. So fix it. Add an async option (XHR has both sync and async options, recall), add a few more methods to it as needed, and we’re good. That just requires W3C and browser vendors to agree on doing that and we’re done.

… OK, maybe that will take a while. :-)

March 5th, 2012 at 09:55
Steve Souders

The problem IMO is that the browser disk cache doesn’t deliver the necessary performance and features. Studies show that normal users have less cache “hits” than we would expect. Purging algorithms are under-developed (eg, prioritization based on content-type). There’s no guarantee of space per website. And no API.

It’s been 2 years since I asked browsers to focus on disk cache ( http://www.stevesouders.com/blog/2010/04/26/call-to-improve-browser-caching/ ). At that time I warned that localStorage was being misused as a disk cache alternative (FB was already doing this). We’ve had some good improvements in disk cache – mostly in size – but we’re far short of where we should be. The horse has left the barn. Trying to fix the problem by evangelizing a move away from localStorage will not be successful.

LocalStorage is here to stay. The best solution is for browser vendors to address the issues. (Sorry – I know that means more work for you. I wish we had fixed disk cache before this got away from us!) What should browsers do?

1. add callback functionality – The leading gurus will use this wisely, but we must realize a majority of sites won’t adopt this new pattern for years.

2. better UI – It’s difficult for users to figure out how to see, edit, clear, & limit their localStorage. A better UI is needed.

3. LRU purging – If localStorage becomes widely used then 5MB per site may claim a large amount of user disk space. Browsers should add most-recently-used info to localStorage so the bytes for unused sites can be reclaimed.

4. preloading – Using this LRU data browsers can more intelligently determine whether to read strings from localStorage in to memory as soon as the website URL is requested (in anticipation of that HTML document response call getItem). This allows various memory use optimizations, eg, if a key hasn’t been accessed in a long time it shouldn’t be preloaded.

I’m sure there are other improvements browser teams can think of, and perhaps have already implemented. My main point is, the best way out of this is for the browser teams (once again) to step up to the plate so that web devs can focus on their app and not on the performance of features that have already been added to browsers.

March 5th, 2012 at 10:58
1. Taras Glek
  
  Steve, we have started addressing cache issues:
  * Firefox cache can now reach 1gb(which turns out to be too much on some machines).
  
  * A lot more work remains to be done. Some fundamental assumptions changed since browser cache was first introduced (ie harddrives are no longer strictly faster than network) and size of items cached has grown rapidly.
  
  * Stuff like html5 video requires cache to cache items to be sparsely stored(ie when you seek around in a video while it’s downloading).
  
  I think there is a lot of room for optimization in the browser cache…and given that a lot of use-cases for Local Storage seem to be to avoid dealing with browser cache, perhaps providing better cache-control from the browser will solve some LS woes.
  
  March 5th, 2012 at 12:12
  1. Larry Garfield
    
    Improved browser caching would be very much appreciated, so that we don’t have to try and come up with crazy crazy ideas like this: http://www.garfieldtech.com/blog/caching-tng
    
    :-)
    
    March 5th, 2012 at 12:19
    1. Taras Glek
      
      Larry,
      That’s awesome. I tried to summarize LS->Cache thoughts in
      http://groups.google.com/group/mozilla.dev.tech.network/browse_thread/thread/4acf1c6e5b071193#
      
      I noticed that most LS usage is to bypass the cache…So it makes some sense to just make the cache more useful rather than design a whole new subsystem. Thanks for planting the idea!
      
      March 5th, 2012 at 13:20
staras

In 1/6/2010 we read the article: Firefox 4: An early walk-through of I by Arun Ranganathan and Shawn Wilsher. Today after 1 and a half year we read again for an another article for indexedDB that this is the future of web storage. Where is the progress in indexedDB?
Why stop using WebSql?
Why stop using localstorage?
There is no logical.
please support easy implementation that exist..

March 5th, 2012 at 11:31
Jonas Sicking

I’d love to see a proposal for a simpler API a’la localStorage, but that is still asynchronous a’la IndexedDB. It doesn’t need to be W3C that drafts something up. If anyone drafts something up that seems to get developers excited, please don’t hesitate to send it to me and I’ll help pushing it to web browsers.

It should even be easy to write an implementation of such an API as a JS library backed by IndexedDB/WebSQL.

There are a number of ways to design such an API, so it’s not so much a question of can we do it, but rather what capabilities do we want such an API to have.

One solution would be something like:

getBetterLocalStorage(function(storage) {
x = storage.foo;
storage.bar = calculateStuff(y);
storage.baz++;
});

This API is very simple, but would require the implementation to load all the data into memory before calling the callback function. I.e. this API doesn’t scale very well if you want to store lots of data in it. But it works well for small amounts of data.

Another possible API would be:

betterLocalStorage.get(“foo”, function(x) {
doStuffWith(x);
betterLocalStorage.set(“foo”, x+1);
});

The downside with this type of API is that it’s very easy to accidentally have race conditions in the code. Or rather, it’s impossible not to. If you wanted to increase the value for the “foo” key by 1, you’d first read the data using an asynchronous call, then in a separate call set it. But there’s nothing guaranteeing that the value won’t change in between, for example due to the same page being open in another tab.

So another solution would be:
lock = betterLocalStorage.getLock();
lock.get(“foo”, function(x) {
doStuffWith(x);
lock.set(“foo”, x+1);
}

So the question is, what capabilities do we want the new API to have? How much complexity in the API are we willing to have in order to get something that scales to large amounts of data, or that doesn’t have race hazards, or that solves all sorts of use cases?

IndexedDB definitely lays on the pretty complex end of the range. It supports a lot of use cases and should scale well, but the result is more complex than all of the above APIs.

For what it’s worth, all of the above APIs can be prototyped with just a few lines of javascript using IndexedDB (and depending on what data types you want, WebSQL).

March 5th, 2012 at 14:11
1. Rakesh Pai
  
  If I had to pick one (and I don’t like changing what we already have), I’d pick your second example. That’s mostly because it’s simple, to work with. Also, it’s possible to make it backwards compatible with existing code. If the developer wants ease of use, use the current style. If the data grows too big that disk IO seems to be a bottleneck, supply a callback to make it async. The API implementation just checks if a callback has been passed in or not, and based on that decides if it should be async.
  
  The race-condition problem might not be such a big deal. We don’t see people complaining about race-condition problems with Ajax, for example. Granted it might have tripped people off, but it wasn’t a huge learning curve for most. I’m guessing, of course.
  
  Of course, I still maintain that the API shouldn’t change from what we have now. I’m pretty sure that browsers can do more to make the performance overhead negligible with the current API.
  
  March 6th, 2012 at 20:31
2. Ian Bicking
  
  To handle locking, perhaps:
  
  betterStorage.get(“key”, function (x) {
  betterStorage.set(“key”, x+1, x, function (succeeded) {
  // if succeeded is false, then the precondition that key == x failed
  });
  });
  
  This makes naive race-condition-ignoring access fairly easy (leave out the precondition), and still allows careful programming, but avoids deadlocks. localStorage is not particularly reliable, so it’s not surprising that few people worry about the lack of ways to handle race conditions – it’s not where people are keeping critical data. Adding locks could help data integrity, but introduces other programming errors that can stall the page (or at least cause a callback to never be called, etc). I think that would be a bad compromise for this kind of API.
  
  March 15th, 2012 at 13:15
Kim T

Front-end developers really have no need to create and update databases, especially if the schema could change.

Why can’t we simply have an attribute in JavaScript which defines the variable as cached:

cache var items = {};

Then we can overwrite this object simply using:
items = {};

Or remove using:

delete items;

simples…

March 5th, 2012 at 16:15
1. Chris Heilmann
  
  I really like that idea. There could be a trigger for the “do you want this” but it makes a lot of sense.
  
  March 5th, 2012 at 16:27
  1. Rakesh Pai
    
    If I understand correctly, this doesn’t solve most of the problems. The data will need to be stored/retrieved from disk, and that’ll mean we’ll either need it to be sync, or that we’ll need a callback-based async API. Also, this holds all the data in memory, so doesn’t help with reducing memory footprint. Also, there’s no mechanism described to control data growth, so I’m guessing the 5mb limit applies here too, and a permission based (or other) mechanism to get more storage will still be necessary.
    
    Or again, I’m missing something. :D Chris, not sure what you meant by “trigger for ‘do you want this'”.
    
    PS: The captcha is really hard! 7 refreshes before I could find a readable one.
    
    March 5th, 2012 at 20:39
2. Matthew Holloway
  
  That would probably be syncronous and blocking, right? I don’t see a way that could have a callback, or a promise object, to defer accessing the data.
  
  And when a browser sees a cache statement would it have to load in the whole map at that point? It would probably be difficult to make it granular enough to lazy load values from keys. If so it couldn’t really be used for in-browser file systems which could need to be able to store 100s of megabytes.
  
  How would you uncache? You’ve got an example that sets the cache to empty but it would presumably still retain the value of an empty map.
  
  March 5th, 2012 at 20:20
3. Jonas Sicking
  
  This is basically exactly what the localStorage API is. And thus will have exactly the same problems.
  
  To fix the problems with localStorage, we really do need something that is asynchronous.
  
  March 6th, 2012 at 01:55
forrestliu

maybe the async file api would make more scene.

March 5th, 2012 at 18:17
Henri Sivonen

The list of problem lacks the most famous localStorage problem: In multiple event loop browsers (such as IE and Chrome) the implementation of localStorage either has to expose race conditions to Web content or use a very-bad-for-performance cross-process lock.

March 6th, 2012 at 00:41
Andy Walpole

Why the choice of IndexedDB OR WebSQL? Why can’t browsers implement support for both?

March 6th, 2012 at 01:46
1. John Thomas
  
  It would not be without precedent, after all, once upon a time browsers looked at innerHTML and DOM support as competing specs, but eventually decided to go with both. Moreover, it could simply be designed as: here is a low-level API and here is a high level one. Thirdly, even if mozilla just wanted to implement a compatibility layer, if that layer was in-browser, you could utilize the SQL parser used for SQLite, which I think would give you much better performance (Yes, everyone says that js performance is near-ing native performance, maybe if the javascript is written for high-performance and then native-code isn’t, but even then, I’ve yet to see that happen).
  
  March 6th, 2012 at 11:35
  1. Shawn Wilsher
    
    Mozilla ships a newer version of SQLite that it uses internally than what the WebSQL spec requires, so they can’t just use the SQLite parser they already ship.
    
    March 8th, 2012 at 11:53
John Thomas

To the idea that the WebSQL spec was a poor spec: Fair enough but if that was Mozilla’s issue with WebSQL, there was a very simple and straight forward way to fix it – write out a spec for the SQL support based on SQLite’s.

The bigger issue is that there was a good idea (the general concept of a web sql database rather than the particular specs) that was gaining momentum and Mozilla killed that momentum, now we will have to wait years for Mozilla to convince all the other browser makers to make its standard, we need to wait for more test implementations and surprise issues (again since your designing a new API, you’re going to encounter people putting it to weird uses and going to need to adjust for those use-cases, whereas with an SQL-based API, you can take the historic patterns of SQL usage as warning about potential issues), before we have a cross-browser solution for web databases.

And to the point that you can build an SQL-compatibility layer on top of IndexDB, yes, but with javascript having to implement the SQL-parser and mapper to IndexDB operations (especially if you want to add support for queries with functions in them), I doubt that SQL-compatibility layer will be at all reasonable in performance.

Then there’s the pain of developers having to use another API, another set of best practices (even with a simple API, best practices can be a killer, jQuery’s selector syntax is beautifully simple, but things start to get more complex when you start worrying about performance and selector optimization). In my experience, I would say at least 3/4 of web developers on the front end have some back-end experience and some exposure to SQL, which would make HTML5 storage a lot easier to start using if it used that approach.

March 6th, 2012 at 11:28
1. Shawn Wilsher
  
  You’ve got your history wrong. The IndexedDB spec was originally from Oracle, and Google, Microsoft, and Mozilla liked it enough to jump in and help standardize it.
  
  March 8th, 2012 at 11:52
John Thomas

Now that I’ve ranted quite a bit about WebSQL, let me step back and say, sorry if I’ve been harsh, the world of web technology is steadily moving away from the vision I had of it, but that’s my issue not Mozilla’s. As my disillusionment grows, I will probably switch to doing more native programming and be better off for it.

But here’s the essential problem with HTML-application storage, users need full control of their storage options (with minimal hassle in managing it), while applications depend on persistent storage being, well, persistent. This is going to be a complicating factor to any storage API and needs to be solved sensibly first (even if we’re talking just for cookie storage).

If I were a spec-ing man, and I’m not, I might write up a spec that incorporated an API for saving web-data. That is, the site could trigger a prompt (do you wish to save your data from this session?) Moreover, when you clear your cookies (which should clear all persistent data, since users are only used to cookies as a form of persistent data and shouldn’t overlook something they meant to clear), perhaps users could get a prompt (with an option never to show again), “You have X many web applications with saved information in your browser do you want to delete this” -> the user should also then get a trigger for a per-application editing of which to save and which to discard.

In addition to all that, the user should any point have the option to “export” their data for a particular web application and re-load it in another application. This is a point where most storage methods fall short and while some sync services provide sync-ing of say cookie data,

A. they usually don’t provide full-syncing of non-cookie storage
B. if they do it’s usually browser-specific
C. I, at least, like to have a file for my exports, especially since I like to mess around with said files.

What would be really nice, and I realize browser vendors may not care about this, given that they are trying to kill plugins (without providing similar or near similar functionality in extensions, this is one aspect where Mozilla shines), but it would be nice if an API was presented to plugins to utilize this for their data storage (what would be really cool if you could trigger an option to only use plugins that use the API for persistent data storage (ie, they could still save to %TEMP% for cache data, or to this API, but not to the underlying filesystem), that may not work for all plugins, so I would really like it as a switch presented to users (in the options menu, all options should be accessible somehow through the options menu about:config))).

March 6th, 2012 at 11:54
1. John Thomas
  
  Realize I’m reposting quite a bit (and my lunch hour is pretty much over, so I am officially slacking), but what if you could simply have a store webpage state option. This doesn’t exactly solve the whole user needs for erasable storage vs. persistent storage needs dilemma, but even with a full API for a js database, either websql or indexdb, developers are going to end up storing a lot of data in the DOM as attributes and hidden elements. If they want to make that data persistent, they would need to pull it out and serialize it even if there was a full-functioning persistent storage API (and nowadays most developers serialize the data and post it to the server where it exists as session data).
  
  But what if they could just save the current state of the web page in the browser, and then the site can ask to say, resume named session (there could be an input type session for example).
  
  This isn’t exactly a replacement for a persistent storage api (after all you may have a massive page and just a little persistent data), but it would be cool at least.
  
  March 6th, 2012 at 12:05
Kim T

if something along the lines of
cache var item = {}

Didnt work then i think the next best thing would be a json feed save otion
window.store(feed, ‘data.json’);

Then we could also save objects to disk and load them back afterwards in the same format as we created or loaded

March 6th, 2012 at 13:18
1. Jonas Sicking
  
  Like I stated above, if people can come to an agreement on a *asynchronous* API that they’d like browsers to implement, that would be a huge first step.
  
  I even provided three examples above of what such an API could look like.
  
  Everyone has different use cases, so finding a solution that fits everyone is not easy.
  
  March 6th, 2012 at 14:59
greg

Android, iOS, Symbian have SQLite and app developers are happy. SQL is too complex? But JavaScript and CSS positioning aren’t?

Sure, continue search for your own WinFS.

March 6th, 2012 at 16:22
1. Robert Nyman
  
  I believe the cons of WebSQL are covered in comments above, in terms of specific version and other issues.
  
  March 7th, 2012 at 01:31
  1. Pete
    
    I still try to find out why the Mozilla guys are so against WebSQL without any rational arguments.
    
    It is also stated above that currently WebSQL is covered by almost all mobile devices and all are happy.
    
    March 7th, 2012 at 02:04
    1. Jonas Sicking
      
      There are many reasons we didn’t like WebSQL.
      
      1. No two SQL engines implement the same dialect of SQL. Your most basic “Select * from table” query will work everywhere, but things quickly start behaving somewhat differently. So unless all browsers agree to use the same SQL engine backend (and agree to do so for the rest of time, even if other faster database engines are created), you’ll have different browsers doing different things. That’s no good on the web.
      
      2. There are often small differences in the SQL dialect that different versions of the *same* SQL engine. As most server developers know, you often need to make small changes in your queries when upgrading your SQL engine to a new major release. That’s fine on the server where you are in control of when the SQL engine is updated. It’s not fine on the web where you don’t know when users are going to update their browsers and when different users have different versions of browsers. And browsers will be forced to upgrade their SQL engines since old versions stop being supported and have known security issues. Also we want to be able to use the latest version for performance reasons.
      
      3. Even for SQL queries where SQL engines give the same result, the performance can be vastly different. If a SQL engine doesn’t realize it should use a particular index it could easily be 1000 times slower even for moderate data set sizes. Would you really be ok with some function in your site running 1000 slower in a particular browser? This is not a big problem in a server environment where you can test your query against the SQL engine you’re planning on using, but it’s not ok on the web where we want APIs to work consistently across browsers.
      
      4. Performance in SQL is notoriously hard. Missing an index can be the difference between having your query hang on execute in a fraction of a second. One of the things that make performance hard is that you’re not actually expressing what you want the SQL engine to do. You express some constraints on the result that you want and hope that the engine is smart enough to use an entirely different execution strategy which is actually fast. Any SQL developer that has had to deal with complex performance critical queries have sat tweaking their queries between different expressions that actually mean the same thing, until the SQL engine suddenly “gets it” and uses the index that you want. Another problem is that if you have bad performance, the problem might not be the part of the code where performance is slow, but rather where you constructed your table as to have the wrong order of columns, or the wrong set of indexes.
      
      5. SQL isn’t particularly “Webby”. I.e. you can’t store your JSON objects directly. Instead you have to manually pick your data apart and stick it into columns. Sure, you can use JSON.stringify, but then you can’t index on any of the data in your object.
      
      6. SQL-injection attacks. They happen *everywhere* where people use SQL in combination with untrusted data. Even when there are API features to help you avoid it. Concatenating strings is just so darn easy that people constantly fall into that trap.
      
      March 7th, 2012 at 20:12
      1. greg
        
        1. Just use SQLite. “So unless all browsers agree to use the same SQL engine backend” – So the problem is political. There are also small differences in HTML/CSS implementations. So being consistent you should stop supporting these specifications also.
        
        2. Thank God you don’t work in IE team and we don’t have to live with IE6 bugs for the rest of time. BTW SQLite is small enough so it is possible to keep few versions inside one browser.
        
        3. Is this some kind of joke? Millions of developers already test web apps in IE, FF, Chrome and Safari.
        
        4. Just wow! Remove loops and arrays from JavaScript. Do you know how hard is to sort numbers in O(n) time? Just add too many numbers to an array, sort them with bubble sort and your browser is frozen!
        
        5. And with localStorage it is possible to index objects by fields?
        
        6. Nice try. On the client side you don’t have to worry about SQL injections at all. Clue: you have to validate all data once again on the server side.
        
        March 8th, 2012 at 12:26
Lars Gunther

No one has mentioned this yet, but one main argument against WebSQL is that it is not safe by default against SQL-injection.

Considering how many enterprise applications that have been vulnerable to SQL-injection despite being developed by well paid and supposedly knowledgeable programmers, it is safe to say that unsafe defaults are not a good thing.

P.S. please stop using reCAPTCHA. It is beyond absurd annoying!

March 7th, 2012 at 07:38
1. abc
  
  You seriously worry about SQL-injection on the client side? Why would anyone bother with that kind of attack, when he can easily change data in the storage anyway?
  
  November 6th, 2012 at 05:45
Alenas

This is another example of all the the problems that seems to be impossible to fix with HTML. Web needs to move on to better technology than HTML/DOM/JavaScript…All these clever people got stuck on one idea and they stopped innovating. Web needs complete overhaul…There is simply no need to kick the dead horse…

March 7th, 2012 at 21:03
Pete

In reply to 1-6: Instead of dropping it at all you could have done some fine tuning in the spec, i.e. define a stable SQL command set. Hopefully you agree that 99% of the SQL programmers do the same queries with some JOINs. Regarding SQL injection it would have been easy to redefine the spec that only prepared statements are allowed. And regarding “that’s not webby”, well…

You could also have build IndexedDB on top of WebSQL. So the developers would have had a choice and maybe even access the same DB in two different ways.

Lastly, the reason why IndexedDB did not convince everyone is that it is by far less powerful than WebSQL but at the same time too complex compared with simple async key-value storage.

As someone stated earlier, sometimes it is better to enhance and improve something which has momentum instead of having fragmentation. And that’s where we are. Right now we have 90%+ coverage of IndexDB on the desktop and 90%+ coverage of WebSQL on the mobile market.

I guess since Mozilla decided against WebSQL they don’t have the balls to redecide. That’s fine and time will tell what technology will succeed.

One question though: How about adding an async option like
localStorage.setItem(“key1”, “value1”); –> localStorage.setItem(“key1”, “value1”, function(){…});
and
a=localStorage.getItem(“key1”) –> localStorage.getItem(“key1”, function(a) {})
and telling developers that that’s the faster way. Same as with WebSQL. Instead of introducing something new, enhance the current one…

March 8th, 2012 at 00:53
regis

Hi!
We worked a lot on localStorage and i can say performance are good.
The only function which takes time is removeItem (only with firefox) wich can be replaced by setItem(key, “”).
The real limitation of localstorage is about the capacity (5M)
The W3 standard says the browser should throw a dialog-box when a website requires more space. There is only Opera who did it…

March 8th, 2012 at 06:29
Red

SQL is an ISO standard. Compared to this IndexedDB is a child’s toy. It is very sad that Mozilla (standards proponent) killed WebSQL. By this move, we’re stuck with localStorage for years.

March 9th, 2012 at 07:21
1. Shawn Wilsher
  
  Show me a SQL engine that implements the entirety of the ISO standard.
  
  March 13th, 2012 at 23:31
  1. Red
    
    You don’t need complete implementation of this standard. Simple joins, and grouping, and you have much more than indexeddb.
    
    There is no complete implementation of html or css in browsers and nobody here wants it to be abandoned.
    
    Today, we have sql engines everywhere: browsers, phones, servers etc. The fact is: huge databases like Oracle or MS are minority, and sqlite offers good sql92 subset.
    
    March 14th, 2012 at 23:26
    1. Pete
      
      Absolutely right. Thanks to Mozilla we now have
      
      – LocalStorage which is not async,
      – IndexedDB which is too sophisticated for simple key-value storage but not powerful enough compared to WebSQL, and
      – WebSQL which is no standard (anymore)
      
      And now we talk about adding a forth option.
      
      it’s still not too late to include WebSQL in order to have at least for mobile devices a common ground but I’m afraid that Mozilla doesn’t have the balls to re-decide.
      
      Actually there was a WebSQL addon in addons.mozilla.org but it has been removed for some reason. The only true argument against WebSQL I found here was from Jonas Sicking saying “It is not webby enough”. Sigh…
      
      March 15th, 2012 at 01:55
Nigel Kelly

Comepletely agree. LocalStorage is dodgy. I developed a game for webOS on the Palm Pre a while back and initially used LocalStorage for level saves. I used to to get loads of user complaints. I switched to websql and no problems ever again. Async guarantees that js will not run off by itself without your data. Sync does not guarantee this and hence the user complaints.

March 14th, 2012 at 05:10
richtaur

Where are the benchmarks??

March 14th, 2012 at 10:33
steve

“The main issue: terrible performance”

If this is the main issue then write better code and fix it. Yes, it really is that simple.

If the browser is loading all localStorage into cache when first visiting a web site then stop it. Any wonder you think it is slow. A single host may have many separate web applications which use localStorage so you should not be loading everything into memory at once. This is just common sense. Let the developers determine the best time to load any data, and also deal with their own caching. This is what we are doing as we can not guarantee that every browser is going to cache localStorage at all. So for those browsers that are caching the data it is just a waste of memory – again stop doing it.

Bottom line – there is nothing wrong with the localStorage API only how it is being implemented.

March 15th, 2012 at 05:26
Gregor

I use localStorage heavily in my app. It’s downloading user’s entire data to localStorage, only text data, but still up to 2MB for some.

Question:
Am I understanding it right, that the entire localStorage is loaded into memory after the first access, even if I only read out one key? And that reading the 2MB from disk is the main cause for blocking the UI, to the succeeding read / writes?

March 19th, 2012 at 08:56
Foo

Why not just add getItemAsync to localStorage, and an optional callback to setItem?

localStorage.getItemAsync(key, callback);
localStorage.setItem(key, data, optional_complete_callback);

setItem returns and has immediate effect, but the data is persistently on disk only when optional_complete_callback is called.

On the other hand, getItemAsync returns nothing, but passes the data to callback.

March 19th, 2012 at 11:07
Dannii

One problem that no one has mentioned is that neither localStorage nor IndexedDB work reliably (cross browser) for file:.

March 19th, 2012 at 22:53
Red

Did I guess well? The source of evil in localStorage is use of array operator? If so, can you flag it as deprecated? I haven’t found any word about it in Web Storage specification…

Nothing in the methods getitem / setItem does not require loading the entire database into memory …

You can also add optional callback argument to getItem.

The second issue is WebSql. Killing it does not help. The argument about incomplete implementations is highly ineffective, especially in a world where there is no consensus to the html / css / javascript between browsers.

Worse. This is a very shortsighted approach. Currently Javascript is not just a browser, the server developers have no choice – try to use something else than SQL and you may end killed by laughter.

I do not see the reason why Mozilla encounters many problems with SQLite. The same engine works in other browsers, mobile phones and many other devices.

March 20th, 2012 at 03:34
Parashuram

You could also use the IndexedDB polyfill over WebSql – this way we have to just deal with the IndexedDB API and it works on browsers that support IndexedDB or WebSQL (most browsers !!)

The polyfill is here – http://nparashuram.com/IndexedDBShim

June 7th, 2012 at 00:27
Marcello Nuccio

I’m using applicationCache + indexedDB to build off-line applications. IndexedDB makes it possible to store user input while off-line. That is great, but I’m concerned of how much persistent this data is. I’ve found that, in Chrome, the persistent storage “is available only to apps that use the Files System API”[1]. What is the policy used by Firefox? I can’t find any docs on this.

[1]: https://developers.google.com/chrome/whitepapers/storage

June 19th, 2012 at 23:22
1. Taras Glek
  
  We have an indexeddb prompt, once user accepts the prompt, data persists forever.
  We plan to add chrome-like promptless operation that autocleans the data, but we will still allow a prompt to persist data.
  
  June 20th, 2012 at 11:37
  1. Marcello Nuccio
    
    Looks great: default behaviour is automatic clean-up, but the user can choose to make data persistent. Chrome’s policy makes IndexedDB pointless for off-line use: how can I use it to store data if I don’t have any control on how much persistent it is?
    
    The sad thing is that, it will be difficult to support both browsers, because I will need to use File System API on Chrome, and IndexedDB on Firefox…
    
    June 20th, 2012 at 23:23
    1. JulienW
      
      I think you misunderstood something.
      
      Chrome supports both IndexedDB and their Filesystem API but the paper you referenced only talks about the Filesystem API.
      
      So you can use IndexedDB in Chrome as well.
      
      If you want to support Safari, Opera, and mobile browsers as well, you may use the shim there : http://nparashuram.com/IndexedDBShim/
      
      June 21st, 2012 at 00:08
      1. Todd Blanchard
        
        The shim (and IndexedDB for that matter) doesn’t provide any support for joining data.
        
        I need JOINS. This isn’t kid stuff – we have grownup data requirements needing complex views of our data. A key/value store isn’t going to cut it.
        
        November 9th, 2012 at 10:56
Marcello Nuccio

I know that Chrome has IndexedDB support. I’m using it and it works fine. The only problem is data persistence. The paper that I have referenced, talks about HTML5 off-line storage in general. The summary table[1], says: IndexedDB has temporary storage; File System API has persistent storage.

If I understand correctly, Firefox will not have the File System API, but IndexedDB storage is (and will be optionally made) persistent. Then the problem is: how to ensure persistence on both, Chrome and Firefox?

[1]: https://developers.google.com/chrome/whitepapers/storage#table

PS: Posting my reply I’ve found the problem described ad http://en.forums.wordpress.com/topic/you-are-posting-comments-too-quickly-slow-down-but-i-am-not

June 21st, 2012 at 01:05
Micah

Has anything about this article changed in the last 6 months? I see articles like this http://www.netmagazine.com/tutorials/make-disaster-proof-html5-forms and it makes me interested to use it for future projects with forms.

September 14th, 2012 at 00:12
1. Robert Nyman
  
  I would say that the pros and cons still stand.
  
  September 14th, 2012 at 02:01
Vance Ingalls

As I understand it (and I’m not too sure about this), as soon as a reference to localStorage is seen on a page (compile time?) it blocks the thread and reads the data from disk to populate the localStorage memory for reference. Or maybe it looks for data on disk if the domain has stored data in the past and so it knows localstorage is used there already.

In either case, say we insert an iframe into the page, which is dynamically loaded with a different domain as its source. If we use this iframe to do all the localstorage reading and writing via postMessage as described in this post (http://www.nczonline.net/blog/2010/09/07/learning-from-xauth-cross-domain-localstorage/), wouldn’t the thread only be blocked as soon as we load the iframe? Does this give us an asynchronous method of using localstorage? Or am I totally off?

September 17th, 2012 at 16:29
Stan

Just try and implement some simple joins and grouping in your data and you will see why IndexedDB is a terrible choice. WebSQL was the way to go.

October 10th, 2012 at 06:37
Todd Blanchard

So because SQLite is so good that everybody chose it and writing a standards document was superfluous – we are not going to get a real database on the browser? The choice to drop spec work was TWO YEARS AGO. I now sit trying to develop an offline web app for smartphones and desktop browsers and my choice is what exactly?

On the server, we are using a SQL database. We have written some very nifty complex queries around data to provide interesting VIEWS of the data. These queries do a lot of JOINS in interesting ways. It is very convenient to carve out a few tables from the back end database and project fragments of them into the browser’s local SQL database and then we just THE SAME QUERIES on both the server and the client (with minor syntactic differences – easily dealt with).

IndexedDB lacks JOINS. Thus, it lacks adequate power to run my app. It is not a solution. For people who can’t come to grips with SQL, perhaps a different career might suit you better. Failing that, we have the IndexedDBShim over on github – a simple library providing IndexedDB api on top of WebSQL. This is easy to do because IndexedDB is LESS CAPABLE than WebSQL.

Today – TWO YEARS LATER – we have made zero progress and I cannot support my app on a simple key-value store lacking JOINS.

November 9th, 2012 at 10:53
1. Pete
  
  Your choice for offline smartphones apps is WebSQL because almost every device is powered by Webkit which supports WebSQL.
  
  November 10th, 2012 at 01:48
  1. Todd Blanchard
    
    Yes but my choice on desktop is not so clear
    
    November 10th, 2012 at 12:01
Red

@Todd, I agree with you in 100%.

Reject ISO standard because there is no perfect implementation? So what? There are no perfect implementations of C++, Javascript, HTML, POSIX either.

Without SQL we will have to go back to the (binary) trees and hashes.

In my opinion this move prooved Mozilla is not mature. It forced me to go back to native developement.

November 10th, 2012 at 12:54
Ivan

@Pete, Todd, Red,

Any chance an effort to revive WebSQL even as a de-facto standard?

It’s funny to hear that WebSQL and even localstorage seem to doing alright with webkit in mobile environments, where even less resources are available.
There is a good number of reasons why sqlite was chosen as one of the original implementations, not the least, it has proven its capability over time in real world applications and its limitations are well understood but workable.

@Chris,
As HTML5 evangelist, you should be leading your team on solving the implementation issues for your browser and explaining to the community how your mozilla platforms are going to be the best to support html5, whether its on the desktop or in mobile.

December 2nd, 2012 at 21:01
Marco

“Any chance an effort to revive WebSQL even as a de-facto standard?”

This is something I’d be extremely interested in following, or at least hearing about. The arguments and bantering can only do so much, and they only supplement what most of us already realize: web developers want, if not need, WebSQL.

It’s time we start figuring out how to get this back into the spec process.

December 16th, 2012 at 04:40
1. Red
  
  The main argument against WebSQL was that there was only implementation based on Sqlite.
  
  Writing a server side implementation based on Rhino and JDBC is very easy (I have already written something similar a few years ago, so I know it’s not a problem).
  
  If it really is a major obstacle, WebSQL should be easily revived.
  
  Unfortunately, main opponent was Oracle and Microsoft, both database producers… These companies have a lot to lose in the popularization of portable SQL
  
  February 5th, 2013 at 04:17
  1. JulienW
    
    Nope, the main argument is that it’s error-prone and subject to SQL-injections.
    
    February 5th, 2013 at 05:40
    1. Pete
      
      If it is subject to SQL injections then the implementation is wrong. But the specifications deal very good with arguments in SQL calls.
      
      And maybe you can enlighten my when you explain what you mean by error-prone exactly.
      
      Bottomline: Dropping WebSQL in favor if indexDB was/is one of the biggest mistakes of Mozilla within the last couple of years.
      
      February 5th, 2013 at 06:22
      1. JulienW
        
        > And maybe you can enlighten my when you explain what you mean by error-prone exactly.
        
        I was mainly thinking of String concatenation. I _know_ that you can use SQL with auto-escaping arguments API but the fact is that you can also use it using String concatenation. So developer will do this. So the web will be more insecure.
        
        IndexedDB does not permit this by design.
        
        > Bottomline: Dropping WebSQL in favor if indexDB was/is one of the biggest mistakes of Mozilla within the last couple of years.
        
        Well, no.
        
        I do agree there are limitations in IndexedDB. I believe this will need to be expanded (especially with JOIN stuff) but I believe this is the direction to go.
        
        February 5th, 2013 at 06:48
    2. Todd Blanchard
      
      >Nope, the main argument is that it’s error-prone and subject to SQL-injections.
      
      And this is a problem why?
      
      It is a local database possessed entirely by ONE user. If the user wants to hack his OWN database I don’t really see the problem if the user wants to mess up his own database.
      
      >I was mainly thinking of String concatenation. I _know_ that you can use SQL with auto-escaping arguments API but the fact is that you can also use it using String concatenation.
      
      You haven’t used WebSQL I think. The api is totally based on bindings (you don’t “auto-escape” – you “bind” data to variable placeholders).
      
      >I do agree there are limitations in IndexedDB. I believe this will need to be expanded (especially with JOIN stuff) but I believe this is the direction to go.
      
      This “discussion” has been going on for a year now. IndexedDB has appeared on no more platforms, not improvements towards relational capabilities have appeared. And the oft-repeated argument of being able to “emulate sql on top of IndexedDB” hasn’t appeared either and if it would, it has all the same non-problems you object to.
      
      I find you’re arguments, and IndexedDB completely uncompelling.
      
      February 5th, 2013 at 11:15
      1. Pete
        
        > I _know_ that you can use SQL with auto-escaping arguments API but the fact is that you can also use it using String concatenation. So developer will do this. So the web will be more insecure.
        
        Sorry, but this is simply stupid. You can use it for any other API. Also there are guys out there, who submit complete SQL statements to the _server_. What do you want to do with this. Prevent HTTP GET/POST?
        
        February 5th, 2013 at 11:23
      2. JulienW
        
        > It is a local database possessed entirely by ONE user. If the user wants to hack his OWN database I don’t really see the problem if the user wants to mess up his own database.
        
        Well, you must know that a normal web app can use input from a variety of sources: url parameters, other users content, etc. So this is not only one user with his database.
        
        > You haven’t used WebSQL I think. The api is totally based on bindings (you don’t “auto-escape” – you “bind” data to variable placeholders).
        
        I admit I haven’t. “binding” is the correct word, thank you (i’m sorry, english is not my native language). Still, some dev will concatenate strings to generate SQL queries. Because that’s possible.
        
        > IndexedDB has appeared on no more platforms
        What about Internet Explorer 10 ? Chrome ? Even the Blackberry 10 browser seems to support it. Really, the only platforms that does not improve are Safari and Opera here.
        
        February 5th, 2013 at 11:28
    3. Red
      
      @JulienW any text based format is vulnerable to injection attack *if* *used* *improperly*. WebSQL uses data binding. It is safe.
      
      According to W3C (http://www.w3.org/TR/webdatabase/) the main problem is SQLite only implementation:
      
      “The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations ”
      
      Stop thinking about browsers only. Javascript on server side is getting more and more popular. This means: We need SQL API.
      
      February 5th, 2013 at 13:07
      1. Julien Wajsberg
        
        Red> I see no problem having SQL on the server. But then I believe we need more than WebSQL (ie something with the features of a specific version of SQLite). We need something more generic to be able to use several DB. And you know what ? we have it :) http://nodejsdb.org/
        
        > WebSQL uses data binding. It is safe.
        Yeah, just like you can use data binding in all server-side environment for some time now. Still we have SQL injections.
        
        Pete> Hey, this is a so nice way to be right. Clap clap.
        
        IndexedDB is here, deal with it, learn it, learn its strengths, use it. That won’t change soon.
        
        February 6th, 2013 at 00:56
    4. Red
      
      @Julien
      The purpose of using Javascript on server side is to have single API. We have WebWorkers, DOM, etc, etc… However IndexedDB will never be taken seriously for many reasons. On the other hand, the implementation of WebSQL can be written by single person.
      
      Implementation for Node.js is… implementation, not a standard. There is no way to run the same code on another server or browser. I have my own code, which I feel pretty good. Unfortunately, its portability is zero, null.
      
      Your arguments about SQL Injections are flawed. Its FUD spreading, nothing more.
      
      There are many uneducated developers who write stupid things, as a result, we have many HTML injections too. I don’t see movement to abandon html because of this?
      
      Have you even saw the specifications I mentioned? There is a short section on the said issue.
      
      February 6th, 2013 at 06:33
Pete

> I admit I haven’t

Thanks for clarifying your qualification

February 5th, 2013 at 12:22
1. Todd Blanchard
  
  >What about Internet Explorer 10 ? Chrome ? Even the Blackberry 10 browser seems to support it. Really, the only platforms that does not improve are Safari and Opera here.
  
  Which is the entire significant mobile market. The one place where network connections are so intermittent, slow, and expensive that having offline storage really pays off and offline operation is routine. Way to prioritize implementation.
  
  We’ve decided that we are going to support offline operation only on mobile devices with websql and require network otherwise. Making our app with all its different views performant without SQL is just not practical.
  
  February 6th, 2013 at 08:14

Comments are closed for this article.

Hacks

There is no simple solution for local storage

By Chris Heilmann

The main issue: terrible performance

Alternatives

Now what?

About Chris Heilmann

124 comments

There is no simple solution for local storage

By Chris Heilmann

The main issue: terrible performance

Alternatives

Now what?

About Chris Heilmann

Learn the best of web development

Thanks! Please check your inbox to confirm your subscription.