Today's Web applications handle large amounts of data processing on the client side. They may even need to be able to operate offline. These demands things go a long way toward explaining why client-side data storage is vital to next-generation Web-based applications.
Until recently, however, cookies were pretty much the only way to store data in a user's browser. If a Web app needed repeated access to some bit of data, it needed to ship that data to the server, then make repeated requests to the server, or store that data in cookies.
Cookies offer only limited storage space -- a maximum of 4K or 4096 bytes -- so large amounts of data had to be broken into cookie-size chunks and managed explicitly and directly.
This isn't a workable approach to storage allocation and management. Obviously, a new approach was needed.
It Doesn't Take Much for Cookies to Crumble
Web browsers were originally nothing more than applications for loading documents via HTTP and parsing HTML. Shortly after the first Netscape browser appeared, however, it became obvious that much more was both possible and necessary, but that some mechanism for tracking state using the inherently stateless HTTP protocol would be needed. Lou Montulli thus implemented the browser cookie (also known as the "magic cookie") in 1994, for version 0.9b of the Mosiac Netscape browser.
Cookies, along with access to server-side scripts that the Common Gateway Interface (CGI) provided, enabled the earliest Web applications. Ultimately, this started us down a path toward browsers transforming into a universal application platform.
Alas, cookies are flawed. As noted, they can store only very small amounts of data, and they're vulnerable to numerous types of attack, which limits their usefulness for storing personal or sensitive data.
Cookies travel from the browser to the server with every HTTP request. Say a Web page contains four images, an external CSS document and a JavaScript document. Set a 4K cookie on that domain and the browser transfers that cookie to the server once for the HTML page, once for each image, once for the CSS document and once for the JavaScript document.
Further compounding the problem is the observation that this theoretical 4K cookie is transferred from the browser to the server; most users have asynchronous Internet connections, whereupon upload speeds are slower than download speeds. Inevitably, transferring cookie data inside HTTP headers creates unnecessary overhead.
Because of these limitations, most cookies are significantly smaller than 4K in size. Google recommends that cookies occupy no more than 400 bytes (or 200 characters) to maximize performance. They also recommend that static files such as images, CSS, and JavaScript come from a distinct domain with cookies disabled.
With so many negatives attached to using cookies for local storage, it's no wonder that various competing proposals have emerged to get this job done right. In recent months, two such proposals have been on a fast track to W3C recommendation status -- which makes browser support for local storage good and about to get better.
There are four main methods for storing large amounts of client-side data today: Web SQL, IndexedDB, Web Storage and Application Cache. The sections that follow examine each method individually and explore basic aspects of their programming and operation.
Web SQL: Familiar (But Obsolete?) Database Creation, Execution
Web SQL is an API for storing data in a database and retrieving it using SQL. Until recently, Safari, Chrome, and Opera supported Web SQL over the competing IndexedDB standard. However, in 2010, SQLite was the only database that worked with Web SQL, and the W3C ceased working on this specification, citing a lack of implementations as its reason.
That said, Web SQL works in a cool way, so let's look at some sample code.
Working with Web SQL databases will be familiar to anyone who's worked with relational databases and SQL. The first step in using a database is to create it and open it. If you don't want to specifically create a database, you can just start using a database, and the API creates it for you.
Here's some code for creating a database:
var db = openDatabase('cats', '1.0',
'a catalog of my cats', 2 *1024 * 1024);
Left to right, the parameters taken by openDatabase are database name, version number, text description, and estimated database size.
Once you've created a database, you can start using it. Executing SQL on a WebSQL database is as easy as creating a transaction object and then executing it:
db.transaction(function (tx) {
tx.executeSql('CREATE TABLE cats (id unique, name)');
tx.executeSql('INSERT INTO cats (id,name) VALUES (1,"Mr. Jones")');
});
Although Safari, Chrome, Opera and Mobile Safari still support this API, nothing new has happened with Web SQL since 2010, so it's unlikely to emerge as a standard for local storage.
Web Storage: Benefits of Cookies Without the Bloat
Web Storage provides a simple means to store key/value pairs in a user's browser. But its similarities with cookies end there.
Web Storage is persistent. Once a value is stored, it doesn't disappear or expire until it's explicitly removed by the application or the user.
Web Storage can handle large amounts of data. Current browsers limit total size per storage area to 5MB.
Web Storage doesn't depend on the server and sends no data to the server. You're free to store data locally and sync it with the server asynchronously, of course, but Web Storage works equally well and is just as useful offline as it is online.
Web Storage provides four primary methods -- getItem(key); setItem(key,value); removeItem(key); and clear().
But...: Web Storage Loophole Can Be Abused to Fill Hard Disks With Junk Data
Lastly, Web Storage includes of two different types of storage: SessionStorage and LocalStorage.
SessionStorage limits the scope of data saved in the current browser window to just that browser window. When used with an ecommerce application, for example, using sessionStorage to record the contents of a user's shopping cart eliminates the potential for accidental double purchases.
LocalStorage, meanwhile, persists across windows and tabs within a single browser. So, if you have the same site open in three windows in Chrome, all could all share the same localStorage container. If you have three different sites at different domains open, each one needs its own container. Likewise, if you have the same site open in different browsers, each browser needs its own container, since they don't share a common runtime context.
To set a new key-value pair then retrieve it, you could use the following JavaScript:
//first set firstname equal to Sparky.
localStorage.setItem( "firstname", "Sparky" );
//next, get the value of firstname (hint, it will be Sparky).
localStorage.getItem( "firstname" );
The Web Storage API reached W3C Recommendation status in summer 2013. Going forward, Web Storage can pretty much be used anywhere you would normally have used cookies.
News: W3C's Web Storage Technology Goes Live
But Web Storage is also capable of much more. If your data sets aren't enormous, Web Storage provides what's perhaps the simplest way -- even easier than cookies -- to set and retrieve key-value pairs in a browser.
IndexedDB: Searchable, and Without File Size Limits
Indexed Database is an API for storing and retrieving data using an indexed, transactional database of key-value pairs on the user's computer. IndexedDB provides faster, more sophisticated data storage and retrieval than simple key-value pair storage of the type available from Web Storage or cookies.
The IndexedDB API, like Web Storage, took a major step forward in the Web standards process this summer, becoming a W3C candidate recommendation in July 2013.
IndexedDB offers four specific benefits over Web Storage:
Indexed data can be searched efficiently.
Databases allow multiple values to be stored as a key, whereas key-value data requires every key to be unique.
Transactional databases offer some protection against system and application failures. If a transaction doesn't successfully complete, it can be rolled back.
IndexedDB databases impose no size limitations. In Firefox, the browser will ask for permission to expand a database beyond 50MB, but the sky's the limit (actually, the volume or disk drive's the limit) for storing data in IndexedDB.
All major browsers except Safari currently support IndexedDB. Because Safari supports Web SQL, however, it's possible to use an IndexedDB polyfill (called a shim) that implements the features and syntax of IndexedDB using Web SQL.
The first step in using IndexedDB is to open a database.
var request = indexedDB.open("myDatabase");
After you've created the database, you can create an object store (which is something very much like a table) and add data to it. Assume we have the following data:
const petData = [
{ id: "00-01", firstname: "Butters", age: 2, type: "dog" },
{ id: "00-02", firstname: "Sammy", age: 2, type: "dog" }
];
We can then create a data store and use it with the following code. Note the onupgradeneeded handler: This is the method to call when you need to change the structure of your database.
request.onupgradeneeded = function(event) {
var db = event.target.result;
var objectStore = db.createObjectStore("customers", {keyPath: "id"});
for (var i in customerData) {
objectStore.add(customerData[i]);
}
}
How-to: Improve Application Performance and Reduce Latency
IndexedDB excels at searching large data sets and can speed up Web app performance by moving structured data to the client side wherever possible. It's approaching W3C Recommendation status and is usable in every browser -- albeit with some differences between implementations and, as noted, only through a polyfill in Safari.
Application Cache: Making Offline Client-Side Storage Happen
The Application Cache isn't like other client-side data storage APIs listed here, but it's worth mentioning, as it's an important component in enabling offline client-side Web apps.
The Application Cache uses a cache manifest. This is a simple text document listing resources that should and shouldn't be cached, to tell the browser to download certain files, hold onto them and use the cached version rather than make a request to the server. Every major Web browser supports the Application Cache.
To use the Application Cache, save a text file with the extension .appcache in the root of the website containing the files you want to cache. Depending on the Web server you're using, you may need to create a custom MIME type for .appcache files to make sure they're served to the browser correctly and can be read as application cache files.
Here's an example of a cache manifest file:
CACHE MANIFEST
CACHE:
/css/styles.css
/js/javascript.css
/img/logo.gif
FALLBACK:
/img/weathertoday.png /img/weathernotavailable.png
NETWORK:
*
And here's what each part does:
The CACHE section tells the browser which resources to cache for viewing offline. These files will be cached until the cache manifest changes. Remember that requirement; it's important.
The FALLBACK section tells the browser what files to display instead of a non-cached resource. For example, in the FALLBACK section above, a graphic that presumably says that the latest weather isn't available in offline mode will display if the latestweather.png graphic can't be downloaded.
The NETWORK section tells the browser which resources are only available in online mode. An asterisk indicates that anything that isn't cached is a network resource.
Related: 9 Programming Tools Tuned for JavaScript
More: 17 JavaScript Tools for the HTML5 Generation
Application Cache is a handy tool with almost no downsides if used correctly. If you simply cache everything on your site, you (and your site visitors) will soon be wondering why the content never changes. If you cache only resources that don't change very often, or are diligent about keeping your cache manifest up to date and releasing a new version when you update files, then Application Cache will make your application work offline as well as be much more responsive in online mode.
Local browser storage has undergone a major upgrade in the last few years. The variety and similar names of the different implemented APIs and recommendations have led to quite a bit of confusion over what's OK to use now versus what should be left alone. The bottom line is that different ways of storing data in the browser are available -- and each has its own place.
The days of developers trying to use cookies for more than sending very simple and very small name-value pairs to the server, however, are over. Today, much better options are available.
Chris Minnick runs a Web design and development company and regularly teaches HTML5 classes for Ed2Go. Ed Tittel is a full-time freelance writer and consultant who specializes in Web markup languages, information security and Windows OSes. Together, Minnick and Tittel are the authors of the forthcoming book Beginning Programming with HTML5 and CSS3 For Dummies, due Sept. 3, as well as numerous other books.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.