7.2 Local storage

By default, Deno provides access to local storage for persistently storing data that survives Deno runtime restarts. The data is stored in a file on the disk. The local storage implementation doesn't support saving data remotely. This problem has been solved by Deno KV, which will be topic of the next book revision.

The local storage (and session storage too) is internally supported by SQLite. The users would never know or see SQLite. For them, it is a simple & local KV storage. The local storage mechanism is a simple KV store that reside on the disk.

An overview of the local storage architecture is as follows:

The user applications call the local storage APIs provided by Deno. The local storage APIs in JS space are supported by the OPs in rust space. The local storage APIs in rust, in turn, calls the SQLite APIs, which works with the database file on the disk.

In this section, we'll learn:

The location of the database
The queries used to work with the database

For all the interactions with the SQLite database, Deno uses rusqlite crate.

Location of the database

The default location of the SQLite database that would hold the data for local storage is:

On linux, the location of database is $HOME/.cache/deno
On Windows, the location of database is %LOCALAPPDATA%/deno
On Mac, the location of database is $HOME/Library/Caches/deno

In short, the location of the database is inside the cache folder that is used by Deno for other purposes.

Unless used at least once, Deno doesn't create any database file. There is no point in creating database files on the disk if the applications on that server have never or are never going to use it.

~/Library/Caches/deno/location_data: ls -l
total 0

As soon as any of the storage API is called, Deno ensures that a database exists. This is achieved through the following function (get_webstorage):

fn get_webstorage(
  state: &mut OpState,
  persistent: bool,
) -> Result<&Connection, AnyError> {
  let conn = if persistent {
    if state.try_borrow::<LocalStorage>().is_none() {
      let path = state.try_borrow::<OriginStorageDir>().ok_or_else(|| {
        DomExceptionNotSupportedError::new(
          "LocalStorage is not supported in this context.",
        )
      })?;
      std::fs::create_dir_all(&path.0)?;
      let conn = Connection::open(path.0.join("local_storage"))?;
      // Enable write-ahead-logging and tweak some other stuff.
      let initial_pragmas = "
        -- enable write-ahead-logging mode
        PRAGMA journal_mode=WAL;
        PRAGMA synchronous=NORMAL;
        PRAGMA temp_store=memory;
        PRAGMA page_size=4096;
        PRAGMA mmap_size=6000000;
        PRAGMA optimize;
      ";

      conn.execute_batch(initial_pragmas)?;
      conn.set_prepared_statement_cache_capacity(128);
      {
        let mut stmt = conn.prepare_cached(
          "CREATE TABLE IF NOT EXISTS data (key VARCHAR UNIQUE, value VARCHAR)",
        )?;
        stmt.execute(params![])?;
      }
      state.put(LocalStorage(conn));
    }

    &state.borrow::<LocalStorage>().0
  } else {
    if state.try_borrow::<SessionStorage>().is_none() {
      let conn = Connection::open_in_memory()?;
      {
        let mut stmt = conn.prepare_cached(
          "CREATE TABLE data (key VARCHAR UNIQUE, value VARCHAR)",
        )?;
        stmt.execute(params![])?;
      }
      state.put(SessionStorage(conn));
    }

    &state.borrow::<SessionStorage>().0
  };

  Ok(conn)
}

This function takes the following actions:

Creates a database file
Establish a connection to the database
Apply some default settings
Create a table in the database

The default name for the table is 'data'. This table contains exactly two columns:

key
value

Both key and values are strings, as suggested by the web storage standards. The table is created only if it doesn't exist. If it exists, the table is left untouched. This ensures that the data present in the table remains intact.

As soon as any of the local storage APIs are used, we'll see database files in the storage directories. If we use repl, there will be a storage directory for repl that gets used for all repl sessions.

$ deno
Deno 1.40.2
exit using ctrl+d, ctrl+c, or close()
REPL is running with all permissions allowed.
To specify permissions, run `deno repl` with allow flags.
> localStorage.getItem("abcd");
null

Now, there is a directory that holds the database files (there are more than one).

~/Library/Caches/deno/location_data: ls -ltr
total 0
drwxr-xr-x  5 mayankc  staff  160 Jan 27 18:11 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14

~/Library/Caches/deno/location_data: ls -l 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14/
total 104
-rw-r--r--  1 mayankc  staff   4096 Jan 27 18:11 local_storage
-rw-r--r--  1 mayankc  staff  32768 Jan 27 18:11 local_storage-shm
-rw-r--r--  1 mayankc  staff  12392 Jan 27 18:11 local_storage-wal

We can set some item in the local storage for repl, and try to get it in another repl session.

// REPL SESSION

$ deno 
Deno 1.40.2
exit using ctrl+d, ctrl+c, or close()
REPL is running with all permissions allowed.
To specify permissions, run `deno repl` with allow flags.
> localStorage.setItem("k1", "v1");
undefined
> 

// STORAGE DIRECTORY

~/Library/Caches/deno/location_data: ls
92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14

// ANOTHER REPL SESSION

$ deno 
Deno 1.40.2
exit using ctrl+d, ctrl+c, or close()
REPL is running with all permissions allowed.
To specify permissions, run `deno repl` with allow flags.
> localStorage.getItem("k1");
"v1"
> 

// STORAGE DIRECTORY

~/Library/Caches/deno/location_data: ls
92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14

Custom databases

The local storage in the browser is defined as:

The localStorage read-only property of the window interface allows you to access a Storage object for the Document's origin; the stored data is saved across browser sessions.

This means that the local storage is tied to the origin of the URL in the browser's address bar. This also means that there is separate storage available for each origin.

Deno takes the same concept forward and makes a different database available to different origins. However, there is no concept of address bar here. Therefore, Deno takes origin from two sources:

The main program name
The --location argument if specified

This ensures that each application has their own database.

// storageTest.js

const d = localStorage.getItem("k1");
console.log(d);

// Test

$ deno run storageTest.js 
null

// Storage directory

~/Library/Caches/deno/location_data: ls -l
total 0
drwxr-xr-x  3 mayankc  staff  96 Jan 27 18:28 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14
drwxr-xr-x  3 mayankc  staff  96 Jan 27 18:29 ccd539c6c7447c6d3830b2816f029f5dda4a347e383a608fa7fef00e36432b29

As we can see, in addition to repl database, there is a new one specifically created for this application. A new database gets created for each application that attempts to use local storage.

// storageTest1.js

const d = localStorage.getItem("k1");
console.log(d);

// Test

$ deno run storageTest1.js 
null

// Storage directory

~/Library/Caches/deno/location_data: ls -l
total 0
drwxr-xr-x  3 mayankc  staff  96 Jan 27 18:31 845857c5ba27e9912cfc1de8b6f8261b86fd13e857aa12f044d59895ebac4ccd
drwxr-xr-x  3 mayankc  staff  96 Jan 27 18:28 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14
drwxr-xr-x  3 mayankc  staff  96 Jan 27 18:29 ccd539c6c7447c6d3830b2816f029f5dda4a347e383a608fa7fef00e36432b29

Within an application, there is a way to segregate data further by specifying the --location argument. This creates a separate database for each location, which is separate from the database that gets created if location is not specified. Each application can open and use a different database by utilizing the --location argument.

// Using location

$ deno run --location https://deno.com storageTest.js 
null

// Storage directory

~/Library/Caches/deno/location_data: ls -l
total 0
drwxr-xr-x  3 mayankc  staff  96 Jan 27 18:31 845857c5ba27e9912cfc1de8b6f8261b86fd13e857aa12f044d59895ebac4ccd
drwxr-xr-x  3 mayankc  staff  96 Jan 27 18:28 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14
drwxr-xr-x  3 mayankc  staff  96 Jan 27 18:29 ccd539c6c7447c6d3830b2816f029f5dda4a347e383a608fa7fef00e36432b29
drwxr-xr-x  3 mayankc  staff  96 Jan 27 18:32 d1e195fe7f09a0f7760e7ebbe4e23ac702228747ddf77b771b3055c3e7e5d29f

// Using a different location

$ deno run --location https://denodeploy.com storageTest.js 
null

// Storage directory

~/Library/Caches/deno/location_data: ls -l
total 0
drwxr-xr-x  3 mayankc  staff  96 Jan 27 18:31 845857c5ba27e9912cfc1de8b6f8261b86fd13e857aa12f044d59895ebac4ccd
drwxr-xr-x  3 mayankc  staff  96 Jan 27 18:32 90a9a993d66b0c1d647ea90e4039ebd4733927f43fa583c3636f5a338be65a3e
drwxr-xr-x  3 mayankc  staff  96 Jan 27 18:28 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14
drwxr-xr-x  3 mayankc  staff  96 Jan 27 18:29 ccd539c6c7447c6d3830b2816f029f5dda4a347e383a608fa7fef00e36432b29
drwxr-xr-x  3 mayankc  staff  96 Jan 27 18:32 d1e195fe7f09a0f7760e7ebbe4e23ac702228747ddf77b771b3055c3e7e5d29f

That's all about the database files. We'll now turn our attention to the internals of a couple of heavily used local storage APIs: getItem and setItem.

Get item from local storage

Getting an item from local storage is one of the two most used local storage APIs. The other one is setItem, which we'll see in the next section. This is a simple API with only one input: Key.

We'll trace the journey starting from the user space.

JS space

The getItem API code is as follows:

  getItem(key) {
    webidl.assertBranded(this, StoragePrototype);
    const prefix = "Failed to execute 'getItem' on 'Storage'";
    webidl.requiredArguments(arguments.length, 1, prefix);
    key = webidl.converters.DOMString(key, prefix, "Argument 1");

    return op_webstorage_get(key, this[_persistent]);
  }

This is a very simple function that

Validates the presence of input key
Defers the work to a rust OP called op_webstorage_get

Rust space

The OP function called op_webstorage_get is a synchronous function, that runs a query in the SQLite database and sends the result back. Note that the getItem and setItem are synchronous functions.

#[op2]
#[string]
pub fn op_webstorage_get(
  state: &mut OpState,
  #[string] key_name: String,
  persistent: bool,
) -> Result<Option<String>, AnyError> {
  let conn = get_webstorage(state, persistent)?;

  let mut stmt = conn.prepare_cached("SELECT value FROM data WHERE key = ?")?;
  let val = stmt
    .query_row(params![key_name], |row| row.get(0))
    .optional()?;

  Ok(val)
}

The code of op_webstorage_get is much simpler than our expectations. This function:

Opens a connection to database (if not already open)
Prepares a query
Executes the query
Return the results back

The important part here is the query, which is:

SELECT value FROM data WHERE key = ?

As simple as it looks, that's all about getting an item from local storage. Simply run a query in the SQLite data and return the result. This is no different from running a query directly (which we'll do shortly).

Set item in the local storage

Setting an item into local storage is the other of the two most used local storage APIs.This is a simple API with two inputs: Key & Value.

Again, we'll trace the journey starting from the user space.

JS space

The setItem API code is as follows:

  setItem(key, value) {
    webidl.assertBranded(this, StoragePrototype);
    const prefix = "Failed to execute 'setItem' on 'Storage'";
    webidl.requiredArguments(arguments.length, 2, prefix);
    key = webidl.converters.DOMString(key, prefix, "Argument 1");
    value = webidl.converters.DOMString(value, prefix, "Argument 2");

    op_webstorage_set(key, value, this[_persistent]);
  }

This is a very simple function that

Validates the presence of input key & value
Defers the work to a rust OP called op_webstorage_set

Rust space

The OP function called op_webstorage_set is a synchronous function, that runs a DML operation in the SQLite database. Note that the getItem and setItem are synchronous functions.

#[op2(fast)]
pub fn op_webstorage_set(
  state: &mut OpState,
  #[string] key: &str,
  #[string] value: &str,
  persistent: bool,
) -> Result<(), AnyError> {
  let conn = get_webstorage(state, persistent)?;

  size_check(key.len() + value.len())?;

  let mut stmt = conn
    .prepare_cached("SELECT SUM(pgsize) FROM dbstat WHERE name = 'data'")?;
  let size: u32 = stmt.query_row(params![], |row| row.get(0))?;

  size_check(size as usize)?;

  let mut stmt = conn
    .prepare_cached("INSERT OR REPLACE INTO data (key, value) VALUES (?, ?)")?;
  stmt.execute(params![key, value])?;

  Ok(())
}

The code of op_webstorage_set is also much simpler than our expectations, but slightly longer than the get function. There are multiple size related checks.

Mirroring browser implementations, Deno enforces a 10MB size limit on local storage per origin. This constraint ensures database integrity and prevents excessive storage consumption. The op_webstorage_set function diligently safeguards database sanity by preventing size thresholds from being exceeded.

This op_webstorage_set function:

Opens a connection to database (if not already open)
Checks if the length of key & value is not more than 10MB
Checks if current database size is less than 10MB
Executes the DML operation to insert or update record in the database

The important part here is the DML statement, which is:

INSERT OR REPLACE INTO data (key, value) VALUES (?, ?)

As simple as it looks, that's all about setting an item into local storage. This API returns nothing back.

Other APIs

Before closing this section, we'll take a quick at a couple of other interesting APIs: removeItem & clear.

The removeItem API removes the record from the database. It issues a DELETE command to make this happen.

  removeItem(key) {
    webidl.assertBranded(this, StoragePrototype);
    const prefix = "Failed to execute 'removeItem' on 'Storage'";
    webidl.requiredArguments(arguments.length, 1, prefix);
    key = webidl.converters.DOMString(key, prefix, "Argument 1");

    op_webstorage_remove(key, this[_persistent]);
  }

#[op2(fast)]
pub fn op_webstorage_remove(
  state: &mut OpState,
  #[string] key_name: &str,
  persistent: bool,
) -> Result<(), AnyError> {
  let conn = get_webstorage(state, persistent)?;

  let mut stmt = conn.prepare_cached("DELETE FROM data WHERE key = ?")?;
  stmt.execute(params![key_name])?;

  Ok(())
}

Lastly, the clear API removes all records from the table. This API also issues the DELETE command without any inputs.

  clear() {
    webidl.assertBranded(this, StoragePrototype);
    op_webstorage_clear(this[_persistent]);
  }

#[op2(fast)]
pub fn op_webstorage_clear(
  state: &mut OpState,
  persistent: bool,
) -> Result<(), AnyError> {
  let conn = get_webstorage(state, persistent)?;

  let mut stmt = conn.prepare_cached("DELETE FROM data")?;
  stmt.execute(params![])?;

  Ok(())
}

Database file

Fulfilling our earlier promise and satisfying our curiosity, we'll briefly examine the database file within the SQLite shell. The local_storage file, residing within any database folder in the root database directory, can be directly opened using the SQLite shell.

~/Library/Caches/deno/location_data: ls -ltr
total 0
drwxr-xr-x  5 mayankc  staff  160 Jan 27 21:33 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14
~/Library/Caches/deno/location_data: cd 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14/
~/Library/Caches/deno/location_data/92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14: ls -ltr
total 136
-rw-r--r--  1 mayankc  staff   4096 Jan 27 21:33 local_storage
-rw-r--r--  1 mayankc  staff  32768 Jan 27 21:33 local_storage-shm
-rw-r--r--  1 mayankc  staff  28872 Jan 27 21:33 local_storage-wal

First, we'll set a couple of keys into the local storage using deno repl:

$ deno 
Deno 1.40.2
exit using ctrl+d, ctrl+c, or close()
REPL is running with all permissions allowed.
To specify permissions, run `deno repl` with allow flags.
> localStorage.setItem('k1', 'v1');
undefined
> localStorage.setItem('k2', 'v2');
undefined

The file named local_storage can be opened directly inside the SQLite shell:

~/Library/Caches/deno/location_data/92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14: sqlite3 local_storage
SQLite version 3.39.5 2022-10-14 20:58:05
Enter ".help" for usage hints.
sqlite> .headers on
sqlite> .schema data
CREATE TABLE data (key VARCHAR UNIQUE, value VARCHAR);
sqlite> select * from data;
key|value
k1|v1
k2|v2
sqlite>

As you can see, this is the same type of SQLite database file that you may have used in the past. We've verified that a table named "data" exists with two keys that we set earlier. We've also verified that the table contains two rows matching the data we set through the Deno REPL.

This concludes our exploration of local storage internals in Deno. The next section is about session storage, building upon the detailed background we've established. The focus on session storage will be concise, leveraging your newly acquired understanding of local storage mechanisms.

Previous7.1 Introduction Next7.3 Session storage

Last updated 2 months ago