5.13 File fetching
In the previous section, we went over how module graphs are built. We'd discussed fetching modules but didn't really go into details. In this section, we'll go over how files are fetched by Deno.
Either the main module or the imports, both are basically files. They need to be fetched from their source which could either be local or remote (HTTP or HTTPS).
There are three places to fetch the file from:
- Local
- The file is located on the disk
- Remote
- The file is not local and can be accessed via HTTP or HTTPS
- Cache
- The file is present in the cache
In code, fetching from cache is a part of fetching from remote. But we've shown it separately to make it clear.
File fetcher fetches the file given a module specifier. File fetcher hides the details by returning the file regardless of the source of the file.
Here is the source of the fetch function:
pub async fn fetch(
&self,
specifier: &ModuleSpecifier,
permissions: &Permissions,
) -> Result<File, AnyError> {
debug!("FileFetcher::fetch() - specifier: {}", specifier);
let scheme = get_validated_scheme(specifier)?;
permissions.check_specifier(specifier)?;
if let Some(file) = self.cache.get(specifier) {
Ok(file)
} else {
let is_local = scheme == "file";
if is_local {
fetch_local(specifier)
} else if !self.allow_remote {
Err(custom_error(
"NoRemote",
format!("A remote specifier was requested: \"{}\", but --no-remote is specified.", specifier),
))
} else {
let result = self.fetch_remote(specifier, permissions, 10).await;
if let Ok(file) = &result {
self.cache.insert(specifier.clone(), file.clone());
}
result
}
}
}
The code is very simple. Fetch file from local or remote based on the location of the file. For the files fetched from the remote, also add them to the internal cache as remote fetch is significantly expensive than local fetch.
If the file is located on the disk, it would be fetched locally. The function is called fetch_local.
fn fetch_local(specifier: &ModuleSpecifier) -> Result<File, AnyError> {
let local = specifier.as_url().to_file_path().map_err(|_| {
uri_error(format!("Invalid file path.\n Specifier: {}", specifier))
})?;
let bytes = fs::read(local.clone())?;
let charset = text_encoding::detect_charset(&bytes).to_string();
let source = strip_shebang(get_source_from_bytes(bytes, Some(charset))?);
let media_type = MediaType::from(specifier);
Ok(File {
local,
maybe_types: None,
media_type,
source,
specifier: specifier.clone(),
})
}
Fetch_local simply reads the file from the disk and returns the source of the file. Local files are not cached.
If the file is located remotely, it would be fetched via an HTTP client. HTTP client could fetch over HTTP or HTTPS. Once the file is fetched, it would be cached as remote fetching is expensive.
The source of remote fetch is a bit long, but it has several interesting pieces.
fn fetch_remote(
&self,
specifier: &ModuleSpecifier,
permissions: &Permissions,
redirect_limit: i64,
) -> Pin<Box<dyn Future<Output = Result<File, AnyError>>>> {
debug!("FileFetcher::fetch_remote() - specifier: {}", specifier);
if redirect_limit < 0 {
return futures::future::err(custom_error("Http", "Too many redirects."))
.boxed_local();
}
if let Err(err) = permissions.check_specifier(specifier) {
return futures::future::err(err).boxed_local();
}
if self.cache_setting.should_use(specifier) {
match self.fetch_cached(specifier, redirect_limit) {
Ok(Some(file)) => {
return futures::future::ok(file).boxed_local();
}
Ok(None) => {}
Err(err) => {
return futures::future::err(err).boxed_local();
}
}
}
if self.cache_setting == CacheSetting::Only {
return futures::future::err(custom_error(
"NotFound",
format!(
"Specifier not found in cache: \"{}\", --cached-only is specified.",
specifier
),
))
.boxed_local();
}
info!("{} {}", colors::green("Download"), specifier);
let file_fetcher = self.clone();
let cached_etag = match self.http_cache.get(specifier.as_url()) {
Ok((_, headers)) => headers.get("etag").cloned(),
_ => None,
};
let specifier = specifier.clone();
let permissions = permissions.clone();
let http_client = self.http_client.clone();
// A single pass of fetch either yields code or yields a redirect.
async move {
match fetch_once(http_client, specifier.as_url(), cached_etag).await? {
FetchOnceResult::NotModified => {
let file = file_fetcher.fetch_cached(&specifier, 10)?.unwrap();
Ok(file)
}
FetchOnceResult::Redirect(redirect_url, headers) => {
file_fetcher
.http_cache
.set(specifier.as_url(), headers, &[])?;
let redirect_specifier = ModuleSpecifier::from(redirect_url);
file_fetcher
.fetch_remote(&redirect_specifier, &permissions, redirect_limit - 1)
.await
}
FetchOnceResult::Code(bytes, headers) => {
file_fetcher.http_cache.set(
specifier.as_url(),
headers.clone(),
&bytes,
)?;
let file =
file_fetcher.build_remote_file(&specifier, bytes, &headers)?;
Ok(file)
}
}
}
.boxed_local()
}
Remote fetching is a recursive call as an HTTP call could result in a redirect. This function recursively calls itself every time there is a redirection.
Here are the steps in detail:
- Check and return an error if the redirect limit has been crossed
- If the specifier is present in the cache
- return from cache
- Fetch file through the HTTP client
- If there is a redirection,
- call fetch_remote with redirected URL
- Else
- return fetched file
Fetch_remote can take up to 10 redirections.
Whenever a file is fetched from remote, we can see the famous Download message on the console:
