5.11 Recursive module loading
As shown in the previous section, recursive module loading is complicated work. The output of the recursive module loading is a module graph that contains parsed modules. All the dependencies get loaded at the end of this step. Here is the part of code in JS runtime's load_module() which is relevant for recursive module loading:
let load = RecursiveModuleLoad::main(
self.op_state(),
&specifier.to_string(),
code,
loader,
);
let (_load_id, prepare_result) = load.prepare().await;
let mut load = prepare_result?;
while let Some(info_result) = load.next().await {
let info = info_result?;
self.register_during_load(info, &mut load)?;
}
There are some steps here:
- Create a recursive module loader (one of the args is the module loader)
- Prepare modules
- Fetch
- Insert in graph
- Loop through all the modules and register them (also known as instantiation)
Recursive module loading isn't as direct as it is shown in the diagram above. If we try to look at the chain of calls, it'd span over multiple classes. Here is a simplified class diagram for recursive module loading:
The main work of module loading is present in two places:
- JS runtime's load_module
- Program state's prepare_module_load
Others are more or less a chain.
Let's dive into how modules are brought into Deno. This starts with the initialization of RecursiveModuleLoad, followed by a call to the prepare() function of the recursive module loader. Prepare is where the action happens. Here is the code of the prepare() function:
pub async fn prepare(self) -> (ModuleLoadId, Result<Self, AnyError>) {
let (module_specifier, maybe_referrer) = match self.state {
LoadState::ResolveMain(ref specifier, _) => {
let spec =
match self
.loader
.resolve(self.op_state.clone(), specifier, ".", true)
{
Ok(spec) => spec,
Err(e) => return (self.id, Err(e)),
};
(spec, None)
}
LoadState::ResolveImport(ref specifier, ref referrer) => {
let spec = match self.loader.resolve(
self.op_state.clone(),
specifier,
referrer,
false,
) {
Ok(spec) => spec,
Err(e) => return (self.id, Err(e)),
};
(spec, Some(referrer.to_string()))
}
_ => unreachable!(),
};
let prepare_result = self
.loader
.prepare_load(
self.op_state.clone(),
self.id,
&module_specifier,
maybe_referrer,
self.is_dynamic_import(),
)
.await;
match prepare_result {
Ok(()) => (self.id, Ok(self)),
Err(e) => (self.id, Err(e)),
}
}
The module loader (CLI Module Loader) came during the initialization of RecursiveModuleLoad. The loader is an implementation of the base class ModuleLoader.
There are two simple steps in the prepare function:
- Resolve module
- Resolve main module, or
- Resolve the imported module
- Call prepare_load of the loader
In this particular case where we're running a local file, that doesn't have any imports, the only work is about resolving the main module which isn't much. The input and output of the module resolution are the same:
url file:///Users/mayankc/Work/source/deno-vs-nodejs/helloLog.ts
We'll see module resolution in detail in the next program which would have ES imports in the main module.
ModuleLoader's prepare_load() does some bookkeeping and then defer the actual work to the program state's prepare_module_load(). Program state's function does the actual work. The function is quite big. Here is the code:
pub async fn prepare_module_load(
self: &Arc<Self>,
specifier: ModuleSpecifier,
lib: TypeLib,
runtime_permissions: Permissions,
is_dynamic: bool,
maybe_import_map: Option<ImportMap>,
) -> Result<(), AnyError> {
let specifier = specifier.clone();
if lib == TypeLib::DenoWorker || lib == TypeLib::UnstableDenoWorker {
runtime_permissions.check_specifier(&specifier)?;
}
let handler =
Rc::new(RefCell::new(FetchHandler::new(self, runtime_permissions)?));
let mut builder =
GraphBuilder::new(handler, maybe_import_map, self.lockfile.clone());
builder.add(&specifier, is_dynamic).await?;
let mut graph = builder.get_graph();
let debug = self.flags.log_level == Some(log::Level::Debug);
let maybe_config_path = self.flags.config_path.clone();
let result_modules = if self.flags.no_check {
let result_info = graph.transpile(TranspileOptions {
debug,
maybe_config_path,
reload: self.flags.reload,
})?;
debug!("{}", result_info.stats);
if let Some(ignored_options) = result_info.maybe_ignored_options {
warn!("{}", ignored_options);
}
result_info.loadable_modules
} else {
let result_info = graph.check(CheckOptions {
debug,
emit: true,
lib,
maybe_config_path,
reload: self.flags.reload,
})?;
debug!("{}", result_info.stats);
if let Some(ignored_options) = result_info.maybe_ignored_options {
eprintln!("{}", ignored_options);
}
if !result_info.diagnostics.is_empty() {
return Err(anyhow!(result_info.diagnostics));
}
result_info.loadable_modules
};
let mut loadable_modules = self.modules.lock().unwrap();
loadable_modules.extend(result_modules);
if let Some(ref lockfile) = self.lockfile {
let g = lockfile.lock().unwrap();
g.write()?;
}
Ok(())
}
The essence of the above function is to build a module graph. Building a module graph isn't as straightforward as it sounds. Modules are fetched, parsed, and added to the graph as they are processed.
In detail, here are the steps that are taken to prepare the graph:
- Create a fetch handler
- Fetch handler is used to fetch the modules or files
- Even local files need fetching/reading
- Build module Graph
- Deno uses module graphs to keep track of the import tree
- First, a GraphBuilder is created
- Add the root or main module to the graph
- This step is recursive
- It ends by adding all the dependencies into the graph
- Check or transpile graph
It's obvious that the main work is on building a module graph. The building of the module graph comprises fetching, parsing, adding to the graph. The process is recursive.
--
In the next section, we'll see how module graphs are built. We'll also see how the module graph looks like for our example.