5.11 Recursive module loading

Overview

As demonstrated in the preceding section, the process of loading modules recursively is a complex task. The result of this recursive module loading process is a module graph, which holds parsed modules. All the dependencies of these modules are loaded upon the completion of this phase. Going further, let's explore a segment of code within the JavaScript runtime's load_main_module() function that holds significance for the recursive module loading process:

let mut load =
      ModuleMap::load_main(module_map_rc.clone(), &specifier).await?;

while let Some(load_result) = load.next().await {
  let (request, info) = load_result?;
  let scope = &mut self.handle_scope(isolate);
  load.register_and_recurse(scope, &request, info).map_err(
    |e| match e {
      ModuleError::Exception(exception) => {
        let exception = v8::Local::new(scope, exception);
        exception_to_err_result::<()>(scope, exception, false).unwrap_err()
      }
      ModuleError::Other(error) => error,
    },
  )?;
}

Here are the required procedures in this section:

1. Establishing a Recursive Module Loader: To begin, we must construct a module loader that works in a loop, repeatedly calling itself. This loader takes various arguments, including the module loader itself.

2. Getting Modules Ready: Next, we need to make sure our modules are ready for use. This involves two main actions:

   • Fetching: We retrieve the necessary modules, ensuring they are available for the program.

   • Inserting into Graph: We organize these modules within a graphical structure, creating connections between them based on their dependencies.

3. Iterating Through Modules and Registration: After the preparation phase, we proceed to loop through all the modules. This process is also referred to as instantiation or registration. During this loop, each module is formally registered, allowing it to be utilized as part of the program's execution.

The primary function is called "register_and_recurse," and it's quite intricate in its workings:

• It performs a recursive journey through all the modules using a graph-like structure of data.

• During this process, it covers all the distinct modules present.

• The function retrieves all the necessary imports that a module needs.

• This action involves further recursive exploration of all the imports.

pub(crate) fn register_and_recurse(
    &mut self,
    scope: &mut v8::HandleScope,
    module_request: &ModuleRequest,
    module_source: ModuleSource,
  ) -> Result<(), ModuleError> {
    let expected_asserted_module_type = module_source.module_type.into();
    let module_url_found = module_source.module_url_found;
    let module_url_specified = module_source.module_url_specified;

    if module_request.asserted_module_type != expected_asserted_module_type {
      return Err(ModuleError::Other(generic_error(format!(
        "Expected a \"{}\" module but loaded a \"{}\" module.",
        module_request.asserted_module_type, module_source.module_type,
      ))));
    }

    // Register the module in the module map unless it's already there. If the
    // specified URL and the "true" URL are different, register the alias.
    let module_url_found = if let Some(module_url_found) = module_url_found {
      let (module_url_found1, module_url_found2) =
        module_url_found.into_cheap_copy();
      self.module_map_rc.borrow_mut().alias(
        module_url_specified,
        expected_asserted_module_type,
        module_url_found1,
      );
      module_url_found2
    } else {
      module_url_specified
    };

    let maybe_module_id = self
      .module_map_rc
      .borrow()
      .get_id(&module_url_found, expected_asserted_module_type);
    let module_id = match maybe_module_id {
      Some(id) => {
        debug!(
          "Already-registered module fetched again: {:?}",
          module_url_found
        );
        id
      }
      None => match module_source.module_type {
        ModuleType::JavaScript => {
          self.module_map_rc.borrow_mut().new_es_module(
            scope,
            self.is_currently_loading_main_module(),
            module_url_found,
            module_source.code,
            self.is_dynamic_import(),
          )?
        }
        ModuleType::Json => self.module_map_rc.borrow_mut().new_json_module(
          scope,
          module_url_found,
          module_source.code,
        )?,
      },
    };

    // Recurse the module's imports. There are two cases for each import:
    // 1. If the module is not in the module map, start a new load for it in
    //    `self.pending`. The result of that load should eventually be passed to
    //    this function for recursion.
    // 2. If the module is already in the module map, queue it up to be
    //    recursed synchronously here.
    // This robustly ensures that the whole graph is in the module map before
    // `LoadState::Done` is set.
    let mut already_registered = VecDeque::new();
    already_registered.push_back((module_id, module_request.clone()));
    self.visited.insert(module_request.clone());
    while let Some((module_id, module_request)) = already_registered.pop_front()
    {
      let referrer = ModuleSpecifier::parse(&module_request.specifier).unwrap();
      let imports = self
        .module_map_rc
        .borrow()
        .get_requested_modules(module_id)
        .unwrap()
        .clone();
      for module_request in imports {
        if !self.visited.contains(&module_request)
          && !self
            .visited_as_alias
            .borrow()
            .contains(&module_request.specifier)
        {
          if let Some(module_id) = self.module_map_rc.borrow().get_id(
            module_request.specifier.as_str(),
            module_request.asserted_module_type,
          ) {
            already_registered.push_back((module_id, module_request.clone()));
          } else {
            let request = module_request.clone();
            let specifier =
              ModuleSpecifier::parse(&module_request.specifier).unwrap();
            let visited_as_alias = self.visited_as_alias.clone();
            let referrer = referrer.clone();
            let loader = self.loader.clone();
            let is_dynamic_import = self.is_dynamic_import();
            let fut = async move {
              // `visited_as_alias` unlike `visited` is checked as late as
              // possible because it can only be populated after completed
              // loads, meaning a duplicate load future may have already been
              // dispatched before we know it's a duplicate.
              if visited_as_alias.borrow().contains(specifier.as_str()) {
                return Ok(None);
              }
              let load_result = loader
                .load(&specifier, Some(&referrer), is_dynamic_import)
                .await;
              if let Ok(source) = &load_result {
                if let Some(found_specifier) = &source.module_url_found {
                  visited_as_alias
                    .borrow_mut()
                    .insert(found_specifier.as_str().to_string());
                }
              }
              load_result.map(|s| Some((request, s)))
            };
            self.pending.push(fut.boxed_local());
          }
          self.visited.insert(module_request);
        }
      }
    }

    // Update `self.state` however applicable.
    if self.state == LoadState::LoadingRoot {
      self.root_module_id = Some(module_id);
      self.root_asserted_module_type = Some(module_source.module_type.into());
      self.state = LoadState::LoadingImports;
    }
    if self.pending.is_empty() {
      self.state = LoadState::Done;
    }

    Ok(())
  }
}

From the long function above, we can look at two important points:

Get the imports present in a module

while let Some((module_id, module_request)) = already_registered.pop_front()
{
  let imports = self
    .module_map_rc
    .borrow()
    .get_requested_modules(module_id)
    .unwrap()
    .clone();
    
    .....
    
    for module_request in imports {
      ....
      already_registered.push_back((module_id, module_request.clone()));
      ....
    }
}

Add imports to the list and visit them if not already visited.

--

In the next section, we'll explore the process of building module graphs, which will reveal the interconnectedness of modules. We'll also examine the specific module graph related to our example, providing a detailed visual representation of its structure and relationships.