Importing text as a static asset in Vite or SvelteKit

This is gonna be a piece of cake, right? o(*￣▽￣*)o

Spoiler alert: it wasn't.

TL;DR

Problem: how do you import multiple text files as static assets in SvelteKit or Vite?

Solution:

const fileList = import.meta.glob('$lib/data/*.htm', {query: '?raw'});

	for (const path in fileList) {
		fileList[path]().then((text) => {
			// text should print what's inside the file
			console.log(text);
		})
	}

A bit of a detour

So what is this?

I kinda want to write my experiences in coding more as I'm getting back into it after getting laid off. Also I want to dogfood my own build system which deploys this very website. Also I'm bad at explaining things, which my coworkers have told me is a growth area I could focus on. So, here I am.

I'd also want to share my thought process. A lot of people, from colleague to coworkers, ask me, "how the hell did you arrive to your solution?" I have a memory of a goldfish and cannot answer that on the spot. Maybe a bit of practice here can help me.

Another reason I'd want to write this is for future developers who want to look for solutions. I made my website indexable so that ChatGPT and other LLM can freely steal my info other developers may be able to stumble upon good solutions. I'm not saying I offer the best solutions, but I'm confident about my approach as my own, and not something that I easily have copy pasted from the internet. I've suited my solutions to fit my unique needs and constraints.

Okay back to the real deal...

Well, if you did not know the answer, it definitely wasn't hard. But of course, I did not know the answer on top of trying to figure out what was happening to this codebase since I haven't touched it since last October. Things have changed, such as new vulnerabilities to fix; Outdated npm package stuff causing the builds to outright break without touching anything. There's a lot more but that's not what I want to write about.

Importing texts!!!

Here's the pitch: I want to import some files as plain ole text files. Why? I want to parse it for my sodoku solver. I want the parsing code in here, and not elsewhere. It may inflate the webpage's bundle size, but it allows for more future proofing and for the possibility of a singular code for also importing custom files during runtime on the user's end. That's it. But here is the iteration of solutions I've thought about, and why I moved on to the next step.

• My first instinct was to use fs or Web API File and Directory Entries API but the problem here is that this would only work locally.
• To iterate on the previous solution, we might want to read the data and generate pages during build time, but that would be more work. I have a gut feeling that there are already existing conveniences provided by modern web dev build systems. So, I should look into those.

• My instinct's telling me I should look into how json files are imported. That's the closest thing to what I have. So, I got this: use the Fetch API. I don't like this solution, because one of the core ideas I had when making my website was to provide users with all the information they have from the html file that was first sent. Any other dynamic data could be loaded later. The files I have are currently not that many, and I think I could serve the users these files statically. I could look into rendering this on the server side. If you're a non web dev after compiling the code, there are an additional four possible parts of the internet's plumbing that modifications can happen to your html file:

  1. Static generation: these are the files as-is ever since the server has existed.
  2. Server side: the server makes everything (at least what needs to be changed) for you on the spot.
  3. Incremental server side generation: the server makes changes to the existing file on the server only if changes are deemed necessary. It's often detected when some sort of event has happened (I clicked published on a blog but that depends too), or when the user tries to access a page and the server reacts. This is different from caching where the intention is to speed up serving the files from static generation. You can think of it as server side generation but you keep the file once it's generated to be used later.
  4. Client side: the website is being built right on your browser through the power of Javascript. This is how most of the internet has worked nowadays.

  I explained all this just to say that I prefer everything from static generation. I'm open to using other forms of web page generation, but I want to avoid client side generation as much as possible. Server side generation is also something I'm leaning on, but I would think that incremental server side generation would be the coolest thing to implement, but way out of my own scope at the moment.

  Anyways... time to move on...

• But that StackOverflow question actually has something I'm looking for! The question itself uses Glob import. We have the same question, but the asker is already on the right track of what I want. Glob import allows for static generation, but it does it better since it lets Vite, the web dev build tool I use to generate this website and its corresponding server, do all the hardwork of correctly splitting up the files such that it does not bloat the file size received by the end user. It might do more than just a simple static generation under the hood, employing all forms of website generation when needed. But, the problem is we can't use it directly. Nothing happens or it just breaks everything as-is.
• So, I wanted to investigate how to convert all these blobs into plain string. This led me to this answer: using BlobOptions. I wanna spoil it right here, that this is the actual answer, but I fumbled implementing it because their code uses the deprecated field As. So, I had to look for other answers.
• I wanted to look at as: "raw" in the given solution. So, I tried importing the asset as string by suffixing the path in import.meta.glob, but that does not yield anything. No errors were returned by nothing from my console log also gets printed with the content. So, I'm moving forward from this solution.

• This was a good answer I was very focused on but it led me to stray further from the light: Extracting text from a module on the SvelteKit backend. This is close to what I want but the type of asset they're loading is markdown. Luckily for the author, markdown is automatically detected by Vite so it gives it back into what it is. What I'm trying to import is a file that Vite cannot process as-is, so with the same code, it does not work.

• So, nothing in the first page of Google or the Vite Github page discussion has any unique answers. They all lead to the solutions above, but mostly using the Fetch API which I don't like due to it being purely client-side generation. What I'd often do at this step is to either make my own solution employing a mix of the types of web generation I want or maybe dig into the code myself. Surprisingly, digging into the code myself is much more helpful since I'm going into it with my own needs and not anyone else's opinion presented. Well, aside from the writer of the code itself, whose opinion I value much more. There I spotted this:

  <export interface ImportGlobOptions<
  	  Eager extends boolean,
  	  AsType extends string,
  	> {
  	  /**
  	   * Import type for the import url.
  	   *
  	   * @deprecated Use `query` instead, e.g. `as: 'url'` -> `query: '?url', import: 'default'`
  	   */
  	  as?: AsType
  	  /**
  	   * Custom queries
  	   */
  	  query?: string | Record<string, string | number | boolean>
  	}>

  The person who mentioned using GlobOptions was correct, but instead of using as?, we now use query?.

The final solution I have is this:

const fileList = import.meta.glob('$lib/data/*.htm', {query: '?raw'});

	for (const path in fileList) {
		fileList[path]().then((text) => {
			// text should print what's inside the file
			console.log(text);
		})
	}

That was an exhausting hour of looking up for a solution! But... writing and publishing this took waaaay longer. Maybe two or three hours. I write too slow. 😭

List of things I got to fix that I discovered while writing this includes:

1. Syntax highlighting inside Codeblocks
2. This blog post does not appear in the current list of blog posts in /Blogs
3. As a follow up from the previous discoveries, migrating from my own code generation to this new found static asset generation for listing the blogs during before server side or whatever
4. Proper metadata overriding since Fat Reinhard appears in all web preview thumbnail in Discord or Twitter
5. I have a bigger project which is to streamline the process of publishing blogs by using Google Docs as a source of truth but that's a bigger project at way out of my scope for the time being