Table of Contents
WHY
As you can see, my blog may contain a large number of mathematical formulas. I am currently loading Katex's JS and CSS in the Header to achieve rendering and style presentation. However, this means that every time the page is opened, a large JS file needs to be loaded, and the page renders slowly. Zola is written in Rust, so perhaps we can render the mathematical formulas into Katex tags together with the rendering of MD files into HTML. This way, we would only need to include the corresponding CSS styles, which should improve the rendering speed of the page.
Previous Trials
This PR#1073 and this Issue#1695 provide some attempts, but they did not reach a final, usable state.
katex-rs
But they provided some good ideas. We can use the katex-rs package for formula rendering, which is a Rust wrapper for katex.js, internally using quick-js, duktape, and wasm-js. According to previous discussions, quick-js is a JS engine on the UNIX platform, but it seems to also support compiling x86_64-pc-windows-gnu using MSYS2 on Windows. Duktape is based on C/C++, which can be compiled on both UNIX and Windows. wasm-js provides wasm support. According to previous discussions, katex-rs could not be compiled on all platforms, but it now appears to be feasible.
Regex
In previous attempts, there was an effort to use Regex to extract the part between two dollar signs ($) as an inline mode mathematical formula, and the part before two double dollar signs ($$) as a display mode mathematical formula. These were then passed to Katex for rendering into HTML, and sent back to the original string as text. This way, pulldown-cmark could render them into HTML tags and directly display them on the webpage.
The challenge lies in distinguishing between the real dollar sign with an escape character (\$) and the single dollar sign ($) used for formulas. This might mean that the Regex needs to support lookaround, or backtracking. However, the current Rust regex library sacrifices this feature in favour of efficiency and stability.
Additionally, I personally think using Regex could introduce many issues. For example, there might be code in CodeBlock that uses dollar signs, and dollar signs might also appear in non-Text content such as HTML blocks, link names, or image titles. Using Regex to bluntly extract all content between dollar signs could lead to serious compatibility problems.
My try
After carefully reading the code related to Zola's rendering, I found that pulldown-cmark turns the entire document into an Event iterator, using many tags to indicate the structure of the document. For instance, our main text is placed in the Text Event, and code is placed in the Code Event. There are also many pairs of Start Event and End Event to indicate scopes. I believe starting from this point should allow for more controlled operations than conducting a blunt Regex search on the entire text.
Merge Text
So, my approach is as follows: first, traverse the entire Event iterator once, merging all consecutive Text Events to ensure that the string we are dealing with is continuous, and two dollar signs are not separated.
fn merge_continuous_text_events(events: Vec<Event>) -> Vec<Event> {
let mut merged_events = Vec::new();
let mut current_text = String::new();
for event in events {
match event {
Event::Text(text) => {
// accumulate text
current_text.push_str(&text);
}
_ => {
// not text, merge
if !current_text.is_empty() {
merged_events.push(Event::Text(current_text.clone().into()));
// reset
current_text.clear();
}
// just add non-text event to vec
merged_events.push(event);
}
}
}
if !current_text.is_empty() {
merged_events.push(Event::Text(current_text.into()));
}
merged_events
}
Display Math
Next, we start by rendering the relatively simpler display mode mathematical formulas. In the Event iterator, we'll find that when using $$, we use a "enter" to create a new line. The "\n" character is marked by pulldown-cmark as SoftBreak, which means we can quickly locate Text tags that only contain $$ symbols. By removing SoftBreak between two tags, we can obtain the complete mathematical formula text.
fn handle_display_math(events: Vec<Event>) -> Vec<Event> {
let mut processed_events = Vec::new();
let mut in_display_math = false;
let mut math_content = String::new();
let k_opts = katex::Opts::builder().display_mode(true).build().unwrap();
for event in events {
match event {
// detect math start or stop
Event::Text(ref text) if text.trim() == "$$" => {
if in_display_math {
// thats the end of math
processed_events.push(Event::Html(
render_with_opts(&math_content, k_opts.clone()).unwrap().into(),
));
math_content.clear();
in_display_math = false;
} else {
// start math mode
in_display_math = true;
}
}
Event::SoftBreak if in_display_math => math_content.push(' '),
// collect text in math mode
Event::Text(text) if in_display_math => {
math_content.push_str(&text);
}
// not in math mode, copy
_ if !in_display_math => processed_events.push(event),
// drop every thing else inside math block
_ => {}
}
}
processed_events
}
Here, we merge the entire mathematical formula text and use the render_with_opts function from katex-rs to render the formula text into Katex HTML tags. Then, we add it back into the Event sequence as Event::Html, allowing pulldown-cmark to complete the next step of the rendering.
Inline Math
Next, consider the inline mode math. Since Zola has implemented rendering and highlighting for code blocks, we can process the mathematical formulas after Zola has handled the code. This approach ensures that code blocks and short code areas won't be processed by Katex, maintaining their integrity.
Now, we just need to process all Event::Text occurrences. As we further traverse the Event iterator, each Text event is processed individually. If it contains inline math formulas, we extract them and split the content into small vector of Text and HTML, which are then added back to the original Event sequence. This method allows us to seamlessly integrate mathematical formulas into the text while preserving the original structure and formatting.
fn handle_inline_math(events: Vec<Event>) -> Vec<Event> {
let mut processed_events = Vec::new();
let k_opts = katex::Opts::builder().display_mode(false).build().unwrap();
for event in events.into_iter() {
match event {
// handle text
Event::Text(text) => {
let text_events = render_math_inline(text.to_string(), &k_opts);
processed_events.extend(text_events);
}
_ => {
processed_events.push(event);
}
}
}
processed_events
}
Next, we need to implement the render_math_inline function. This function will iterate through the text character by character. When it encounters content wrapped in unescaped $ characters, it sends this content to Katex for rendering. For the text before and after the formula, it creates Event::Text events. Finally, it assembles these into a small vector and returns it. This approach ensures that inline mathematical expressions are correctly identified and rendered, while the surrounding text remains unaffected and properly formatted.
To avoid dealing with double-escaped characters, before parsing with pulldown-cmark, I first protect the "\$" by replacing it in advance. Here, I used an out-of-table Unicode character, which, in theory, should not be written into the document.
However, this approach has significant limitations. By globally replacing "\$" with an unknown character, it will cause replacements in code blocks and other areas as well, but I will not process code blocks, so the "\$" will never be displayed in them. This is quite problematic.
// handle dollor sign to avoid Escape
let content: String = content.replace("\\$", "\u{F8FF}");
// render inline math
fn render_math_inline(text: String, opts: &Opts) -> Vec<Event<'static>> {
let mut events = Vec::new();
let mut tmp_text = String::with_capacity(text.len());
let mut tmp_math = String::with_capacity(text.len());
let mut in_math = false;
for c in text.chars() {
match c {
'\u{F8FF}' => {
// this is a escaped dollor sign
if in_math {
tmp_math.push('$')
} else {
tmp_text.push('$')
};
}
'$' => {
if !in_math {
// not in math, means this is a start of math
if !tmp_text.is_empty() {
// first clear prev text
events.push(Event::Text(tmp_text.clone().into()));
tmp_text.clear();
}
in_math = true;
} else {
// already in math, means this is an end of math
if !tmp_math.is_empty() {
events.push(Event::Html(render_with_opts(&tmp_math, opts).unwrap().into()));
tmp_math.clear();
}
in_math = false;
}
}
_ => {
if in_math {
tmp_math.push(c)
} else {
tmp_text.push(c)
};
}
}
}
// push rest text
if !tmp_text.is_empty() {
events.push(Event::Text(tmp_text.into())); // push non-math
}
events
}
Then, we can add the previously written handle_display_math and handle_inline_math functions before the pulldown-cmark rendering process. This integration ensures that both display and inline mathematical formulas are handled and rendered correctly, enriching the text with properly formatted mathematical expressions before the markdown is converted into HTML, thus maintaining the flow and structure of the document.
// accumulate events and render katex
let events = merge_continuous_text_events(events);
let events = handle_display_math(events);
let mut events = handle_inline_math(events);
Limitation
Addressing the processing method as it stands, it's clear there are several challenges and potential issues that might arise. These can include:
- Edge Cases in Parsing: The method relies heavily on correctly identifying mathematical expressions within text, which can be complicated by edge cases not covered by simple parsing logic. For instance, when there is only one valid $ present, should it still render with Katex? Additionally, if there is an issue with how the mathematical formula is written, leading to a rendering failure, it could also cause the program to crash with an error.
- Performance Concerns: Iterating through the document and processing each piece of text for mathematical content could introduce performance bottlenecks, especially for large documents with complex structures. Perhaps parallel processing of multiple documents, as well as parallel processing of multiple text blocks, could be considered. However, how to ensure their order remains unchanged requires further consideration.
- Compatibility with Other Markdown Features: Ensuring that the processing of mathematical formulas does not interfere with other Markdown features, like nested formatting, links, or images, can be challenging. There's a risk that the introduced logic might incorrectly process or render non-math content.
Additionally, I found that when running Katex on Windows, Duktape is unable to properly render matrices and encounters errors that lead to crashes, which can can refer to this discussion Issue#12. However, there are no issues when using QuickJS. Until this problem is resolved, true cross-platform compatibility is not yet achievable.
Some possible solutions include:
- Wait for Duktape to update and fix the issue, which seems unlikely in the short term.
- Wait for Quick-JS to achieve full platform support, which also seems unlikely as the development community is not very active.
- Assist katex-rs in using a better backend, such as rust-v8, but this would significantly increase the size, as rust-v8 is much larger than Zola itself, making it a counterproductive solution. Alternatively, wait for Boa to mature, which also requires time. Mozilla's Servo also seems to be slow to update.
During compilation in the container, I also encountered issues with openssl-sys not being able to compile. This can be resolved by installing pkg-config and libssl-dev. We can refer to this discussion Issue#1021 for more information.
A Turning Point
Since I need to counteract the escape characters themselves and also counteract pulldown-cmark escaping of characters, this double escaping makes any changes very difficult. Therefore, starting from pulldown-cmark itself, making it support the recognition of two kinds of mathematical modes would be the best solution.
Fortunately, I found this PR#622. This PR implemented support for mathematics a year ago but has lacked merging. Although the reason is unclear, it seems to be very suitable for my current situation.
First, I need to change my dependency on pulldown-cmark from the official version on crates.io to a specific branch of a certain fork.
pulldown-cmark = { git = "https://github.com/notriddle/pulldown-cmark.git", branch = "math" , version = "0.10.0"}
pulldown-cmark-escape = { git = "https://github.com/notriddle/pulldown-cmark.git", branch = "math" }
Then, running cargo update will do the trick. By the way, I will delete all the previously mentioned cumbersome handling functions. We will be able to support Katex with just 6 lines of code.
let k_inline_opts = Opts::builder().display_mode(false).build().unwrap();
let k_display_opts = Opts::builder().display_mode(true).build().unwrap();
Event::InlineMath(text) => {
events.push(Event::Html(render_with_opts(&text, &k_inline_opts).unwrap().into()))
}
Event::DisplayMath(text) =>{
events.push(Event::Html(render_with_opts(&text, &k_display_opts).unwrap().into()))
}
By doing this, it seems we can simultaneously solve the issues of compatibility and performance. As long as we find a suitable cross-platform compatible JS backend, everything appears to be without problems.
Test
Text mode escape: $ and $$ and escaped \$ and \$
Inline math
Block Display Math: Code Block:
E = m c^2
Table :
| $ E = m c^2 $ |