Unlock Non-Unicode Glyphs In Lua(La)TeX With Harfbuzz

Aug 10, 2025 by Rajiv Sharma 54 views

Accessing Non-Unicode Glyphs by Name with Lua(La)TeX and Harfbuzz

Hey guys! Ever stumbled upon a font with amazing glyphs that just don't seem to play nice with Unicode? You're not alone! This article is your ultimate guide to unlocking those hidden typographic treasures using Lua(La)TeX and Harfbuzz. We're diving deep into how you can access non-Unicode glyphs by their names, making your documents look absolutely stunning. This is a continuation of a previous discussion on accessing non-Unicode glyphs from OTFs with Lua(La)TeX, which solved the problem with the Node renderer, but not for Harfbuzz. Let's get started!

Understanding the Challenge: Unicode vs. Non-Unicode Glyphs

Before we jump into the technical stuff, let's quickly break down what we're dealing with. Unicode is like the universal language for characters – it assigns a unique number to almost every character you can imagine. Most fonts are designed with Unicode in mind, making it super easy to type and display text. However, some fonts, especially older ones or those with decorative glyphs, contain characters that aren't part of the Unicode standard. These are the non-Unicode glyphs, and accessing them can be a bit tricky.

When you're working with Lua(La)TeX, you have two main ways to render fonts: the Node renderer and Harfbuzz. The Node renderer is great, but Harfbuzz is often preferred for its advanced typographic features like ligatures and kerning. The challenge arises because Harfbuzz relies heavily on Unicode. So, how do we tell Harfbuzz to use those non-Unicode glyphs? That's the puzzle we're going to solve.

To truly grasp the essence of this challenge, it's essential to delve deeper into the mechanics of how fonts and glyphs are handled within the TeX ecosystem. Think of a font as a vast library of shapes, each representing a character or symbol. These shapes, known as glyphs, are stored in a font file, typically in formats like OpenType (OTF) or TrueType (TTF). Within these files, each glyph is identified by a unique index or name. Unicode glyphs are those whose indices correspond to Unicode code points, allowing for seamless integration with standard text encoding. However, fonts often contain additional glyphs that fall outside the Unicode range, such as swashes, ornaments, or historical forms. These non-Unicode glyphs, while visually appealing, pose a challenge for rendering engines that primarily rely on Unicode for character mapping. Harfbuzz, a powerful shaping engine, excels at handling complex scripts and typographic features, but its reliance on Unicode presents a hurdle when dealing with these non-standard glyphs. The core of the problem lies in bridging the gap between glyph names and the rendering engine's expectations. We need a way to instruct Harfbuzz to access these glyphs not by their Unicode counterparts, but directly by their names or indices within the font file. This requires a deeper understanding of Lua(La)TeX's font management capabilities and the intricacies of Harfbuzz's shaping process. By mastering these concepts, we can unlock the full potential of our fonts and create documents that are not only typographically accurate but also visually stunning.

The Lua(La)TeX and Harfbuzz Solution: A Step-by-Step Guide

Okay, let's get our hands dirty with some code! Here's the general idea:

Load the font: We'll use Lua(La)TeX's font loading mechanism to bring our font into the document.
Access the glyph: We'll need to find a way to reference the glyph by its name. LuaTeX gives us some powerful tools to do this.
Tell Harfbuzz: We'll use Lua code to insert the glyph into the text stream in a way that Harfbuzz understands.

Let's break this down with a practical example. Imagine we have a font called "MyFont.otf" with a cool swash glyph named "swash.1". Here's how we might access it:

\documentclass{article}
\usepackage{fontspec}
\usepackage{luacode}

\begin{document}

\fontspec{MyFont.otf}

\begin{luacode}
  local fontdata = font.getfont "MyFont.otf"
  local glyph_id = fontdata.characters["swash.1"]
  if glyph_id then
    tex.sprint(string.char(glyph_id))
  else
    tex.sprint("Glyph not found!")
  end
\end{luacode}

\end{document}

Let's walk through this code:

\usepackage{fontspec}: This package is essential for font management in Lua(La)TeX.
\usepackage{luacode}: This lets us embed Lua code directly into our LaTeX document. Super handy!
\fontspec{MyFont.otf}: This loads our font.
font.getfont "MyFont.otf": This Lua function retrieves font data.
fontdata.characters["swash.1"]: This is the magic! We're accessing the glyph ID by its name.
tex.sprint(string.char(glyph_id)): This inserts the glyph into the document.

This example showcases the fundamental steps involved in accessing non-Unicode glyphs with Lua(La)TeX and Harfbuzz. However, to truly master this technique, it's crucial to delve deeper into the nuances of font handling and glyph manipulation within the LuaTeX environment. Let's dissect the Lua code snippet further. The font.getfont function is a cornerstone of LuaTeX's font management system, providing access to a wealth of information about a loaded font. This information includes not only the glyph outlines themselves but also metadata such as kerning pairs, ligature definitions, and, most importantly for our purposes, the glyph name to ID mapping. The fontdata.characters table is a Lua table that acts as a dictionary, mapping glyph names to their corresponding glyph IDs. This is the key to accessing non-Unicode glyphs, as it allows us to retrieve the numerical identifier of a glyph based on its name, rather than relying on Unicode code points. The tex.sprint function is the bridge between the Lua world and the TeX typesetting engine. It allows us to inject TeX code or, in this case, a glyph ID into the TeX input stream. However, simply printing the glyph ID as a character might not always work, especially with Harfbuzz. This is where string.char comes in. It attempts to convert the glyph ID into a character, which Harfbuzz can then process. However, this approach has limitations, as it relies on the assumption that the glyph ID can be represented as a valid character code. In many cases, especially with non-Unicode glyphs, this assumption doesn't hold true. This is where more advanced techniques, such as using the oexpand primitive or directly manipulating the Harfbuzz shaping process, become necessary. We'll explore these techniques in more detail later in the article. For now, the key takeaway is that this simple Lua code provides a foundation for accessing non-Unicode glyphs, but further refinements are often required to ensure correct rendering with Harfbuzz.

Advanced Techniques: Fine-Tuning Harfbuzz

The previous example is a great starting point, but sometimes Harfbuzz needs a little more guidance. Here are some advanced techniques to consider:

Using oexpand: This TeX primitive can prevent expansion, ensuring that the glyph ID is passed directly to Harfbuzz.
Direct Harfbuzz manipulation: LuaTeX allows you to hook into the Harfbuzz shaping process, giving you ultimate control.
Glyph substitution: You can define rules to substitute specific character sequences with your non-Unicode glyphs.

Let's look at an example using \noexpand:

\documentclass{article}
\usepackage{fontspec}
\usepackage{luacode}

\begin{document}

\fontspec{MyFont.otf}

\begin{luacode}
  local fontdata = font.getfont "MyFont.otf"
  local glyph_id = fontdata.characters["swash.1"]
  if glyph_id then
    tex.sprint("{\\noexpand\\char" .. glyph_id .. "}")
  else
    tex.sprint("Glyph not found!")
  end
\end{luacode}

\end{document}

Here, we're using \noexpand\char to tell TeX to treat the glyph ID as a character code without expanding it. This can often help Harfbuzz render the glyph correctly.

Now, let's delve into the realm of advanced techniques for fine-tuning Harfbuzz. While the previous example provides a solid foundation, there are situations where more direct control over the shaping process is required. One such technique involves using the \noexpand primitive in conjunction with \char. As demonstrated in the code snippet, this approach prevents TeX from expanding the glyph ID, ensuring that it is passed directly to Harfbuzz for rendering. However, it's important to understand why this works and when it's necessary. Harfbuzz operates on a stream of glyph IDs, and it relies on these IDs to identify the glyphs to be rendered. When we simply print a glyph ID using tex.sprint, TeX might interpret it as a character code and attempt to perform character-level operations, such as ligature substitution or kerning adjustments. This can lead to unexpected results, especially when dealing with non-Unicode glyphs that don't have a direct mapping to Unicode code points. By using \noexpand\char, we bypass TeX's character-level processing and ensure that the glyph ID is treated as a raw glyph identifier, which Harfbuzz can then use to access the glyph directly from the font file. However, even with \noexpand\char, there might be cases where Harfbuzz still doesn't render the glyph correctly. This is where more advanced techniques, such as direct Harfbuzz manipulation and glyph substitution, come into play. Direct Harfbuzz manipulation involves using LuaTeX's hooks into the Harfbuzz shaping process to modify the glyph stream directly. This allows for fine-grained control over glyph positioning, ligature formation, and other typographic features. Glyph substitution, on the other hand, involves defining rules that replace specific character sequences with non-Unicode glyphs. This can be useful for creating custom ligatures or swashes that are not part of the font's default glyph set. Both of these techniques require a deeper understanding of Harfbuzz's shaping process and LuaTeX's font management capabilities. They also involve more complex Lua code and a greater level of experimentation. However, the rewards are well worth the effort, as they allow you to unlock the full potential of your fonts and create documents that are truly unique and visually stunning.

Real-World Examples: Showcasing the Power

Let's look at some real-world scenarios where accessing non-Unicode glyphs can make a huge difference:

Historical Typography: Many historical fonts contain ligatures and swashes that aren't in Unicode. Accessing these can add authenticity to your documents.
Calligraphy and Decorative Fonts: Decorative fonts often have unique glyphs that can add flair to headings and titles.
Custom Symbols and Icons: You can use non-Unicode glyphs to create custom symbols and icons for your documents.

Imagine you're typesetting a historical document using a font that includes long s (ſ) and other archaic characters. Without the ability to access non-Unicode glyphs, you'd be stuck with modern characters, losing the authentic feel. Or, picture designing a logo using a calligraphy font with beautiful swashes. Accessing these swashes allows you to create a logo that truly stands out.

To further illustrate the power of accessing non-Unicode glyphs, let's delve into specific real-world examples where this technique can significantly enhance the visual appeal and historical accuracy of documents. Consider the realm of historical typography. Many fonts designed to replicate historical typefaces contain a wealth of ligatures, swashes, and alternate characters that are not encoded in Unicode. These glyphs, often representing historical letterforms or abbreviations, are crucial for capturing the authentic look and feel of historical texts. For instance, the long s (ſ), a character commonly used in early modern printing, is absent from the standard Unicode repertoire. Similarly, ligatures such as ct, st, and ff, which were frequently employed in historical typography to improve readability and aesthetic appeal, are often represented as separate characters in Unicode. By accessing these non-Unicode glyphs, we can accurately reproduce the typographic conventions of the past, creating documents that are not only legible but also visually faithful to their historical context. Beyond historical typography, the ability to access non-Unicode glyphs is invaluable in the design of logos, posters, and other visual materials. Calligraphy fonts, in particular, often feature elaborate swashes, flourishes, and alternate glyph forms that can add a touch of elegance and sophistication to a design. These decorative glyphs, typically not encoded in Unicode, provide designers with a palette of visual elements that can be used to create unique and memorable compositions. For example, a logo for a luxury brand might incorporate a swash from a calligraphy font to convey a sense of exclusivity and craftsmanship. Similarly, a poster for a theatrical production might use alternate glyph forms to evoke a specific historical period or artistic style. Furthermore, non-Unicode glyphs can be used to create custom symbols and icons for various applications. Imagine designing a user interface that requires a set of unique icons. Instead of relying on standard icon fonts or bitmap images, we can leverage the flexibility of non-Unicode glyphs to create custom symbols that perfectly match the aesthetic of the interface. This approach not only ensures visual consistency but also allows for greater control over the size and rendering of the icons. In essence, accessing non-Unicode glyphs empowers us to transcend the limitations of standard character encoding and unlock the full potential of our fonts. Whether it's for historical accuracy, artistic expression, or practical design, this technique is an indispensable tool for anyone who cares about typography and visual communication.

Troubleshooting Common Issues

Sometimes, things don't go as planned. Here are some common issues you might encounter and how to fix them:

Glyph not found: Double-check the glyph name. Font names are case-sensitive!
Glyph renders incorrectly: Try using \noexpand or explore direct Harfbuzz manipulation.
Font loading issues: Make sure the font file is in a location that Lua(La)TeX can access.

If you're getting a "Glyph not found" error, the first thing to do is to carefully verify the glyph name. Font names are case-sensitive, so "Swash.1" is different from "swash.1". It's also worth checking the font's documentation or using a font editor to confirm the exact name of the glyph you're trying to access. If the glyph is found but renders incorrectly, the issue might be related to how Harfbuzz is interpreting the glyph ID. As we discussed earlier, using \noexpand can often resolve this problem by ensuring that the glyph ID is passed directly to Harfbuzz without any intervening processing by TeX. However, if \noexpand doesn't work, you might need to delve into direct Harfbuzz manipulation, which involves modifying the glyph stream at a lower level. This is a more advanced technique that requires a deeper understanding of Harfbuzz's shaping process, but it can provide the ultimate level of control over glyph rendering. Font loading issues can also be a source of frustration. If Lua(La)TeX can't find the font file, it will typically throw an error message. Make sure the font file is in a location that Lua(La)TeX can access. This could be a system font directory or a directory within your TeX distribution. You can also specify the font's path explicitly using the fontspec package's Path option. For example, \fontspec[Path=/path/to/fonts/]{MyFont.otf} would load the font from the specified directory. In addition to these common issues, there are other potential pitfalls to be aware of. For instance, some fonts might have glyphs with the same name but different forms, depending on the context. In such cases, you might need to use more sophisticated techniques, such as feature files or OpenType layout features, to control which glyph form is selected. Furthermore, the interaction between different font features, such as ligatures and kerning, can sometimes lead to unexpected results when dealing with non-Unicode glyphs. Experimentation and careful testing are often necessary to achieve the desired typographic outcome. Remember, accessing non-Unicode glyphs is a powerful technique, but it also requires a certain level of technical expertise. Don't be afraid to consult the LuaTeX documentation, online forums, and other resources to troubleshoot issues and learn more about the intricacies of font handling in Lua(La)TeX.

Conclusion: Unleash Your Typographic Creativity

Accessing non-Unicode glyphs with Lua(La)TeX and Harfbuzz opens up a world of typographic possibilities. By mastering these techniques, you can create documents that are not only visually appealing but also historically accurate and uniquely expressive. So, go ahead, explore your fonts, and unleash your typographic creativity!

We've covered a lot in this article, from the basics of Unicode and non-Unicode glyphs to advanced techniques for fine-tuning Harfbuzz. The key takeaway is that Lua(La)TeX provides a powerful and flexible platform for working with fonts, even those that contain non-standard glyphs. By leveraging Lua's scripting capabilities and TeX's typesetting engine, we can access these glyphs and incorporate them into our documents, adding a touch of personality and authenticity. Whether you're a historian recreating historical documents, a designer crafting a logo, or simply a typography enthusiast looking to push the boundaries of visual communication, the ability to access non-Unicode glyphs is a valuable asset. It allows you to go beyond the limitations of standard character encoding and express your creativity in new and exciting ways. So, don't be afraid to experiment, explore your font libraries, and discover the hidden gems that lie within. With a little bit of Lua code and a dash of typographic ingenuity, you can transform your documents from ordinary to extraordinary. Remember, the world of typography is vast and ever-evolving. There's always something new to learn, a new technique to master, or a new font to explore. By embracing the challenges and opportunities that arise when working with non-Unicode glyphs, you can deepen your understanding of typography and become a more skilled and versatile typographer. So, go forth and create, and let your typographic creativity shine!