Using sed to make indexes for books (1997)

Previously:

Using sed to make indexes for books (1997) - https://news.ycombinator.com/item?id=16080505 - Jan 2018 (17 comments)

Two thought:

1. I feel the use of AWK, SED, GREP and Bash (to name a few) is general waning. I rarely see it outside of HN posts. I wish I were better at using it. Sometimes for a special need I’ll ask a LLM to write me a script in bash just to keep this spirit alive.

2. On the other hand, I’ve written a book and added an index by using Microsoft Word, and it was basically pretty easy to add. I don’t know why I should want to go back to a world of intricate command line tools to do a job that should just be easy enough?

Previously:

Using sed to make indexes for books (1997) - https://news.ycombinator.com/item?id=16080505 - Jan 2018 (17 comments)

Two thought:

When this post was written (1997 updated in 2001), awk wasn't very portable because you still had the awk,nawk,gawk flavors. With sed if you stuck with BRE and avoided ERE you were going to be more portable.

But this email is a there because it is a bit of a feat of strength, thus why it was sent to the seders list. I wouldn't have used sed for all of this at the time, and I am wondering what they were using for publishing in anyway. In 1997/2001 I would have just taken text inputs and used LaTeX, and even indexes with troff was easier.

QuarkXPress and Pagemaker were very popular at that time, with QuarkXPress having almost all the commercial market in the 90s, so they must have been using something very specific for this religious market.

But also note that VS Code/ides with Vim keybindings is extremely popular among modern software developers, so while they may not call sed directly they still use it with `:`

That is the same reason we used it in the 80-90's, you had sed, vi, ed, etc... that were at least usable without much pain across all the drift of the UNIX wars.

But I am really curious on why this email even existed at all, it doesn't make sense outside of someone playing with UNIX/Linux as a side quest, unless they were *troff/TeX based?

> I feel the use of AWK, SED, GREP and Bash (to name a few) is general waning. I rarely see it outside of HN posts. I wish I were better at using

You can find plenty of examples with a quick github search[0]. The query in the url is

  path:bin dotfiles language:Shell

A shell script is my first option for any automation. sed is for quick transformations and awk is for more procedural ones.

[0] https://github.com/search?q=path%3Abin+dotfiles+language%3AS...

> I feel the use of AWK, SED, GREP and Bash (to name a few) is general waning. I rarely see it outside of HN posts. I wish I were better at using

You can find plenty of examples with a quick github search[0]. The query in the url is

  path:bin dotfiles language:Shell

A shell script is my first option for any automation. sed is for quick transformations and awk is for more procedural ones.

[0] https://github.com/search?q=path%3Abin+dotfiles+language%3AS...

But also note that VS Code/ides with Vim keybindings is extremely popular among modern software developers, so while they may not call sed directly they still use it with `:`

That is the same reason we used it in the 80-90's, you had sed, vi, ed, etc... that were at least usable without much pain across all the drift of the UNIX wars.

But I am really curious on why this email even existed at all, it doesn't make sense outside of someone playing with UNIX/Linux as a side quest, unless they were *troff/TeX based?

> awk wasn't very portable because you still had the awk,nawk,gawk flavors

Debian and Ubuntu both ship with mawk and Alpine has the busybox flavor of awk (nawk like). MacOS still ships with a default nawk. I don't know the scene has changed much, still many flavors out there.

Perl was in 1997 and is today a better awk+sed.

> awk wasn't very portable because you still had the awk,nawk,gawk flavors

The issues from the 80s/90s were mostly fixed by POSIX in the 2000's. IIRC all major Unix-like systems ship with awk implementations that are POSIX-compliant by default today.

With a few warts of course, but not like back then.

Perl was in 1997 and is today a better awk+sed.

Better is to use the typesetting language

Even on MacOS, save this as `doc.tr`

     .\" Index Macro
     .de IX
     .tm \\$1^\\$2^\\n%
     ..
     .\" Content
     .sp 2
     This is the first page.
     We are talking about the history of Unix.
     .IX "Operating Systems" "Unix"
     .bp
     This is the second page.
     Here we discuss the C language.
     .IX "Languages" "C"
     .bp
     This is the third page.
     We go back to talking about Unix.
     .IX "Operating Systems" "Unix"

The index will be output on stderr with:

     % groff doc.tr > output.ps 2> raw.idx

Which will output the following that you could use bash/sed/awk/perl/... to make into another troff file.

     % sort -t '^' -k2,2 -k1,1 raw.idx                    
     Languages^C^2
     Operating Systems^Unix^1
     Operating Systems^Unix^3

(Note: `^` as a delimiter is an old Unix thing, just going really retro)

Even inserting the `.IX ...` lines on a string match in the troff file is fairly easy. But somehow they were parsing the typeset output, not sure what kind. But no need to try and track page numbers when typesetting languages do that for you.

TL;DR What I was saying is if you are using a typesetting language, use the tools in the typesetting language when possible. If you can do it in troff you can do it in LaTeX with makeindex etc...