A tale of two path separators

Old Macs (which I grew up with) had even more baroque path handling than mentioned in the blog post:

Double colon (::) meant the same as .. on Unix/DOS, that is "go up one level". So you have to be careful when concatenating paths to not get double separators.

Paths starting with : were relative. If a path didn't start with the separator, the first component was the volume name (disk partition). Again, quite unlike Unix.

Also, remember it was common to have spaces in names on Mac, even the default harddrive on Macs was named "Macintosh HD". So an absolute path like "Macintosh HD:Programs:MacWrite" would have been common. (I grew up with Macs in Swedish, so I'm back translating the names here, could be that the names were slightly different in English.)

It took me a long time to understand why colon wasn't a valid character for file names on Mac and I still find the colon separator to be the least visible these days. Finder can display paths with the forward slash separator (defaults write com.apple.finder _FXShowPosixPathInTitle -bool YES), and yet forward slash may be used in a file name created through Finder as noted in the post, while colon cannot (which is not addressed), but creating a file in the terminal named with a colon is possible and the shell will escape it correctly in use. This file then shows up with a slash in place of the colon when viewed in Finder, and conversely the file with a slash in the name shows up in Terminal with a colon!

> Windows is weird for using the backwards slash

Windows handles slash as well, also part of a unification with UNIX style paths intended for XEDOS.

Another Windows oddity: each drive letter has its own current directory. D: doesn't mean the root of D:, it means "wherever you last were on D:". Same with C:foo, which is relative to C:'s current directory. DOS baggage that's still around.

I'm curious how much of this behaviour is still intentional design vs. just inertia. Are the modern filesystems still constrained by these older choices, or is it mostly for compatibility?

I was expecting the story of the magical ¥ path separator

Now imagine operating system, that has no directories (and no path separators) or no filesystem at all.

> Windows is weird for using the backwards slash

Windows handles slash as well, also part of a unification with UNIX style paths intended for XEDOS.

Old Macs (which I grew up with) had even more baroque path handling than mentioned in the blog post:

Double colon (::) meant the same as .. on Unix/DOS, that is "go up one level". So you have to be careful when concatenating paths to not get double separators.

Paths starting with : were relative. If a path didn't start with the separator, the first component was the volume name (disk partition). Again, quite unlike Unix.

Fun things is that I encountered that for the first time when using Clozure CL (https://ccl.clozure.com/) which quotes colons when converting paths to string even on Linux:

  $ cat <<'EOF' >x.lisp
  heredoc> (require :uiop)
  heredoc> (let ((p (make-pathname :name "foo:bar")))
  heredoc>   (format t "~@{~A~%~}" (namestring p) (uiop:native-namestring p)))
  heredoc> EOF
  $ ccl -b -Q -l x.lisp </dev/null
  foo\:bar
  foo:bar
  $ sbcl --script x.lisp
  foo:bar
  foo:bar

Current macOS Finder let's you name files with a slash in them, rendered as a : in the terminal.

You still have to worry about colons on Linux; while they're valid in file and folder names, they prevent folders from being put on PATH and understood properly.

(Other characters of course cause usability problems and are potentially even a security vulnerability depending on the terminal. But they're still "valid".)

I'm curious how much of this behaviour is still intentional design vs. just inertia. Are the modern filesystems still constrained by these older choices, or is it mostly for compatibility?

Now imagine operating system, that has no directories (and no path separators) or no filesystem at all.

I was expecting the story of the magical ¥ path separator

Filesystems usually do not see path separators at all, it's something handled at VFS level

And you need `cd /d` to switch drives. This was how I rendered a Windows computer non-bootable for the first time. Ran Command Prompt as admin (because I was logged in as a user that didn’t have write access to D:\backups), and it starts in a rather important directory, then:

  C:\WINDOWS\system32>cd D:\backups\some-huge-directory
  C:\WINDOWS\system32>del /s *

Oops. I learned to look twice before running a big dangerous command. And to use /d.

Like CP/M, DOS v1, RT-11 etc.

VMS used:

    node::device:[dir1.dir2.dir3]filename.extension;version

from memory, you could have up to 15 nested directories.

The versioning was cool as long as you remembered to clean them up.

Would you mind sharing a related link? I'm finding it to be impossible to search "¥ path separator", the ¥ is stripped out by like every search engine even when quoted

Fun things is that I encountered that for the first time when using Clozure CL (https://ccl.clozure.com/) which quotes colons when converting paths to string even on Linux:

  $ cat <<'EOF' >x.lisp
  heredoc> (require :uiop)
  heredoc> (let ((p (make-pathname :name "foo:bar")))
  heredoc>   (format t "~@{~A~%~}" (namestring p) (uiop:native-namestring p)))
  heredoc> EOF
  $ ccl -b -Q -l x.lisp </dev/null
  foo\:bar
  foo:bar
  $ sbcl --script x.lisp
  foo:bar
  foo:bar

You still have to worry about colons on Linux; while they're valid in file and folder names, they prevent folders from being put on PATH and understood properly.

(Other characters of course cause usability problems and are potentially even a security vulnerability depending on the terminal. But they're still "valid".)

Current macOS Finder let's you name files with a slash in them, rendered as a : in the terminal.

Isn't that just what the original article that we all commented on described? I don't understand what you are trying to add to the conversation here.

I think that the slash might be the rendering though? If you

    $ touch "foo:bar"

In the Terminal, then Finder renders it as foo/bar.

So who is lying, how is it stored in the directory entry in APFS itself?

Filesystems usually do not see path separators at all, it's something handled at VFS level

  C:\WINDOWS\system32>cd D:\backups\some-huge-directory
  C:\WINDOWS\system32>del /s *

Oops. I learned to look twice before running a big dangerous command. And to use /d.

But they do see / or : in file names, and the interesting question is which one it is today on which filesystem.

Like CP/M, DOS v1, RT-11 etc.

VMS used:

    node::device:[dir1.dir2.dir3]filename.extension;version

from memory, you could have up to 15 nested directories.

The versioning was cool as long as you remembered to clean them up.

Or even just Amazon S3.

Would you mind sharing a related link? I'm finding it to be impossible to search "¥ path separator", the ¥ is stripped out by like every search engine even when quoted

The short version is, the DOS-era Japanese character sets had replaced \ with ¥, and modern Windows still shows paths as C:¥WINDOWS if you're using the Japanese locale.

Search for "yen sign path separator".

Isn't that just what the original article that we all commented on described? I don't understand what you are trying to add to the conversation here.

I think that the slash might be the rendering though? If you

    $ touch "foo:bar"

In the Terminal, then Finder renders it as foo/bar.

So who is lying, how is it stored in the directory entry in APFS itself?

But they do see / or : in file names, and the interesting question is which one it is today on which filesystem.

The filesystems of macOS are particularly opinionated, much more than most Unices which tend toward "anything is allowed [and usually preserved] except \0 and /".

macOS supports case-insensitivity[0] and performs unicode normalization[1] on filenames, and decomposes name data to an extent that the question "what does the fs see" is a bit moot.

With that said, the internal storage of filenames in APFS are a nul-terminated UTF-8 string[2], with (i'm pretty sure) colons as colons, which the Finder displays as slashes.

[0] if you make a file named "Makefile" then touch a file named "makefile", it'll touch the first file, instead of making a second file.

[1] if you make a file named "schön" (s-c-h-combining¨-o-n) and then search for (s-c-h-ö-n), you can find it, or vice versa. The particular normalization/canonicalization used is NFD.

[2] j_drec_key_t description in https://developer.apple.com/support/downloads/Apple-File-Sys...

Or even just Amazon S3.

The short version is, the DOS-era Japanese character sets had replaced \ with ¥, and modern Windows still shows paths as C:¥WINDOWS if you're using the Japanese locale.

Search for "yen sign path separator".

The filesystems of macOS are particularly opinionated, much more than most Unices which tend toward "anything is allowed [and usually preserved] except \0 and /".

macOS supports case-insensitivity[0] and performs unicode normalization[1] on filenames, and decomposes name data to an extent that the question "what does the fs see" is a bit moot.

With that said, the internal storage of filenames in APFS are a nul-terminated UTF-8 string[2], with (i'm pretty sure) colons as colons, which the Finder displays as slashes.

[0] if you make a file named "Makefile" then touch a file named "makefile", it'll touch the first file, instead of making a second file.

[1] if you make a file named "schön" (s-c-h-combining¨-o-n) and then search for (s-c-h-ö-n), you can find it, or vice versa. The particular normalization/canonicalization used is NFD.

[2] j_drec_key_t description in https://developer.apple.com/support/downloads/Apple-File-Sys...

> a file named "schön" (s-c-h-combining¨-o-n)

Combining marks come after the character they modify, btw. (Presumably thanks to support from things like harfbuzz, modern systems will happily put two dots above an h.)

> a file named "schön" (s-c-h-combining¨-o-n)

Combining marks come after the character they modify, btw. (Presumably thanks to support from things like harfbuzz, modern systems will happily put two dots above an h.)

I was reminded yesterday that macOS will happily let you create files that appear to have slashes in their name, as shown here by the Finder:

A Finder window with two entries. The first is a folder named 'a/b/c' (a slash b slash c), the second is a text file named 'x/y/z' (x slash y slash z).

I made the folder with the “New Folder” button in Finder, and the text file by saving it in TextEdit. Even though I understand how this works, it breaks my brain a bit every time.

If you didn’t know how this works, you could get a clue by running ls:

$ ls
a:b:c/     x:y:z.txt

The files have either colons or slashes in their name, depending on how you look at them.

I don’t understand all the nuances, but I know this is a side-effect of the fact that macOS has not one but two path separators: the slash (/) and the colon (:). The two separators are used in different contexts, and the system will translate between them as needed.

These two separators reflect the two parent systems of modern macOS: classic Mac OS and the Unix-like NeXTSTEP. When they were joined together, Apple’s engineers had to build a file system that was compatible with both the classic Mac’s file system (the Mac OS Extended File System, aka HFS+), and with NeXTSTEP’s file system (the Unix file system, aka UFS). Among other differences, these systems had different path separators: HFS+ used a colon, while UFS used a slash.

There’s a Usenix 2000 paper written by several Apple engineers which describes the problems of combining classic Mac OS and Unix in more detail. It actually describes exactly what we’re seeing with the Finder and ls:

[The translation layer] can create a user-visible schizophrenia in the rare cases of file names containing colon characters, which appear to Carbon applications as slash characters, but to BSD programs and Cocoa applications as colons.

AppleScript is where you’re most likely to encounter colons as path separators in modern macOS:

$ which osascript
/usr/bin/osascript

$ osascript -e 'tell application "Finder" to get (path to me)'
alias Phoenix SSD:usr:bin:osascript

This is likely because AppleScript goes back to System 7, which only had to care about HFS+ and colons. It can give you a UNIX- (or POSIX-) style path, but you have to ask for it explicitly:

$ osascript -e 'tell application "Finder" to get POSIX path of (path to me)'
/usr/bin/osascript

In 2017, Apple replaced HFS+ with the Apple File System, aka APFS, but these dual path separators remain. I can’t find any documentation to confirm it, but I assume that’s for backward compatibility reasons – picking a single path separator would break an enormous amount of software.

The “slashes in filenames” breaks my brain because I started programming after Unix-style filesystems had become totally dominant. In my mind, slash is the directory separator, and Windows is weird for using the backwards slash (\). In reality, file systems have used a variety of directory separators, and it’s only the current Unix hegemony that makes this seem weird. On another timeline, forward slashes in a filename would seem totally normal.

Hacker Times

Hacker Times

A tale of two path separators

Discussion

Discussion