Shell API

Basic operations

touch("foo")

symLink("a/b", "link")
hardLink("a/b", "link")

if (exists("foo")) { /* do something */ }

val text: String = read("README.md")
val lines: List<String> = readLines("README.md")
write("a/b/README.md", stringOrLinesOrBytes)

val entries: List<Path> = ls("/")
echo(entries)   // Colored ls-style output

The write functions are atomic (write to a temp file and then move into place, deleting on failure).

Recursive operations

By default operations are recursive and create parent directories.

rm("dir")        // Recursively delete dir.
mkdir("a/b/c")   // equivalent to mkdir -p
write("a/b/README.md", "...")   // Creates a/b first

// ... etc ... (see below for more)

cp and mv have more useful behavior than the UNIX equivalents.

cp("src", "dest")
mv("src", "dest")

cp

• Overlays. If source is a directory and dest is a pre-existing directory, the contents of source are recursively copied into dest, replacing whatever was there. That is, source/foo is copied to destination/foo not destination/source/foo as you may expect given the behaviour of the UNIX cp command. If you want that behaviour specify the destination as a subdirectory with the same name as source e.g. cp("src", "dest/src").
• File to directory. If source is a file and dest is a pre-existing directory, it is copied into that directory with the same name, overwriting any file with the same name that's already there.
• Creates directories. If source is a file and dest does not exist, destination is treated as a file and the directory it's in is created along with all parents.
• File overwrites. If source is a file and dest is a file in a pre-existing directory, it is copied to that given path exactly, replacing the file.
• If source is a directory and dest is a pre-existing file, IllegalStateException is thrown.

mv

mv moves from to to, overwriting or merging with to if it exists.

This command has subtly different semantics to the shell mv command or Files.move. It allows you to move a directory 'over' another directory without throwing a DirectoryNotEmptyException. If to is a directory and isn't empty, then instead of it being moved into that directory with the same name, each file is moved individually with directories being created as necessary in the destination, giving a form of merge semantics. If you want to move a directory to a subdirectory of somewhere else, you should therefore re-specify the source directory name as the last component of the target path.

However, if you try to move a file to a directory then it will be moved into that directory with the same name.

Warning

Currently extended attributes won't be copied when doing a merge, nor is the move atomic.

Non-destructive moves

Moves this file to the given path, but if a file with the same name already exists there the move will be to a file named name (1).extension, name (2).extension, etc.

val p1 = path("file.txt")
val p2 = path("README.md")
touch(p2)
p1.moveToNonDestructively(p2)   // actually moved to `README (2).md`

Changing directory and temp dirs

• Changing to a non-existent directory will create it.
• cd can take a code block which changes the working dir only for the duration of that block.
• mktmp returns a temp directory that will be recursively deleted when the script terminates.

cd(mktmp()) 
echo(dir)

// Creates subdir/another-subdir
cd("subdir/another-subdir") {
    echo(dir)
    write("file.txt", "Thoughtful statement.")
    echo(ls())
}

echo(dir)

Hashing files and directories

Secure (slow, large) and insecure (fast, small) hash functions are supported. You can hash a directory tree (see below for details) which is useful to determine if a tree has changed. Hashing a directory tree occurs in parallel: both files and sub-blocks of files will be hashed across multiple CPU cores, so the process is fast. When hashing a tree symlinks aren't followed, instead their target is included into the hash. Progress events are generated automatically.

echo(hash("directory", "fingerprint"))    // non-cryptographic but much faster
check(sha256("file") == hash("file", "SHA-2"))   // secure but slow

The "fingerprint" hash function (in the FINGERPRINT global constant) is a Merkleized variant of xxhash64 that enables parallel hashing. There is also "fingerprint-metadata" (FINGERPRINT_METADATA) that doesn't hash contents, only file name, mtime and other filesystem metadata.

What's actually hashed when hashing a directory tree is implementation defined, however in the current code it's actually a hash of the string containing a Unicode-art tree annotated with metadata and fingerprints.

A common use case for this is determining if files or directories have changed. The mark function hashes a file or directory tree and then stores the result in an extended attribute (alternative file stream on Windows). You can then use modifiedSinceMark to determine if there was any change since the mark was placed (and optionally, to update the mark at the same time). Using extended attributes ensures that the up-to-date check works even if the user moves the directory being observed.

Permissions

On Windows UNIX permissions are mapped to the closest equivalent and then stored in an extended attribute, ensuring that if you create an archive the permissions you set will be preserved on UNIX, even though they didn't make sense on Windows.

chmod can take both deltas to permissions as above, or rw-r--r-- style permission sets.

// Add write for everyone to files inside dir, recursively. 
chmod("dir", "a+w")
// Also make all directories enterable, recursively.
chmod("dir", "a+w", "a+wx")

// Make all files read/write for the owner, and readable by anyone.
chmod("dir", "rw-r--r--", "rwxr-xr-x")

echo(permissionsAsString(dir))   // rwxr-xr-x
val perms: Set<PosixFilePermission> = dir.permissions
val perms = permissions(dir)
echo(perms)   // GROUP_EXECUTE   GROUP_READ   OTHERS_EXECUTE   OTHERS_READ   OWNER_EXECUTE   OWNER_READ   OWNER_WRITE

Archives

tar("dir/", "dir.tar.gz")
zip("dir/", "dir.zip")
archive("dir/", "dir.tar.gz")   // an alias for tar
archive("dir/", "dir.zip")      // an alias for zip

unzip("thing.zip")
extract("thing.zip", "toDir")
extract("thing.tar.gz")   // extracts to thing/ in the current directory

The tar function also supports tar.bz2, tar.xz and tar.Z. Compression level can be controlled via zipOptions.compressionLevel. Single files can also be archived, and will be placed in the root.

Finding things

val matches: SortedSet<Path> = find("**.txt")    // find all text files below this directory
val matches = find("dir/*.txt")
val matches = find("regex:[0-9]+.txt")

val matches: List<String> = findAsStringPaths("**.txt")

Disk usage

val bytesUsed1 = du()
val bytesUsedByPNGs = du(dir, { it.name.endsWith("*.png") })

Accounting of hard links can be controlled.

Extended attributes

Also works on Windows.

xattr("file", "key", "value")
val value = xattr("file", "key")
val attributes = xattrs("file")

Open in a GUI app

open("whatever.html")
open("https://www.google.com")

Environment variables

println("Your editor is " + (ENV["EDITOR"] ?: "no editor set"))
ENV["COLORS"] = "off"   // affects the next external process execution

Directory trees

tree emits Unicode-art directory trees:

cd(mktmp())
touch("dir/file.txt")
touch("another-file.txt")
echo(tree("dir"))

... yields ...

.
├─── another-file.txt
╰─── dir
     ╰─── file.txt

You can annotate each item in the tree and trees can be colored:

echo(tree(".", color = true, annotator = annotateWithFingerprint()))
echo(tree(".", color = true, annotator = annotateWithPosix()))

Beyond debugging the primary use for this is verifying directory trees match what you expect, or quickly checking if a directory has changed. See hashing files and directories for more information.

You can combineAnnotators if you want more than one, or define your own Shell.TreeAnnotator object.

Path.subTree(relative: Boolean) returns a sorted list of all paths under that path, excluding that path.

Encoding things

echo(base64("a string to encode"))
echo(base64(dir / "some file to encode"))
echo(base64(byteArrayOf(1, 2, 3)))

Diffing things

echo(diffStrings("old string", "new string", contextSize = 10))

val unifiedDiff: String = diff("old.txt", "new.txt", contextSize = 10)
write("diff.patch", unifiedDiff)
patch("old.txt", "diff.patch")   // patch in place
patch("old.txt", "diff.patch", "new2.txt")  // create new file

Editing text files

edit locks and opens a text file for editing, providing a mutable StringBuilder within the lambda block. It's assumed the text file is in the system encoding, and that it is less than 2GB in size. If the file doesn't exist, it is created.

edit("file.txt") { stringBuilder ->
    stringBuilder.appendLine("another line")
}

Similar to edit and implemented using it, sed will update the given text file in place, replacing any matches of the given regex with the result of calling a lambda. This is a convenient utility for editing text files when the operation can be expressed as a transformation over a regular expression match.

sed("file.txt", "some (\\w+) words") { matchResult: MatchResult ->
    "replacement with the first group ${matchResult.groupValues[1]}"
}

Please note that despite sed originally being short for "stream editor", this function loads the whole file into memory rather than doing a streaming match. It also doesn't support the same feature set as the real sed.

The regular expression will be used in multi-line mode. If you don't want that then you should use the overload that lets you give a regex object.