Avoid Unicode Mangling in Jamf with shef

Computers are “the equivalent of a bicycle for our minds” as Steve Job once said. Sure enough, like a bicycle, they also require maintenance! Sometimes that requires soliciting the user to take action. Asking at the right time is very important (see my post Don’t Be a Jerk) and so is brevity. I’ve found a sprinkling of emoji and other symbols can help your message cut through the noise. A bright red stop sign ๐Ÿ›‘ can convey its meaning with only a glance, especially for non-native speakers of your language. Your tooling however, may trip you up and mangle your text.

While AppleScript, Swift Dialog and JamfHelper can all handle Unicode, Jamf databases tables by default do not support 4-byte Unicode! I touched on this in my post: jpt 1.0 text encoding fun. While the database engine might support it, chances are your tables are Latin1 which top out at 3-byte UTF-8 encoded characters. 4-byte characters get mangled. Let’s take a look at this in action:

Looks good while editing…
Once saved, the mangling is clear

Only the gear makes it through because it’s actually two 3-byte characters (U+2699 โš™ gear plus variation selector U+FE0F) but who has time to check which one is which? How about an encoding tool written in shell, that allows you to escape and format Unicode for shell scripts and/or Jamf script parameters that can make it through unscathed and un-mangled? Sound good? Great! I present shef, the Shell Encoder and Formatter!

Give it some text, specify an encoding scheme and/or quoting style and out comes your string ready for use! You can put the resulting string in your script as a variable or as a script parameter in a Jamf policy depending on quoting options. I’ve done the hard work of finding the various ways to escape special characters for shell and made this handy script for you!

shef Examples

First, let’s make a simple script for Jamf that processes the output of shef. I’ve chosen AppleScript so you can play along at home even if you don’t have Jamf installed, this can also be applied to “wrapper scripts” that leverage other tools like Swift Dialog or JamfHelper. At it’s heart is the simple technique of encoding the input with shef then decoding it in the script before presentation. If you use bash use echo -e if you use sh or zsh then just echo will work, it’s that simple. Our example script simpleAlert-AS.sh, keeps things small and as minimal as possible but keep in mind my fuller-featured tool shui for more robust scripts where you may need text or password entry, file picking, etc. without external dependencies but don’t want to bother learning AppleScript.

#!/bin/bash
#simpleAlert-AS - Copyright (c) 2023 Joel Bruner (https://github.com/brunerd)
#Licensed under the MIT License

#Simple Applescript alert dialog for Jamf - just a title, a message and an OK button
#Accepts hex (\xnn) and octal (\0nnn) escaped UTF-8 encoded characters (since the default Jamf db character set mangles 4 byte Unicode)
#Use shef to encode your strings/files for use in this script: https://github.com/brunerd/shef

#function to interpret the escapes and fixup characters that can screw up Applescript if unescaped \ and "
function interpretEscapesFixBackslashesAndQuotes()(echo -e "${@}" | sed -e 's/\\/\\\\/g' -e 's/"/\\"/g')

function jamflog(){
	local logFile="/var/log/jamf.log"
	#if we cannot write to the log or it does not exist, unset and tee simply echoes
	[ ! -w "${logFile}" ] && unset logFile
	#this will tee to jamf.log in the jamf log format: <Day> <Month> DD HH:MM:SS <Computer Name> ProcessName[PID]: <Message>
	echo "$(date +'%a %b %d %H:%M:%S') ${myComputerName:="$(scutil --get ComputerName)"} ${myName:="$(basename "${0}" | sed 's/\..*$//')"}[${myPID:=$$}]: ${1}" | tee -a "${logFile}" 2>/dev/null
}


#process our input then escape for AppleScript
message=$(interpretEscapesFixBackslashesAndQuotes "${4}")
title=$(interpretEscapesFixBackslashesAndQuotes "${5}")
#could be a path or a built-in icon (stop, caution, note)
icon="${6}"
#invoke the system open command with this argument (URL, preference pane, etc...)
open_item="${7}"

#these are the plain icons (Applescript otherwise badges them with the calling app)
case "${icon}" in
	"stop") icon="/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/AlertStopIcon.icns";;
	"caution") icon="/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/AlertCautionIcon.icns"
		#previous icon went away in later macOS RIP
		[ ! -f "${icon}" ] && icon="/System/Library/CoreServices/Problem Reporter.app/Contents/Resources/ProblemReporter.icns";;
	"note") icon="/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/AlertNoteIcon.icns";;
esac

#make string only if path is valid (otherwise dialog fails)
if [ -f "${icon}" ]; then
	withIcon_AS="with icon file (POSIX file \"${icon}\")"
fi

jamflog "Prompting user: $(stat -f %Su /dev/console)"

#prompt the user, giving up and moving on after 1 day (86400 seconds)
/usr/bin/osascript <<-EOF
activate
with timeout of 86400 seconds
	display dialog "${message}" with title "${title}" ${withIcon_AS} buttons {"OK"} default button "OK" giving up after "86400"
end timeout
EOF

if [ -n "${open_item}" ]; then
	jamflog "Opening: ${open_item}"
	open "${open_item}"
fi

exit 0

Upload the above script to your Jamf (if you have one), label parameters 4, 5, 6, and 7 as: Message, Title, Icon Path, an Open After OK, respectively. If running local keep this in mind and put 1, 2, 3 as place holder arguments for the first three parameters.

Jamf Script parameters names for

Let’s come up with an example message for our users. How about the common refrain coming from MacAdmins around the world:

๐Ÿ›‘ Stop.
โš™๏ธ Run your updates.
๐Ÿ™ Thanks!

Now if I tried to pass this text to a script within a Jamf policy, all those emoji would get mangled by Jamf as we saw above. Let’s use shef to encode the string for Jamf:

% shef <<'EOF'
heredoc> ๐Ÿ›‘ Stop.
heredoc> โš™๏ธ Run your updates.
heredoc> ๐Ÿ™ Thanks!
heredoc> EOF
\xF0\x9F\x9B\x91 Stop.\n\xE2\x9A\x99\xEF\xB8\x8F Run your updates.\n\xF0\x9F\x99\x8F Thanks!

For the example above I am using a “here-doc”; in practice you can simply supply shef a file path. The output encodes all the newlines in the ANSI-C style of \n and the emoji have all been replaced by their UTF-8 encodings using the hexadecimal escaping of \x. We can take this output and use it in our Jamf policy. The message will get through both textually and symbolically. As an additional feature bonus I added a simple open action in the script. Supply the file path to the Software Update panel and after the user clicks OK, it will be opened. Scope a policy like this to anyone with pending updates with Daily frequency:

Escape the mangling in Jamf!

If you are just running the script locally and calling from the Terminal you’ll want to specify a quoting option. Exclamations are tricky and shells usually love to interpret trailing exclamations as a history expansion command, shef does its best to avoid this:

bash-3.2$ shef -Qd <<'EOF'
> ๐Ÿ›‘ Stop.
> โš™๏ธ Run your updates.
> ๐Ÿ™ Thanks!
> EOF
"\xF0\x9F\x9B\x91 Stop.\n\xE2\x9A\x99\xEF\xB8\x8F Run your updates.\n\xF0\x9F\x99\x8F Thanks"\!""

bash-3.2$ ./simpleAlert-AS.sh 1 2 3 "\xF0\x9F\x9B\x91 Stop.\n\xE2\x9A\x99\xEF\xB8\x8F Run your updates.\n\xF0\x9F\x99\x8F Thanks"\!"" "Software Updates Pending" stop "/System/Library/PreferencePanes/SoftwareUpdate.prefPane"
Our un-mangled output

Stop the Mangling Madness!

Wrapping things up: use shef to encode strings with Unicode so they survive storage in Jamf’s Latin1 encoded db tables. shui my fuller featured AppleScript dialog tool is ready to accept shef encoded strings. You can also use shef minify your text for your shell script. In fact, I used it to encode the help text file down to a single quoted line for use within itself. That’s 1 happy customer and counting, hope you find it useful too!

secret origins: the jpt

On building a JSON tool for macOS without using Python, perl, or Ruby.

In my work as a Macintosh engineer and administrator I’ve noticed macOS has lacked a bundled tool for working with JSON at the command line. Where XML has its xpath, if your shell script needs some JSON chops, it’ll require an external binary like jq or something else scripted in Python, Ruby or perl using their JSON modules. The problem is, those runtimes have been slated for removal from a future macOS. So I took that as challenge to devise a method to query and modify JSON data within shell scripts, that didn’t use one of those deprecated scripting runtimes and didn’t require an external binary dependency either. Could I achieve robust and native JSON parsing on a Mac by simply “living off the land”?

Why not just re-install the runtimes when Apple deprecates them?

“Why limit yourself like this? Just re-install the runtimes and move on”, you may ask. Well, I’d like to think that limitations can inspire creativity but we should also consider there may be some other reasons why Apple is discontinuing the inclusion of those runtimes. Some may say, “Apple Silicon + macOS 11.0 is the perfect time for them to clean house”, to which I’d have to agree that’s a very good reason and likely a factor. Others could say they are looking to tighten the screws to keep out unsigned code: maybe, they do like to glue things shut! But really, I think it’s more akin to the web-plugins of yore like Java and Flash. Apple does not want to be the conduit for deploying 3rd party party runtimes which increase the attack surface of macOS. This seems like the most reasonable of the explanations. So, if you accept that Apple is attempting to reduce attack surface, why increase it by re-installing Python, Ruby, or perl, just so a transient script (like a Jamf Extension Attribute) can parse a JSON file? My answer to that, is you don’t. You play the hand you’re dealt. Game on!

Looking for truffles (in a very small back yard)

Despite having another project (shui) that can output and invoke Applescript from within a shell scripts for generating user interfaces, I definitely knew that Applescript was not the way to go. Apple however, added to the languages Open Scripting Architecture (OSA) supports back in 2014 with OS X Yosemite (10.10), they added Javascript along with a bridge to the Cocoa Objective-C classes and they called it JXA: Javascript for Automation. This seemed like a promising place, so I started playing with osascript and figured out how to load files and read /dev/stdin using JXA, and while looking for an answer for garbled input from stdin I came upon a Japanese blog that mentioned jsc the JavaScriptCore binary which resides in the /System/Library/Frameworks/ JavaScriptCore.framework. Arigato! Pay dirt! ๐Ÿค‘ jsc does exactly what we need it to do: It can interpret Javascript passed as an argument, can access the filesystem and read from /dev/stdin, and best of all is in non-Private System level Framework that exists all the way back to OS X 10.4! Just the kind of foundation on which to build the tool.

Homesteading jsc

The existence of jsc goes back all the way to OS X Tiger and it’s functionality has evolved over the years. In order to have a consistent experience in jpt from macOS 10.4 – 11.x+ a few polyfills had to be employed for missing functions, along with a few other workarounds regarding file loading, printing and exit codes (or lack thereof). Once those were addressed the jsc proved to be a highly optimized Javascript environment that’s blazingly fast. It spans 13 macOS releases and is even present in many Linux distros out-of-the-box (Ubuntu and CentOS) and can even be run on Windows when the Linux Subsytem is installed.

With the host environment sorted, I began working with the original JSONPath code by Stefan Goessner as the query language. I didn’t know about JSON Pointer yet so this strange beast was all I knew! It worked out really. I went full throttle into developing the “swiss army knife” of JSON tools. I really leaned into the Second System Effect, as described in the Mythical Man-month, it’s when you put every doodad, gizmo and doo-hicky in your 2nd product (my first simple JSON pretty-printer built in JS on jsc). Eventually though, after I reached a feature plateau, I came back around to the address the quirks of the original JSONPath code. I ended up rewriting signifigant chunks of JSONPath and released it as it’s own project: brunerd JSONPath. But I digress, let’s get back to the JSON Power Tool.

jpt: powers and abilities

At it’s most basic, jpt will format or “pretty print” any JSON given. jpt can also handle data retrieval from JSON document using either JSONPath or JSON Pointer syntax. JSONPath, while not a standard, is a highly expressive query language akin to XPath for XML, with poweful features like recursive search, filter expressions, slices, and unions. JSON Pointer on the other hand is narrower in focus, succint and easily expressed, and standardized but it does not offer any of the interogative features that I feel make JSONPath so intriguing. Finally, jpt can also modify values in a JSON document using standardized JSON Patch operations like: add, remove, replace, copy, move, and test, as well as the also standardized JSON Merge Patch operation. Altogether, the jpt can format, retrieve and alter JSON documents using only a bit of outer shell script plus a lot more Javascript on any macOS since 10.4! ๐Ÿ˜…

Where can I get the jpt?

Stop by the project’s GitHub page at: https://github.com/brunerd/jpt

There you will find the full source and also a minified version of the jpt for inclusion within your shell scripts. Since it’s never compiled you can always peer inside and learn from it, customize it, modify it, or just tinker around with it (usually the best teacher).

Future Plans

There will undoubtedly be continued work on the jpt. Surely there are less than optimal routines, un-idiomatic idioms, edge cases not found, and features yet to be realized. But as far as the core functionality goes though, it’s fairly feature complete in my opinion. Considering that one of my top 10 StrengthsFinder qualities is “Maximizer”, the odds are pretty good, I’ll keep honing the jpt‘s utility, size (smaller), and sharing more articles with examples on the kinds of queries and data alteration operations the jpt can so perform. Stay tuned!

New Projects: jpt and shui, Now Available

Between March and October 2020 I had some great ideas for command line Mac utilities the MacAdmin could apprecite and I had the time to devote to their realization. I’m excited to present these two open source projects, available on GitHub: jpt and shui. I hope they can add richness to your shell scripts’ presentation and capabilities without requiring additional external dependancies.

jpt – the “JSON Power Tool” is a Javascript and shell script polyglot that leverages jsc, the JavascriptCore binary that is standard on every Mac since 10.4 and since the jpt is purposefully written in ES5 to maintain maximum compatibility, why yes, this tool does run on both PPC and Intel Macs all the way back to OS X Tiger and then all the way forward to the latest 11.0 macOS Big Sur! Many Linux distros like CentOS and Ubuntu come with jsc pre-installed also, even Windows with the Linux Subsystem installed can run jsc and therefore can run the jpt!

What you can do with the jpt? Query JSON documents using either the simple yet expressive JSONPath syntax or the singular and precise JSON Pointer (RFC6901) syntax. The output mode is JSON but additional creative output modes can render JSONPaths, JSON Pointer paths, or even just the property names with their “constructor” types (try -KC with -J or -R) Textual output can be encoded in a variety of formats (hex/octal/URI encoding, Unicode code points, etc…), data can be modified using both JSON Patch (RFC6902) operations (add, replace, remove, copy, move, test) and also JSON Merge Patch (RFC7386) operations. JSON can be worked with in new ways, try -L for “JSONPath Object Literal” output to see what I mean. Or you simply feed jsc a file to pretty-print (stringify) to /dev/stdout. I’ll be writing more about this one for sure.
Github project page: jpt
Tagged blog posts: scripting/jpt

shui – first-class Applescript dialog boxes in your shell scripts without needing to remember esoteric Applescript phrasings! If you think it’s odd for code to have possessive nouns and are more comfortable in shell, you’re not alone. shui can be embedded in either bash or zsh scripts but it can also output Applescript if you really want to know how the sausage is made or want to embed in your script without shui. Hopefully shui will let you forget those awkward Applescript phrasing and focus on your shell script’s features and functionality. It uses osascript to execute the Applescript and launchctl to invoke osascript in the correct user context so user keyboard layouts are respected (vs. root runs). Check out the project page for demo videos and then give shui a try.
Project page: shui
Tagged blog posts: scripting/shui

macOS shell games: long live bash

TL;DR โ€“ Bash ain’t goin’ nowhere on Mac, both version-wise and in terms of its presence. Looking at the longevity of other shells on the system, it will likely be around for a good while longer.

There’s been a lot of hand wringing and angst online about bash and zsh becoming the new default shell. Some folks feel Apple is signaling deprecation and removal and have the crushing feeling they must convert all their bash script to zsh. I think that’s a bit unnecessary.

True, the default shell is changing from bash to zsh, as Apple notes here. This is indeed a Good Thingโ„ข as zsh shell has been one of the most frequently updated shells on macOS. Bash, on the other hand, has been stuck at varying versions of 3.2 for 12 years now! On the plus side, sysadmins have “enjoyed” predictable and stable behavior from bash during this time. Sure, you’d love new features but when you are scripting for the enterprise, across multiple OS versions, this is just the sort of thing you want: boringness and dependability.

As far as deprecations go, the only thing Apple has signaled as being deprecated eventually are scripting language runtimes (not shells):

Scripting language runtimes such as Python, Ruby, and Perl are included in macOS for compatibility with legacy software. Future versions of macOS wonโ€™t include scripting language runtimes by default, and might require you to install additional packages. If your software depends on scripting languages, itโ€™s recommended that you bundle the runtime within the app. (49764202)

Use of Python 2.7 isnโ€™t recommended as this version is included in macOS for compatibility with legacy software. Future versions of macOS wonโ€™t include Python 2.7. Instead, itโ€™s recommended that you run python3 from within Terminal. (51097165)

mac OS Catalina 10.15 Release Notes

I’ve done some digging and culled the shell versions from OS X 10.0 to macOS 10.15, along with their respective release dates. I think it shows that shells, no matter how old and crusty, tend to be long lived and not soon removed on macOS.

Here’s a quick way to check your shell versions (except for dash):

macOSzshbash/shcsh/tcshkshdash
10.03.0.8 (2000-05-16)3.0.8 (zsh, no bash)6.08.00 (1998-10-02)
10.13.0.8 (2000-05-16)3.0.8 (zsh, no bash)6.10.00 (2000-11-19)
10.24.0.4 (2001-10-26)2.05b.0 (2002-07-17)6.10.00 (2000-11-19)
10.34.1.1 (2003-06-19)2.05b.0 (2002-07-17)6.12.00 (2002-07-23)
10.44.2.3 (2005-03-00)2.05b.0 (2002-07-17)6.12.00 (2002-07-23)M p (1993-12-28)
10.54.3.4 (2007-04-19)3.2.17 (2007-05-01)6.14.00 (2005-03-23)M s+ (1993-12-28)
10.64.3.4 (2008-11-03)3.2.48 (2008-11-18)6.15.00 (2007-03-03)M s+ (1993-12-28)
10.74.3.11 (2010-12-20)3.2.48 (2008-11-18)6.17.00 (2009-07-10)M s+ (1993-12-28)
10.84.3.11 (2010-12-20)3.2.48 (2008-11-18)6.17.00 (2009-07-10)JM 93u (2011-02-08)
10.95.0.2 (2012-12-12)3.2.51 (2010-03-17)6.17.00 (2009-07-10)JM 93u (2011-02-08)
10.105.0.5 (2014-01-06)3.2.57 (2014-11-07)6.17.00 (2009-07-10)AJM 93u+ (2012-08-01)
10.115.0.8 (2015-05-31)3.2.57 (2014-11-07)6.18.01 (2012-02-14)AJM 93u+ (2012-08-01)
10.125.2 (2015-12-02)3.2.57 (2014-11-07)6.18.01 (2012-02-14)AJM 93u+ (2012-08-01)
10.135.3 (2016-12-12)3.2.57 (2014-11-07)6.18.01 (2012-02-14)AJM 93u+ (2012-08-01)
10.145.3 (2016-12-12)3.2.57 (2014-11-07)6.18.01 (2012-02-14)AJM 93u+ (2012-08-01)
10.155.7.1 (2019-02-03)3.2.57 (2014-11-07)6.21.00 (2019-05-08)AJM 93u+ (2012-08-01)dash-9 (1993)

As you can see, zsh has has been updated with almost every new release of macOS. Bash really hit a wall with 3.2 and as many have noted, it was v4’s change in licensing to GPLv3 that caused this (sh is really bash in sh compatibility mode so the versions are intertwined). csh/tcsh has the same duality thing going on and took a notably giant 7 year leap in 10.15 to a version from 2019. ksh has remained at the same version just as long as bash yet I don’t think anyone is fretting that ksh will be deprecated or removed. Finally, dash is just a weirdo that apparently disdains versioning! I used a combination of what /bin/dash and man dash to get some sort of crude answer.

So there you go: In my opinion, all signs point to bash being yet another shell on macOS for some time. Removing bash from macOS would break a lot of stuff and while that reason alone hasn’t stopped Apple before, I think they will let sleeping dogs lie. Go ahead and learn zsh, master it, customize it, or make it “sexy” but take the rumors of its demise on macOS with a grain of salt and dose of skepticism.