Exploring Unicode in macOS with clui

My new tool clui, pronounced “clue-ee” offers Command Line Unicode Info with the ability to export to a variety of formats like CSV, JSON, YAML, RTF and more. While I’ve written a few macOS command line tools geared to the Mac Admin like jpt the JSON power tool, ljt the little JSON tool, shui for easily adding AppleScript dialogs to your shell script, and most recently shef a Unicode text encoder and formatter for shell scripters. This is one is almost “just for fun” although you might find some practical uses for it. Writing shef opened my eyes to the stunning amount of detail and craftsmanship in macOS’ Unicode-aware fonts, which comprise not just the alphabets of the world but signs, symbols, and even Egyptian hieroglyphics! While macOS’s built-in Character Viewer does a pretty good job to group and display these characters it’s a painstakingly manual process if you want to get info on a range of characters. I hope clui makes it fun and easy to poke around the vast Unicode neighborhood.

If you don’t feel like reading you can watch this.

Starting with Character Viewer

First let’s take a look at Character View, perhaps you don’t know some of it’s features. If you quickly press the dedicated “globe” 🌐 button on newer Macs (or the keyboard combo Control-Command-Spacebar on older models) it will likely open up Character Viewer in it’s default mini-sized version with preset categories along the bottom. You can click on a symbol or character and it will insert the text in your current app. The real fun begins when you click the little icon in the upper right to expand the view.

The window will expand to show more information: The code point(s) in hexadecimal (U+hhhh), the UTF-8 encoding bytes, and related characters. You will need to double-click these to insert them into your current app.

You can also Customize the List of categories that appear in the left column by clicking the encircled ellipsis…

The one that has everything in it is Unicode under the Code Tables group at the bottom of the list.


You can right-click or control-click on a character to Copy Character Info into the clipboard. In the example below, we discover that what most Mac folks would call ⌘ “Command” is also known as the “Place of Interest Sign”. Whaddya know!

This is all well and good but who’d want to do that for thousands of characters?! What if someone wanted this info at the command line? It got me thinking: There’s Got to Be a Better Way!™

Searching for the Source

The first order of business was looking for where macOS kept it’s naming information and if it was possible to extract that information with command line tools. It ended up being in two files, a SQLite database and a plist. Here’s their full paths:

#Single code point characters and Unicode symbols
/System/Library/Input Methods/CharacterPalette.app/Contents/Resources/CharacterDB.sqlite3
#plist of single and multiple code point Emoji

There is some overlap with CharacterDB.sqlite3 and AppleName.strings where Apple choose to use different phrasing for an Emoji vs. Unicodes name, but in general the former has single code point entries, while the latter has both single and the multi-codepoint Emoji sequences. clui will report on both, putting a semi-colon in between each version, you can also output discretely with the -D option. Descriptions/info fields are converted from uppercase to lowercase BECAUSE WHO LIKES GETTING YELLED AT?! 🙉 Although you can preserve case with -p which can help with deciphering the internal (and unlabeled!) columns Apple uses in the descriptions of the CJK Ideographic ranges.

clui in Action: Practical Examples

Simple Lookups

If you’d like to see what all the options and modes run clui -u to get the “usage” output, or take a look at on clui’s GitHub page. To start with, clui is built to ingest both “regular” characters and also representation of Unicode code points in hexadecimal in the style of: U+hhhhh or 0xhhhhh. Ranges can also be specified by simply adding a hyphen between to characters or codepoints and it can both ascend and descend. The default output is CSV.

Apple doesn’t want to call anyone a nerd apparently, but that’s OK I embrace it.

For multiple code point emoji you can enclose the code point representations in quotes. Spaces within quotes are only used delimit each code point and are not part of the composite character.

The -X expansion option will display the all the code points together, then break out each component

-x will expand the input and break out each code point without showing the composite character

Working with Categories and Groups

clui can tap into categories and groups by leveraging the the plists inside the Resources folder of CharacterPallette.app. These plists contain a mix of 0x code point representations and literal characters. The list option -L takes the upper and lowercase arguments of c or g for exactly what you think, categories and groups! -Lc gets categories marking those with internal subsections with an asterisk * and -LC will expand those categories to include the subsections within. All list outputs are CSV and includes a header row, -h will suppress headers. In this excerpted example below you can see Arrows contains a number of subsections:

clui acts on a categories when you use the -C option and input one or more categories. If no subsection is specified the complete category will be output. Some can be quite large and take several minutes to output! If you are outputting RTF or JSON and redirecting to a file, if you interrupt with Control-C the output will be properly closed up, so it will still be valid.

As you can see at the bottom of the list, not all characters have renderings. However, if you double click those question mark glyphs ⍰ to select and copy them (CSV is nice that way), you will get that exact character. I used the apl “quad question” character (U+2370) above to get something close to it for this example, but don’t let the generic visual representation fool you, it is unique. You can use Character Viewer to see if any other fonts have alternate graphical representations, since Terminal is using only the currently selected font to display output. Update: It will fall back to a font that has a representation if needed. I recommend the free GNU Unifont for “Glyphs above the Unicode Basic Multilingual Plane”, which fills in some of the gaps of Apple fonts (like the LCD-like segmented numbers U+1FBF0-U+1FBF9)

Groups are about the same thing as categories except groups are comprised of multiple categories. -Lg lists all groups and -LG will expand the constituent categories within.

You can use -G like -C specifying one or more groups. If you include a category name (comma delimited) after the group, it’ll simply report that category as if you’d used it solely with -C. Here’s an example of the first group AdditionalModernScripts and the member category CanadianAboriginalSyllabics . For variety I’ve added -h to hide the header row from the CSV output.

CSV Looks Great in QuickLook

Just in case you didn’t know, Quicklook will display files with a csv extension really nicely. Here’s the CanadianAboriginalSyllabics above as seen in Quicklook, you can even select characters from within Quicklook, pretty nice!

CSV Can Look Even Better in Numbers

If you want to work with CSV in Numbers it will do quite nicely. Here’s a little tip for better legibility:

  • Select column A from the top of the column
  • Command ⌘ click on A1 to deselect it
  • Format the text to a larger size (like 50 points or higher)
  • Adjust the column width a bit wider
  • Save it as a .numbers file to retain formatting
Some of these Emoji will make you hungry

Pro Tip: Quicklook will not let you select any text from a Numbers document! However if you click and drag anywhere in the Quicklook contents to the Desktop (or into Preview), it will export a seamless PDF with no page breaks! Then you can select text from the PDF in QuickLook. Who knew?! Now back to clui features!

Get a Good Look at Those Emojis in Rich Text Format!

A late entry feature to clui is RTF (Rich Text Format) output. This enables clui to present the characters in larger sizes without requiring additional work from you (as seen above in Numbers)! The format is the same as the “plain” output (-Op) which does not label the fields, simply use the -Or option

Your best bet with RTF output is to either redirect it to a file like this: clui -Or -C Emoji > Emoji.rtf or pipe it into pbcopy like so: clui -Or -C Emoji | pbcopy. There’s a neat feature in pbcopy that detects the RTF header data and allows you to paste into TextEdit as rich text. You can also specify the font sizes withthis option: -f <char size, info size>

Searching by Description

clui can also search descriptions for multiple words and phrases. In this example I’ll search for magic, castle, and “clock face” using the -Sd (search descriptions) option. If I had searched just for clock I’d have also gotten hits for “clockwise arrows” since it search for substrings.

Searching by Character

Now, if you search for the usage of a single alphabetical character you’ll probably get one hit but macOS also has a database of “related characters” which are similar look-alike letters. You know, like the Subject lines of spam: “𝔅𝗢𝔾𝕆 𝐒𝖆𝓵𝙚 ‼”. Let’s use -SC to search for "a" plus related characters. I’ve trimmed the output to get some of the more interesting characters in this screenshot

When you start searching for symbols you’ll start discovering Emojis constructed using existing symbols and zero-width joiners (ZWJ, U+200D), these are called ZWJ Sequences (and ZWJ is apparently pronounced “zwidge”) . Let’s use -Sc to search for anything with the female sign ♀.

Let’s examine “woman surfer” (BTW: 100 Foot Wave Season 2 is awesome!) with the -X option to expand all component code points with the complete glyph as-is at the top:

The first line has the Emoji sequence as-is, then each individual component that comprises it: a gender neutral surfer (U+1F3CF), a ZWJ (U+200D), the female sign (U+2640), and variation selector 16 (U+FE0F). You may also notice one of the quirks of Terminal: Sometimes pasted input does not fully render as a unified Emoji!

Fitzpatrick Modifiers for Skin Color

To be honest I’d never heard of the Fitzpatrick scale before working on clui! It’s simply a scale of 6 types of skin color. The Unicode modifier combines Types 1 & 2 into U+1F3FB. Let’s demonstrate a search by description and then again with their ranges. Simply specify the range using standard Unicode notiation U+hhhhh with a hyphen in between.

If you search for an Emoji with a Fitzpatrick Modifier you won’t get a hit in the databases. As a convenience clui will detect this and remove the modifier to get the description. Using the -F will also display the sequence without the modifier. We can combine it with -X for expanded output with a a summary.

As you can see the sequence without the modifier is shown, then the original sequence with the Fitzpatrick modifier then each component part: surfer, Fitzpatrick modifier type-6, a ZWJ, female sign, and a variation selector.

On Variation Selectors

To be honest I’d never known about variation selectors before working on this, more info on Emoji Presentation Sequences can be found here. The two most common variation selectors are pretty easy to understand: U+FE0E is “variation selector 15” and it is used to explicitly specify the text/non-graphic version and U+FE0F, “variation selector 16” gives you the emoji/graphic version. Watch (U+231A) is a good example of this. It’s at the discretion of the OS on how to render a glyph. In this case U+231A is rendered in the emoji style of an Apple Watch⌚️by default. When combined with U+FE0E, it turns into ye olde Mac OS watch ⌚︎, this is the “text version”. Adding U+FE0F does nothing to change the appearance since it was already rendered in the emoji style without it.

If you’d like to search for every character with a variation selector that is tracked in macOS’s database, you can run this query: clui -Sc U+FE0E U+FE0F A definitive list can be found here.

Encoding Options

So far we’ve just seen the default uppercase hexadecimal (-Eh) UTF-8 encoding. In the vein of shef, clui can output in various styles of encoding: \x hex escapes (-Ex), octal \nnn (-Eo), leading zero octal \0nnn (-E0), UTF-16 Javascript encoding (-u) and zsh style UTF-32 \U code points (-EU).

Surfing safari, encoding party! 🤙

Formatting output

Besides the beautiful RTF output and functional CSV output, clui can also output characters simply space delimited, without any other data (-Oc).

JSON (-Oj) and JSON Sequences (-OJ) can be had as well. The difference between JSON and JSON Sequence is that JSON will be an array of objects, whereas JSON Sequences are JSON objects delimited with U+1E the “record separator” as ASCII calls it or “information separator two” as Unicode knows it to be and newlines. Both jq and jpt can handle JSON sequneces.

JSON (-Oj)
JSON Sequence (-OJ)

Lastly we have YAML (-Oy) the superset cousin of JSON. My JSON string encoder jse gets some use in clui to encode strings and descriptions for these output modes.

YAML, cousin of JSON

Making it work for you (and me)

I spent a lot of time trying to make clui work in an intuitive way, it replicates the core features of Character Viewer with plenty of bonus functionality thrown in. It may not be something you use daily, but it might come in handy when you get an email from let’s say, tim@аррlе.com. You could run that string through clui in “expand” mode (-x) to analyze on each letter in the string. You might be surprised and perhaps disappointed that perhaps “Tim Apple” did not send you that email.

Cyrillic look-a-likes! 🔎

Or maybe you’d like to figure out the secrets of Zalgo Text or perhaps what characters are in ¯\_(ツ)_/¯ or make a catalog of Emoji in RTF: clui can do it! Head on over to the clui Github page and download the Release if you’d like to try it out on your (Monterey+) Mac, thanks!


Since you made it down here, how about a one-liner that will create RTFs of all the Unicode categories? It will likely take several hours and will open a Finder window when finished.

#make RTFs of every Unicode category, this might take a few hours, if you want to cancel close the Terminal window
mkdir ~/Desktop/clui-rtfs; cd ~/Desktop/clui-rtfs; IFS=$'\n'; for category in $(clui -LC | grep Unicode,); do clui -Or -C "$category" > "$category.rtf"; done; open -R .

Determining eligible macOS versions via script

Every year Mac admins wonder which Macs will make the cut for the new MacOS. While it’s no mystery which models those are, if you’ve got Jamf you’ll be wondering how to best scope to those Macs so you can perhaps offer the upgrade in Self Service or alert the user to request a new Mac! One way to scope a Smart Group is with the Model Identifier criteria and the regex operator, like this one (I even chipped in!). It doesn’t require an inventory and results are near instant. Before I was any good at regex though, I took another route and made an Extension Attribute macOSCompatibility.sh, where the Mac reports back to Jamf. (It also has a CSV output mode for nerdy fun!) Both methods however require manual upkeep and are now somewhat complicated by Apple’s new use of the very generic Macxx,xx model identifier which doesn’t seem to follow the usual model name and number scheme of major version and minor form factor variants (on purpose me-thinks!). Let’s look at some new methods that don’t require future upkeep.

Using softwareupdate –list-full-installers

macOS Big Sur (11) introduced a new command softwareupdate --list-full-installers which shows all eligible installers available for download by the Mac running that command. The funny thing about this is that even though it is a Big Sur or newer feature, if the hardware is old enough, like a 2017, it offer versions all the way back to 10.13 High Sierra! Monterey added build numbers to the output and it can be easily reduced to just versions with awk:

softwareupdate --list-full-installers | awk -F 'Version: |, Size' '/Title:/{print $2}'

This one-liner can be made into a simple function. I’ve added a uniq at the end for case where two differing builds have the same version, like 10.15.7. Here’s that function in a script with a little version check: getSupportedMacOSVersions_SWU.sh

getSupportedMacOSVersions_SWU - Copyright (c) 2022 Joel Bruner
Licensed under the MIT License

function getSupportedMacOSVersions_SWU()( 
#getSupportedMacOSVersions_SWU - uses softwareupdate to determine compatible macOS versions for the Mac host that runs this
	if [ "$(sw_vers -productVersion | cut -d. -f1)" -lt 11 ]; then echo "Error: macOS 11+ required" >&2; return 1; fi
	#get full installers and strip out all other columns
	softwareupdate --list-full-installers 2>/dev/null | awk -F 'Version: |, Size' '/Title:/{print $2}' | uniq

Output from Apple Silicon will never include 10.x versions

Software Update (SWU) Based Extension Attribute for Jamf

The possible inclusion of 10.x versions in the output complicates things a bit. In ye olden OS X days, the “minor version” (after the first period) acted more like the major versions of today! Still it can be done, and we will output any macOS 10.x versions, as if they are major versions like macOS 11, 12, 13, etc. Here’s getSupportedMacOSVersions-SWU-EA.sh

getSupportedMacOSVersions-SWU-EA (Extension Attribute) - Copyright (c) 2022 Joel Bruner
Licensed under the MIT License

function getSupportedMacOSVersions_SWU()( 
#getSupportedMacOSVersions_SWU - uses softwareupdate to determine compatible macOS versions for the Mac host that runs this
	#[ "$(sw_vers -productVersion | cut -d. -f1)" -lt 11 ] && return 1
	if [ "$(sw_vers -productVersion | cut -d. -f1)" -lt 11 ]; then echo "Error: macOS 11+ required" >&2; return 1; fi
	#get full installers and strip out all other columns
	softwareupdate --list-full-installers 2>/dev/null | awk -F 'Version: |, Size' '/Title:/{print $2}'

#get our version

#depending on the model (2020 and under) we might still get some 10.x versions 
if grep -q ^10 <<< "${all_versions}" ; then versions_10=$(awk -F. '/^10/{print $1"."$2}' <<< "${all_versions}")$'\n'; fi
#all the other major versions
version_others=$(awk -F. '/^1[^0]/{print $1}' <<< "${all_versions}")

#echo without double quotes to convert newlines to spaces
echo "<result>"$(sort -V <<< "${versions_10}${version_others}" | uniq)"</result>"
The venerable 2017 MacBook Pro has quite a span

Now if you wanted to make a Jamf Smart Group for those that could run macOS 13 you wouldn’t want to match 10.13 by accident. You could comment out the line in the script that matches versions beginning with ^10 or you could enclose everything in double quotes for the echo on the last line, so the newlines remained or you could use regex to match ([^.]|^)13 that is: not .13 or if the hardware is so new ^13 is at the very beginning of the string. As 10.x capable hardware fades away such regex sorcery shouldn’t be needed.

Using the Apple Software Lookup Service

“What’s the Apple Software Lookup Service?!”, you may be asking? I myself asked the same question! It’s a highly available JSON file that MDM servers can reference. If softwareupdate is acting up or hanging (and it’s been known to do so!), you have all you need in this JSON file to do a little sanity checking of softwareupdate too if you’d like. The URL is found in the Apple MDM Protocol Reference and it contains versions, models and their compatibility.

This method has far fewer patch and point versions than the softwareupdate method above and the OSes start with Big Sur (11). No 10.x versions are in the ASLS. There are two “sets” in ASLS, PublicAssetSets, which has only the newest release of each major version and AssetSets which has additional point releases (use the -a option for this one). plutil has quirky (IMO) rules for what it will and will not output as json but the raw output type can get around. It was introduced in Monterey and it can also be used to count array members, it’s goofy but manageable. Richard Purves has an article on that here. The code is generously commented, so I won’t expound upon it too much more, here’s getSupportedMacOSVersions_ASLS.sh

getSupportedMacOSVersions_ASLS - Copyright (c) 2022 Joel Bruner
Licensed under the MIT License...

function getSupportedMacOSVersions_ASLS()( 
#getSupportMacOSVersions - uses Apple Software Lookup Service to determine compatible macOS versions for the Mac host that runs this
#  Options:
#  [-a] - to see "all" versions including prior point releases, otherwise only newest of each major version shown

	if [ "${1}" = "-a" ]; then

	#get Device ID for Apple Silicon or Board ID for Intel
	case "$(arch)" in
			#NOTE: Output on ARM is Device ID (J314cAP) but on Intel output is Model ID (MacBookPro14,3)
			myID=$(ioreg -arc IOPlatformExpertDevice -d 1 | plutil -extract 0.IORegistryEntryName raw -o - -)
			#Intel only, Board ID (Mac-551B86E5744E2388)
			myID=$(ioreg -arc IOPlatformExpertDevice -d 1 | plutil -extract 0.board-id raw -o - - | base64 -D)

	#get JSON data from "Apple Software Lookup Service" - https://developer.apple.com/business/documentation/MDM-Protocol-Reference.pdf
	JSONData=$(curl -s https://gdmf.apple.com/v2/pmv)

	#get macOS array count
	arrayCount=$(plutil -extract "${setName}.macOS" raw -o - /dev/stdin <<< "${JSONData}")

	#look for our device/board ID in each array member and add to list if found
	for ((i=0; i<arrayCount; i++)); do
		#if found by grep in JSON (this is sufficient)
		if grep -q \"${myID}\" <<< "$(plutil -extract "${setName}.macOS.${i}.SupportedDevices" json -o - /dev/stdin <<< "${JSONData}")"; then
			#add macOS version to the list
			supportedVersions+="${newline}$(plutil -extract "${setName}.macOS.${i}.ProductVersion" raw -o - /dev/stdin <<< "${JSONData}")"
			#only set for the next entry, so no trailing newlines

	#echo out the results sorted in descending order (newest on top)
	sort -rV <<< "${supportedVersions}"

#pass possible "-a" argument
getSupportedMacOSVersions_ASLS "$@"
PublicAssetSets (top) vs. AssetSets (bottom)

The fact that this method requires Monterey for the plutil stuff didn’t agree with me, so I made a version that uses my JSON power tool (jpt) so it will work on all earlier OSes too. It’s a tad large (88k) but still runs quite fast: getSupportedMacOSVersions_ASLS-legacy.sh

Just as with the softwareupdate based function, the same can be done to reduce the output to only major versions and since it is v11 and up, a simple cut will do!

#major versions only, descending, line delimited
getSupportedMacOSVersions_ASLS | cut -d. -f1 | uniq

#major versions only ascending
echo $(getSupportedMacOSVersions_ASLS | cut -d. -f1 | sort -n | uniq)

ASLS Based Extension Attribute

The Apple Software Lookup Service (ASLS) JSON file itself doesn’t care what version of macOS a client is on, but the methods in plutil to work with JSON aren’t available until Monterey. So here’s the ASLS based Extension Attribute a couple ways: getSupportedMacOSVersions-ASLS-EA.sh and getSupportedMacOSVersions-ASLS-legacy-EA.sh both get the job done.

Same. Same.

Bonus Methods for Determining Board ID and Device ID

The ASLS method requires either the Board ID or the Device ID and in a way that worked across all macOS versions and hardware architectures. I’ve updated my gist for that (although gists are a worse junk drawer than a bunch of scripts in a repo if only because it’s hard to get an overall listing) and here’s a few callouts for what I came up with

#DeviceID - ARM, UNIVERSAL - uses xmllint --xpath
myDeviceID=$(ioreg -arc IOPlatformExpertDevice -d 1 | plutil -extract 0.IORegistryEntryName xml1 -o - - | xmllint --xpath '/plist/string/text()' - 2>/dev/null)
#DeviceID - ARM, macOS 12+ only, uses plutil raw output
myDeviceID=$(ioreg -arc IOPlatformExpertDevice -d 1 | plutil -extract 0.IORegistryEntryName raw -o - -)
#NOTE: Different output depending on platform! 
# ARM gets the Device ID - J314cAP
# Intel gets the Model ID - MacBookPro14,3

#Board ID - Intel ONLY, Mac-551B86E5744E2388
#Intel, UNIVERSAL - uses xmllint --xpath
myBoardID=$(ioreg -arc IOPlatformExpertDevice -d 1 | plutil -extract 0.board-id xml1 -o - - | xmllint --xpath '/plist/data/text()' - | base64 -D)
#Intel, macOS 12+ only - uses plutil raw output 
myBoardID=$(ioreg -arc IOPlatformExpertDevice -d 1 | plutil -extract 0.board-id raw -o - - | base64 -D)

Wrapping Up

This was all really an excuse to play around with the Apple Software Lookup Service JSON file, what can I say! And not just to plug jpt either! It was fun to use the new plutil raw type too (is fun the right word?) and “live off the land”. Be aware that the newest macOS version only appears once it’s publicly released, so keep that in mind when scoping. You can still keep scoping in Jamf via Model Identifier Regex or by my older extension attribute just keep in mind you’ll need to update them yearly. Whereas, these newer EAs based on either softwareupate (getSupportedMacOSVersions-SWU-EA.sh) or ASLS (getSupportedMacOSVersions-ASLS-EA.sh, getSupportedMacOSVersions-ASLS-legacy-EA.sh) should take care of themselves into the future.

Detecting and affecting Lockdown Mode in macOS Ventura

Lockdown mode is new feature for macOS Ventura and for many MacAdmins we’ve been wondering how to detect this state. Why? Lockdown mode affects how macOS and Mac apps behave. This is something a helpdesk might like to know when trying to troubleshoot an issue. Also, due to some ambiguous wording by Apple, they made it seem like MDM Config Profiles could not be installed at all when in Lockdown mode, however this is not always the case. The hunt was on!

Detecting Lockdown Mode

I was looking everywhere last week: ps process lists, nvram, system_profiler, kextstat, launchctl, sysdiagnose, a defaults read dump, etc. I was looking high and low for “lock” “down” and “mode” and I got a hit in the com.apple.Safari domain in the sandboxed ~/Library/Containers/Safari path. While it turns out that Safari will in some cases write the button label LockdownModeToolbarIdentifier to that pref domain, it requires Safari to be launched and for the toolbar to be in non-default layout, otherwise the label name is never written! So that was a dead end.

Then a little birdie on MacAdmins pointed me in the right direction and blogged about it and wrote a Jamf extension attribute! 😅 Turns out I had missed the value sitting at the top of the defaults read dump! (d’oh) It was there the whole time in .GlobalPreferences, I just hadn’t done a diff like I should have! That would have revealed the key uses the LDM acronym/mnemonic: LDMGlobalEnabled Funnily enough, when I searched for this key on Google I got 5 hits and all of them for iOS, like this one at the Apple dev forums. However they were all about Swift and iOS, here’s how to do it in shell for the current user:

defaults read .GlobalPreferences.plist LDMGlobalEnabled 2>/dev/null

It’s a boolean value, that will not exist if Lockdown mode has never been enabled, when enabled it will report 1 from defaults and when disabled the key will remain and report 0. What stands out is that this is a per-user preference. Since it makes you reboot I had supposed it was a system-wide setting but sure enough if you log out and into another user, Lockdown mode is disabled. Perhaps that makes sense but I’m not quite sure about that?

Affecting Lockdown Mode

This totally blew me away: You can enable and disable macOS Lockdown mode by writing to your .GlobalPreferences preference domain!

#turn lockdown mode off
defaults write .GlobalPreferences.plist LDMGlobalEnabled -bool false
#turn lockdown mode on
defaults write .GlobalPreferences.plist LDMGlobalEnabled -bool true

That’s right, it’s not written to a rootless/SIP protected file like TCC.db! Just run the command as the user and it’ll turn toggle the behavior for most things. Here’s some details of my findings:

  • Configuration profiles – a restart of System Settings is not required, it will prohibit the manual installation of a .mobileconfig profile file. When Apple says “Configuration profiles can’t be installed” this is what they mean: User installed “double-click” installations of .mobileconfig files cannot be done. When they say “the device can’t be enrolled in Mobile Device Management or device supervision while in Lockdown Mode”, this only applies to these user-initiated MDM enrollments using a web browser that downloads .mobileconfig files. Lockdown mode does not prohibit enrollment into MDM that’s assigned via Apple Business Manager (ABM/DEP). You can initiate enrollment with the Terminal command: sudo profiles renew -type enrollment A Mac in Lockdown mode will be able to successfully enroll into an MDM assigned by ABM. Once enrolled, new Config Profiles can also be installed via that same MDM, even in Lockdown Mode.
  • Messages – a restart of Messages is not required, all messages will be blocked immediately, attachments or not. I’m not sure if that’s a bug or not since Apple only mentions attachments, not plain messages. It does not matter if the sender is in your Contacts or whether you have initiated contact with them before (like in Facetime). Messages will be delivered to any other devices not in Lockdown mode. If Lockdown mode is turned off, those blocked messages may be delivered if sent recently enough but will appear out of sequence. For example, a device that never had Lockdown Mode turned on would see messages: 1,2,3,4,5 while a device that turns it on and then off would see: 1,2,5,3,4
  • Facetime – restart not required, it will immediately begin blocking calls from anyone you have not called previously from that device. Unlike Messages though, it will show a Notification of the blockage.
  • Safari – app restart required. This differs from everything else, however Safari also gives the best visual indications that Lockdown mode is enabled! On the Start Page you’ll see “Lockdown Ready”, once at at website you’ll likely see “Lockdown Enabled” unless you’ve uncheck Enable Lockdown Mode in the top menubar SafariSettings for <site>… in which case you’ll see “Lockdown Off” in red.
Safari’s Lockdown Mode Toolbar states
  • Safari – Another subtle visual cue of Lockdown mode, that aligns with Apple’s “web fonts might not be displayed” guidance, can be seen on a Jamf user-initiated MDM enrollment screen, instead of a check mark you’ll see a square, take heed and turn back now! Since once you get the .mobileconfig files and fumble your way to System SettingsPrivacy & Security, scroll to the bottom of the list to Profiles (UX gripe: it used to just open the dang panel when you double clicked on them!) you’ll be blocked from installing it as seen above.
  • System Settings – an app restart is required for Privacy & Security to reflect the current state of LDMGlobalEnabled, if it was on and you disable via defaults once you launch System Settings again, it’ll let you turn it back on with a reboot and everything!

Wrapping Up

I didn’t try out the other lockdown mode behaviors for things like new Home management invitations or Shared Albums in Photos. Still it’s quite surprising that despite the System Settings GUI making you reboot to turn it on, Lockdown mode is a per-user setting that can seemingly be enabled and disabled dynamically with a simple defaults command run by the user. With the exception of Safari and System Settings it does not require Messages and Facetime to restart! There might be other caveats, it’s hard to tell. Perhaps this is all in the realm of “works as designed” for Apple but when you, the customer, don’t know what that exact design is, it can be quite a surprise!

Update: Looks like they started explaining a bit more about what happens when you enabled Lockdown Mode in macOS Sonoma

One more (unrelated) thing…

Update: As of Dec 13, 2022 Bug 174931 – Implement RegExp lookbehind assertions has been marked “Resolved” and the extensive pull request has all the gory details of the extensive refactoring that was done to implement this. 🎉 Thanks Michael Saboff! 🙏 Now when this will make it into Safari…. we’ll see.

Update 2: Safari 16.4, released March 27, 2023, now supports RegExp lookbehind assertions!

Since this post might get a few eyeballs, I’d also like to shine a light on the perplexing fact that Safari is the only browser that still doesn’t support the four year old ES2018 feature of RegExp lookbehind assertions?! I mean, sure it was a Google engineer who kindly filed this heads up to the WebKit team back in July of 2017 when it was a draft and a full year before it was ratified (Bug 174931 – Implement RegExp lookbehind assertions) but even a silly corporate rivalry couldn’t explain the seeming obstinance in letting this feature languish. I don’t get it, it just doesn’t make sense! There’s a nicely visualized page of where things stand and Safari is keeping company with IE 11 on this one.

Make these red islands green, Apple!

So take a look at the comments on the WebKit bug, some are quite funny, others just spot on, and there’s even one from yours truly. Perhaps add your own? Maybe when a bug gets 100 comments something special happens and we all get cake? 🎂

Determining iCloud Drive and Desktop and Documents Sync Status in macOS

I’m on a roll with iCloud stuff. In this post I’d like to show you how you can determine if either iCloud Drive is enabled along with the “Desktop and Documents Folders” sync feature. While you can use MDM to turn these off, perhaps you like to know who you’d affect first! Perhaps the folks in your Enterprise currently using these features are in the C-Suite? I’m sure they’d appreciate a heads up before all their iCloud docs get removed from their Macs (when MDM disallowance takes affect it is swift and unforgiving).

iCloud Drive Status

When it comes to using on-disk artifacts to figure out the state of macOS I like to use the analogy of reading tea leaves. Usually it’s pretty straightforward but every now and then there’s something inscrutable and you have to take your best guess. For example iCloud Drive status is stored in your home folder at ~/Library/Preferences/MobileMeAccounts.plist but yet the Accounts key is an array. The question is how and why you could even have more than one iCloud account signed-in?! Perhaps a reader will tell me when and how you would ever have more than one? For now it seems inexplicable.

Update: It seems quite obvious now but as many folks did point out, of course you can add another iCloud account to Internet Accounts and it can sync Mail, Contacts, Calendars, Reminders, and Notes, just not iCloud Drive. Only one iCloud Drive user per user on a Mac. While I do have another AppleID I only use it for Media and Purchases and none of the other iCloud services. The code below stays the same as it is looking at all array entries. Aside: Boy, do I wish I could merge or transfer my old iTunes purchasing/media Apple ID with my main iCloud Apple ID! <weakly shakes fist at faceless Apple bureaucracy that for some reason hasn't solved the problem of merging Apple IDs>

Regardless of that mystery jpt can use the JSONPath query language to get us an answer in iCloudDrive_func.sh and the minified iCloudDrive_func.min.sh. Below is an edited excerpt:

#Joel Bruner - iCloudDrive.func.sh - gets the iCloud Drive status for a console user


function iCloudDrive()(

	#for brevity pretend we've pasted in the minified jpt function:
	#for the full function see https://github.com/brunerd/macAdminTools/tree/main/Scripts

	consoleUser=$(stat -f %Su /dev/console)

	#if root grab the last console user
	if [ "${consoleUser}" = "root" ]; then
		consoleUser=$(/usr/bin/last -1 -t console | awk '{print $1}')

	userPref_json=$(sudo -u $consoleUser defaults export MobileMeAccounts - | plutil -convert json - -o -)

	#pref domain not found an empty object is returned
	if [ "${userPref_json}" = "{}" ]; then
		return 1
		#returns the number paths that match
		matchingPathCount=$(jpt -r '$.Accounts[*].Services[?(@.Name == "MOBILE_DOCUMENTS" && @.Enabled == true)]' <<< "${userPref_json}" 2>/dev/null | wc -l | tr -d "[[:space:]]")
		if [ ${matchingPathCount:=0} -eq 0 ]; then
			return 1
			return 0
# MAIN #

#example function usage, if leverages the return values
if iCloudDrive; then
	echo "iCloud Drive is ON"
	exit 0
	echo "iCloud Drive is OFF"
	exit 1

The magic happens once we’ve gotten the JSON version of MobileMeAccount.plist I use jpt to see if there are any objects within the Accounts array with Services that have both a Name that matches MOBILE_DOCUMENTS and have an Enabled key that is set to true, the -r option on jpt tells it to output the the JSON Pointer path(s) the query matches. I could have used the -j option to output JSONPath(s) but either way a line is a line and that’s all we need. Altogether it looks like this: jpt -r '$.Accounts[*].Services[?(@.Name == "MOBILE_DOCUMENTS" && @.Enabled == true)]

Again because Accounts is an Array we have it look at all of them with $.Accounts[*]and in the off chance we get more than one we simply say if the number of matches is greater than zero then it’s on. This works very well in practice. This function could best be used as a JAMF Extension Attribute. I’ll leave that as a copy/paste exercise for the reader. Add it to your Jamf Pro EAs, let sit for 24-48 hours and check for results! ⏲ And while some of you might balk at a 73k Extensions Attribute, the execution time is on average a speedy .15s!

#a pretend iCloudDrive Jamf EA 
#pretend we've pasted in the function iCloudDrive() above

if iCloudDrive; then

echo "<result>${result}</result>"

iCloud Drive “Desktop and Documents” status

Thankfully, slightly easier to determine yet devilishly subtle to discover, is determining the status of the “Desktop and Documents” sync feature of iCloud Drive. After searching in vain for plist artifacts, I discovered the clue is in the extended attributes of your Desktop (and/or Documents) folder! You can find the scripts at my GitHub here iCloudDriveDesktopSync_func.sh and the minified version iCloudDriveDesktopSync_func min.sh

#Joel Bruner - iCloudDriveDesktopSync - gets the iCloud Drive Desktop and Document Sync Status for the console user


#must be run as root
function iCloudDriveDesktopSync()(
	consoleUser=$(stat -f %Su /dev/console)

	#if root (loginwindow) grab the last console user
	if [ "${consoleUser}" = "root" ]; then
		consoleUser=$(/usr/bin/last -1 -t console | awk '{print $1}')

	#if this xattr exists then sync is turned on
	xattr_desktop=$(sudo -u $consoleUser /bin/sh -c 'xattr -p com.apple.icloud.desktop ~/Desktop 2>/dev/null')

	if [ -z "${xattr_desktop}" ]; then
		return 1
		return 0

#example function usage, if leverages the return values
if iCloudDriveDesktopSync; then
	echo "iCloud Drive Desktop and Documents Sync is ON"
	exit 0
	echo "iCloud Drive Desktop and Documents Sync is OFF"
	exit 1

The operation is pretty simple, it finds a console user or last user, then runs the xattr -p command as that user (anticipating this being run as root by Jamf) to see if the com.apple.icloud.desktop extended attribute exists on their ~/Desktop. In testing you’ll find if you toggle the “Desktop and Documents” checkbox in the iCloud Drive options, it will apply this to both of those folders almost immediately without fail. The function can be used in a Jamf Extension Attribute in the same way the iCloudDrive was above. Some assembly required. 💪

So, there you go! Another couple functions to read the stateful tea leaves of iCloud Drive settings. Very useful if you are about to disallow iCloud Drive and/or Desktop and Document sync via MDM but need to know who you are going to affect and let them know beforehand. Because I still stand by this sage advice: Don’t Be A Jerk. Thanks for reading you can find these script and more at my GitHub repo. Thanks for reading!

Determining “iCloud Private Relay” and “Limit IP tracking” status in macOS

If you are a Mac admin you might have noticed some Apple plists are more complex than others these days. As evidenced in my post Don’t Be a Jerk, getting the status of Do Not Disturb in Big Sur was a multi-step exercise. Check out this modified code excerpt from doNotDisturb‘s Big Sur handling


#Big Sur DnD status, returns "true" or [blank] (to be run as console user)
dndStatus="$(/usr/libexec/PlistBuddy -c "print :userPref:enabled" /dev/stdin 2>/dev/null <<< "$(plutil -extract dnd_prefs xml1 -o - /dev/stdin <<< "$(defaults export com.apple.ncprefs.plist -)" | xmllint --xpath "string(//data)" - | base64 --decode | plutil -convert xml1 - -o -)")"

#if we have ANYTHING it is ON (return 0) otherwise fail (return 1)
[ -n "${dndStatus}" ] && { echo ON; exit 0; } || { echo OFF; exit 1; }

Why is it so complex? Well because with the user preference domain of com.apple.ncprefs there is a base64 encoded plist embedded in the dnd_prefs key and requires some massaging to get it to a state where PlistBuddy can read and then extract the status from the :userPref:enabled key. See it’s just that easy! 😅

In macOS Monterey and higher Apple is now using JSON for Focus status which is great because it can can be parsed much easier (perhaps using my ljt or jpt tools) but sometimes not even that is needed! Sometimes file presence or awking is sufficient to get a reliable result too. See the evolution of doNotDisturb.sh for macOS for the myriad techniques one can use.

Now, here we are at Monterey on the verge of Ventura and not all areas of macOS have seen the JSON light and are still using nested base64-encoded Plist blobs to store the states of new features like iCloud Private Relay but luckily not the per interface setting of Limit IP Tracking.

iCloud Private Relay Status

Now why would you want to know these statuses? Perhaps you have a VPN product that doesn’t work when Private Relay is turned on or if Limit IP Tracking is turned on for a particular interface and you need to alert the user. Additionally maybe there’s an issue if your VPN browser handshake fails when Safari is used, if so see: Determining URL scheme handlers in macOS. Back to the task at hand though let’s turn some prefs inside out and look at iCloud Relay status with iCloudPrivateRelayStatus.sh

iCloud Private Relay Status Checker - (https://github.com/brunerd)
Copyright (c) 2022 Joel Bruner (https://github.com/brunerd)
Licensed under the MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.


#include this self-contained function in your script
function iCloudPrivateRelay(){

	#only for Moneterey and up, 11 and under need not apply
	[ "$(sw_vers -productVersion | cut -d. -f1)" -le 11 ] && return 1
	#parent pref domain
	#key that contains base64 encoded Plist within parent domain

	#child key within base64 embedded plist

	#get the top level data from the main domain
	parentData=$(launchctl asuser $(stat -f %u /dev/console) sudo -u $(stat -f %Su /dev/console) defaults export ${domain} -)

	#if domain does not exist, fail
	[ -z "${parentData}" ] && return 1

	#export the base64 encoded data within the key as PlistBuddy CF style for grepping (it resists JSON extraction)
	childData=$(/usr/libexec/PlistBuddy -c "print :" /dev/stdin 2>/dev/null <<< $(plutil -extract "${key}" xml1 -o - /dev/stdin <<< "${parentData}" | xmllint --xpath "string(//data)" - | base64 --decode | plutil -convert xml1 - -o -))

	#if child key does not exist, fail
	[ -z "${childData}" ] && return 1

	#match the status string, then get the value using awk (quicker than complex walk, this is sufficient), sometimes written in multiple places
	keyStatusCF=$(awk -F '= ' '/'${childKey}' =/{print $2}' <<< "${childData}" | uniq)
	#if we have differing results that don't uniq down to one line, throw an error
	[ $(wc -l <<< "${keyStatusCF}") -gt 1 ] && return 2
	#if true/1 it is on, 0/off (value is integer btw not boolean)
	[ "${keyStatusCF}" = "1" ] && return 0 || return 1

# MAIN #

#example - one line calling with && and ||
#iCloudPrivateRelay && echo "iCloud Private Relay is: ON" || echo "iCloud Private Relay is: OFF"

#example - multi-line if/else calling
if iCloudPrivateRelay; then
	echo "iCloud Private Relay is: ON"
	exit 0
	echo "iCloud Private Relay is: OFF"
	exit 1

Even though we work really hard to get the plist data extracted we don’t need to walk the entire XML document (you will find PlistBuddy balks at exporting JSON for a number of reasons). Instead we look for the presence of the key name PrivacyProxyServiceStatus and that is sufficient to reliably detect the state. I have 2 versions in my GitHub iCloudPrivateRelayStatus.sh and the minified one line version iCloudPrivateRelayStatus.min.sh

Limit IP Tracking Status

Limit IP Tracking, is a per-interface setting that is on by default, however has no effect unless Private Relay is enabled. For this we will use ljt my Little JSON Tool to retrieve the value from the massaged JSON conversion of com.apple.wifi.known-networks.plist if on WiFi or if on Ethernet .../SystemConfiguration/preferences.plist

Limit IP Tracking Status Checker - (https://github.com/brunerd)
Copyright (c) 2022 Joel Bruner (https://github.com/brunerd)
Licensed under the MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.


#include this self-contained function in your script
function limitIPTracking()(

	#only for Moneterey and up, 11 and under need not apply
	[ "$(sw_vers -productVersion | cut -d. -f1)" -le 11 ] && return 1

	#Little JSON Tool (ljt) - https://github.com/brunerd/ljt - MIT License
	function ljt () ( #v1.0.8 ljt [query] [file]
	{ set +x; } &> /dev/null; read -r -d '' JSCode <<-'EOT'
	try{var query=decodeURIComponent(escape(arguments[0]));var file=decodeURIComponent(escape(arguments[1]));if(query===".")query="";else if(query[0]==="."&&query[1]==="[")query="$"+query.slice(1);if(query[0]==="/"||query===""){if(/~[^0-1]/g.test(query+" "))throw new SyntaxError("JSON Pointer allows ~0 and ~1 only: "+query);query=query.split("/").slice(1).map(function(f){return"["+JSON.stringify(f.replace(/~1/g,"/").replace(/~0/g,"~"))+"]"}).join("")}else if(query[0]==="$"||query[0]==="."&&query[1]!=="."||query[0]==="["){if(/[^A-Za-z_$\d\.\[\]'"]/.test(query.split("").reverse().join("").replace(/(["'])(.*?)\1(?!\\)/g,"")))throw new Error("Invalid path: "+query);}else query=query.replace("\\.","\udead").split(".").map(function(f){return"["+JSON.stringify(f.replace("\udead","."))+"]"}).join("");if(query[0]==="$")query=query.slice(1);var data=JSON.parse(readFile(file));try{var result=eval("(data)"+query)}catch(e){}}catch(e){printErr(e);quit()}if(result!==undefined)result!==null&&result.constructor===String?print(result):print(JSON.stringify(result,null,2));else printErr("Path not found.")
	queryArg="${1}"; fileArg="${2}";jsc=$(find "/System/Library/Frameworks/JavaScriptCore.framework/Versions/Current/" -name 'jsc');[ -z "${jsc}" ] && jsc=$(which jsc);[ -f "${queryArg}" -a -z "${fileArg}" ] && fileArg="${queryArg}" && unset queryArg;if [ -f "${fileArg:=/dev/stdin}" ]; then { errOut=$( { { "${jsc}" -e "${JSCode}" -- "${queryArg}" "${fileArg}"; } 1>&3 ; } 2>&1); } 3>&1;else [ -t '0' ] && echo -e "ljt (v1.0.8) - Little JSON Tool (https://github.com/brunerd/ljt)\nUsage: ljt [query] [filepath]\n  [query] is optional and can be JSON Pointer, canonical JSONPath (with or without leading $), or plutil-style keypath\n  [filepath] is optional, input can also be via file redirection, piped input, here doc, or here strings" >/dev/stderr && exit 0; { errOut=$( { { "${jsc}" -e "${JSCode}" -- "${queryArg}" "/dev/stdin" <<< "$(cat)"; } 1>&3 ; } 2>&1); } 3>&1; fi;if [ -n "${errOut}" ]; then /bin/echo "$errOut" >&2; return 1; fi
	#test interface specified or fallback to default interface
	interfaceID=${1:-$(route get 2>/dev/null | awk '/interface: / {print $2}')}

	#WIFI: key: PrivacyProxyEnabled, file: /Library/Preferences/com.apple.wifi.known-networks.plist
	#if no error getting WiFi SSID, then we are WiFi
	if networksetup -getairportnetwork "${interfaceID}" &>/dev/null && wifiSSID=$(awk -F ': ' '{print $2}' <<< "$(networksetup -getairportnetwork "${interfaceID}" 2>/dev/null)"); then

		#key name inside plist

		#oddly this file is only read-able by root
		if [ ! -r /Library/Preferences/com.apple.wifi.known-networks.plist ]; then
			echo "Insufficient preferences to determine WiFi state, run as root" >/dev/stderr
			exit 1

		#get JSON version, so much easier to get around, convert date and data types to strings
		wifiKnownNetworks_JSON=$(defaults export /Library/Preferences/com.apple.wifi.known-networks.plist - | sed -e 's,<date>,<string>,g' -e 's,</date>,</string>,g' -e 's,<data>,<string>,g' -e 's,</data>,</string>,g' | plutil -convert json -o - -)
		#if there is NO entry, then it is active, it is opt-out designed
		PrivacyProxyEnabled=$(ljt "/wifi.network.ssid.${wifiSSID}/PrivacyProxyEnabled" 2>/dev/null <<< "${wifiKnownNetworks_JSON}")
		if [ "${PrivacyProxyEnabled}" = "false" ]; then
			return 1
	#ETHERNET: key: DisablePrivateRelay, file: /Library/Preferences/SystemConfiguration/preferences.plist, 
		#get JSON, easily converts with not data or date types within
		systemConfigPrefsJSON=$(plutil -convert json -o - /Library/Preferences/SystemConfiguration/preferences.plist)
		#get current set UUID
		currentSet=$(echo "${systemConfigPrefsJSON}" | ljt /CurrentSet 2>/dev/null)
		#get value for current default interface of current Location set
		DisablePrivateRelay=$(ljt "${currentSet}"/Network/Interface/${interfaceID}/DisablePrivateRelay  2>/dev/null <<< "${systemConfigPrefsJSON}")

		#if it is TRUE we are Disabled, then we are NOT ON, return fail code
		if [ "${DisablePrivateRelay}" = "1" ]; then
			return 1

# MAIN #

if [ ! -r /Library/Preferences/com.apple.wifi.known-networks.plist ]; then
	echo "Insufficient preferences to determine WiFi state, run as root" >/dev/stderr
	exit 1

#use default interface if nothing is specified for argument $1
interface=${1:-$(route get 2>/dev/null | awk '/interface: / {print $2}')}

#returns 0 if ON and 1 if OFF, you may supply an interface or it will fall back to the default
limitIPTracking "${interface}" && echo "Limit IP tracking is ON for: ${interface}" || echo "Limit IP tracking is OFF for: ${interface}"

You can specify an interface or it will use the default interface (there’s a good one-liner to use route get to figure that out). plutil is not initially used to convert com.apple.wifi.known-networks.plist to JSON as it will fail due to XML/plist data types like <date> and <data> that do not have JSON equivalents. First we use sed to change them to <string> types then plutil can convert to JSON. After that it’s a cake walk for ljt to get the value and report back. If the interface is not WiFi then /Library/Preferences/SystemConfiguration/preferences.plist has none of those conversion issues. Check out limitIPTrackingStatus.sh and the minified limitIPTrackingStatus.min.sh at my GitHub.

Thanks for reading! P.S. I won’t be at JNUC 2022, better luck next year!

Improvisational jpt: Processing Apple Developer JSON transcript files

If you’ve downloaded the Developer app for the Mac there’s a trove of JSON transcripts cached in your home folder at ~/Library/Group Containers/group.developer.apple.wwdc/Library/Caches/Transcripts

Being curious I took a look at them using my JSON Power Tool jpt. In it’s default mode it will “pretty print” JSON or in Javascript parlance “stringify” them with a two space indent per nesting level. Inside it can be seen the transcripts are arrays of arrays inisde a uniquely named object. The 1st entry of the array is the time in seconds and the 2nd entry is the string we want.

Arrays of arrays

One of jpt’s cool features is that it supports the venerable yet nascent JSONPath query syntax. Using JSONPath we can use the recursive operator .. to go straight to the transcript object without needing determine the unique name of the parent object, then we want all the array within there [*]and inside those array we want the second entry of the 0 based array [1]. The query looks like this $..transcript[*][1]

Just text please

The query is single quoted so the shell doesn’t interpret the $ as the beginning of a variable name. The -T option for jpt it outputs text without quotes. The default output mode for jpt is JSON (double quoted strings). I added all sorts of other niceties to the script, as you’ll see below. The results are output to your ~/Desktop in a folder called Developer Transcripts

Developer Transcript Extractor Copyright (c) 2022 Joel Bruner. Licensed under the MIT License. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


destinationFolderName="Developer Transcripts"

#you'll need to download the Developer app and launch it: https://apps.apple.com/us/app/apple-developer/id640199958
transcriptPaths=$(find "${HOME}/Library/Group Containers/group.developer.apple.wwdc/Library/Caches/Transcripts/ByID" -name '*json')
contentsJSON="${HOME}/Library/Group Containers/group.developer.apple.wwdc/Library/Caches/contents.json"

# MAIN #

#either jpt should be installed or the function jpt.min can be pasted in here
if ! which jpt &>/dev/null; then
	echo "Please install jpt or embed jpt.min in this script: https://github.com/brunerd/jpt"
	exit 1

#ensure the destination folder exists
[ ! -d "${destinationPath}" ] && mkdir "${destinationPath}"

#ignore spaces in file paths

#loop through each transcript json file
for transcriptPath in ${transcriptPaths}; do
	#id is just the file name without the path and extension
	id=$(cut -d. -f1 <<< "${transcriptPath##*/}")
	#a couple of nice-to-haves
	title=$(jpt -T '$.contents[?(@.id == "'"${id}"'")].title' "${contentsJSON}")
	description=$(jpt -T '$.contents[?(@.id == "'"${id}"'")].description' "${contentsJSON}")
	#change \ (disallowed in Unix) to : (Disallowed in Finder byt allowed in Unix)
	url=$(jpt -T '$.contents[?(@.id == "'"${id}"'")].webPermalink' "${contentsJSON}")

	#"wwdc" always has the year in the id but not tech-talks or insights
	if ! grep -q -i wwdc <<< "$id"; then
		year="$(jpt '$.contents[?(@.id == "'"${id}"'")].originalPublishingDate' "${contentsJSON}" | date -j -r 1593018000 +"%Y")-"
		filename="${year}${id} - ${title}.txt"
		filename="${id} - ${title}.txt"

	#put the ID and Title, the URL and Description at the top of the transcript
	echo "${id} - ${title}" > "${destinationPath}"/"${filename}"
	echo -e "${url}\n" >> "${destinationPath}"/"${filename}"
	echo -e "Description:\n${description}\n\nTranscript:" >> "${destinationPath}"/"${filename}"
	#append the transcript extract
	jpt -T '$..transcript[*][1]' "${transcriptPath}" >> "${destinationPath}"/"${filename}"

	#just echo out our progress
	echo "${destinationPath}"/"${filename}"

The final result is sortable folder of text files that you can easily QuickLook through.

Some serviceably readable contents!

So there you go! Some surprise JSON transcript files from the Apple Developer app, made me wonder how someone would turn them into human readable files. It turned out it was a fun and practical use of jpt and it’s support for JSONPath. You can download an installer package from the Releases page to try it out.

macOS Compatibility Fun!

Compatibility Questions

If you work with Macs and Jamf then you know every year there’s a new per OS Extension Attribute (EA) or Smart Group (SG) recipe to determine if macOS will run on your fleets hardware. However I asked myself: What if a single Extension Attribute script could fill the need, requiring only a periodic updating of Model IDs and the addition of new macOSes?

Then I also asked: Could this same script be re-purposed to output both text and CSV, not just for the script’s running host but for a list of Model IDs? And the answer was a resounding yes on all fronts!

EA Answers

So, my fellow Jamf admin I present to you macOSCompatibility.sh in its simplest form you just run the script and it will provide ultra-sparse EA output like: <result>10.14 10.15 11</result> this could then be used as a Smart Group criteria. Something like “macOS Catalina Compatible” would then match all Macs using LIKE 10.15 or “Big Sur Incompatible” would use NOT LIKE 11, of course care would be taken if you were also testing for 10.11 compatibility, however the versionsToCheck variable in the script can limit the default range to something useful and speeds things up the less version there are. I hope this helps Jamf admins who have vast unwieldy fleets where hardware can vary wildly across regions or departments,

CSV Answers

Now if you provide a couple arguments like so: ./macOSCompatibility.sh -c -v ALL ALL > ~/Desktop/macOSCompatibilityMatrix.csv you will get a pretty spiffy CSV that let’s you visualize which Mac models over the years have enjoyed the most and least macOS compatibility. This is my favorite mode, you can use it to assess the OS coverage of past Macs.

See macOSCompatibilityMatrix.csv for an example of the output. If you bring that CSV into Numbers or Excel you can surely liven it up with some Conditional Formatting! This is the barest of examples:

Can you spot the worst and best values?

Text Answers

If you don’t use the -c flag then it’ll just output in plain or text, like so: ./macOSCompatibility.sh -v ALL ALL

iMacPro1,1: 10.13 10.14 10.15 11
MacBook1,1: 10.4 10.5 10.6
MacBook2,1: 10.4 10.5 10.6 10.7
MacBook3,1: 10.5 10.6 10.7
MacBook4,1: 10.5 10.6 10.7
MacBook5,1: 10.5 10.6 10.7 10.8 10.9 10.10 10.11
MacBook6,1: 10.6 10.7 10.8 10.9 10.10 10.11 10.12 10.13
MacBook7,1: 10.6 10.7 10.8 10.9 10.10 10.11 10.12 10.13
MacBook8,1: 10.10 10.11 10.12 10.13 10.14 10.15 11
MacBook9,1: 10.11 10.12 10.13 10.14 10.15 11
MacBook10,1: 10.12 10.13 10.14 10.15 11
MacBookAir1,1: 10.5 10.6 10.7
MacBookAir2,1: 10.5 10.6 10.7 10.8 10.9 10.10 10.11

Wrapping Up

Now, it’s not totally perfect since some models shared Model IDs (2012 Retina and Non-Retina MacBook Pros for example) but for the most part the Intel Mac Model IDs were sane compared to the PPC hardware Model IDs: abrupt jumps, overlaps, and re-use across model familes. Blech! I’m glad Apple “got religion” for Model IDs (for the most part) when Intel CPUs came along. I did attempt to go back to 10.1-10.3 with PPC hardware but it was such a mess it wasn’t worth it. However testing Intel, Apple Silicon and VMs against macOS 10.4 – 11+ seems to have some real use and perhaps you think so too? Thanks for reading!

jpt is unphased by macOS’s malformed JSON

A funny thing happened on the way to writing a new blog entry. Instead of finding an example JSON file that ships with macOS, I stumbled upon a slow creep of malformed JSON files that started in OS X El Capitan and continues into Big Sur. While I can’t do much about deviant JSON files in macOS, I can control how jpt handles them. So like any good Teachable Moment™ and life lesson, I’ve turned lemons into lemonade and out of challenge comes innovation: jpt v0.9.6 is now nonplussed by these uncouth JSON files, with their comments and trailing commas. Like water off ducks back, jpt keeps on keepin’ on, unphased.

Finding the Misfits

To find these misfits I wrote made a script jsonParsingReportCSV-macOS. This will attempt to parse the JSON it finds using the json_pp binary that ships from Apple, the results are output as CSV and you can then easily sort it to spot troublemakers.

I’ve uploaded a couple annotated version of those searches along with the kind of deviation they exhibit: Malformed-macOS-10.15.csv and Malformed-macOS-11.1.csv

What’s their damage, man?

The two most prevalent kinds of JSON malformation I found in macOS were: trailing commas and comments. Trailing commas meaning the last item in an array or object should not have a comma after it. While Javascript has always allowed trailing commas in array literals and later in object literals (ES5), then in function parameters (ES2017), JSON has never allowed trailing commas nor comments. Neither are in the JSON spec (RFC8259).

The biggest group of offenders were the USD generated JSONs within USDKit (a private framework) and Model I/O. USD is interesting because of it’s lineage and purpose: It’s Pixar’s open source software for combining multiple 3D elements within a common scene. Or as Pixar’s page put it: “Universal Scene Description (USD) is the first publicly available software that addresses the need to robustly and scalably interchange and augment arbitrary 3D scenes that may be composed from many elemental assets.” Keep an eye on this framework 🕶

JSON Lines (Don’t Do It)

An outlier in the mix is defaultConfig.json within CoreAnalytics.framework, it is what some call “JSON Lines” or “concatenated JSON”, where each line is its own a JSON value or object. While doing some research on JSON Lines I came upon an intriguing comment in this post by rurban:

“The delimiters don’t get in the way, they protect you from MITM and other attacks. It’s a security feature, not a bug.”

Reini Urban on JSON vs. “concatenated JSON”

Taking a look at Reini’s GitHub and the level of his work, I’m going to give his statement some credence. To that end, I really didn’t want to add another output mode nor change query behavior to accommodate JSON Lines. My current compromise is to simply to attempt conversion of a JSON Lines file into an array: tack on commas and ensconce in square brackets. It Just Works™ if every line is truly a self contained JSON object. If you want to be made aware of this, use the -c option so jpt it will “complain” on stderr and exit with code 1.

The Rabbit Hole of CoreAnalytics

Within the CoreAnalytics private framework is defaultConfig.json, it is a collection of 4019 objects currently that describe the events and actions Apple is interested in collecting usage information on. They want to know if you’re actually using all those new features in CarPlay, Reminders, Photos, eGPU App usage, etc… If you want to see all the key names try this: jpt $..addTransform.name /System/Library/PrivateFrameworks/CoreAnalytics.framework/Versions/A/Resources/defaultConfig.json 

Do you actually use this stuff? Apple wants to know if it’s worth their while…

A Comment on JSON Comments

Comments in JSON has inspired debate far and wide, this post by getify sums it up very nicely:

A JSON encoder MUST NOT output comments. A JSON decoder MAY accept and ignore comments.

Douglas Crockford in some old Yahoo group that even Wayback can’t find anymore

So that’s just what jpt does now, it will never encode comments but it will strip out any and all comments it recognizes automatically, no switches needed. Which style comments? All of them (or most of them):

# commentlineshell, Perl, PHP, Python, R, Ruby, MySQL
// commentlineC(99), Javascript, Java, Swift
/* comment */blockC, Java, Javascript, Swift
; commentlineassembly language (ASM)
-- commentlineAda, Applescript, Haskell, Lua, SQL
<!-- comment -->blockXML
*** comment ***linemacOS’s ionodecache.json

If you need a test file you can use my worstJSONEver.json. A most horrible JSON file if there ever was one! jpt soldiers on, use -c to see what it does to clean the input (-i0 for no level indents). It shows the initial parse error and each attempt at recovery.

Fixing the Misfits

There is no fixing, there is only acceptance. Since most of these files are in SIP protected areas of macOS so it will take an act of God, a well filed bug report, for Apple to correct these in a future update. But they usually only act if there’s a demonstrable bug, in this case it’s more of an observation about the condition of the JSON files than a clear issue. If someone is really bothered, they could disable SIP and go in and fix them, however future macOS updates will likely revert these changes. Since I haven’t tried this, who knows, macOS may get ornery and trip OS integrity protections, rendering your system un-bootable, YMMV 🤷‍♂️. The real question is if they impact how macOS works and I’m guessing they don’t. The apps and frameworks which use these off-spec JSON file must be doing something to correct the issues before parsing. Rest assured though, if you attempt to parse one of these files with jpt there will be no issues, it Just Works™. However if you would like to know it did something, use -c to “complain” about comments, commas, JSON Lines, and hard wrapping. It will send those notices to stderr and exit with a status of 1.

jpt: malformed JSON? Bring it on.

So after all this I don’t have a good example page for jpt’s multitude of features and usage (yet) but what I do have is a JSON Power Tool with more “dummy-proof” features than it did before. I’ll take that as a win! Check out the latest release of jpt at my Github.

jpt (JSON Power Tool) is a non-compiled tool that can manipulate and query JSON in a variety of ways, it runs on macOS (10.4 – 11.0) PPC/Intel/Apple Silicon; Linux with jsc; Windows with Linux subsystem for Windows and jsc. It can be used standalone or embedded within your shell scripts. I hope you find it useful. Thanks for reading!

Jamf & FileVault 2: Tips & Tricks (and more)

Raiders of the Lost Feature Requests

So there’s this old feature request at Jamf Nation (stop me if you’ve heard this one…) it’s almost 6 years old: Add ability to report on FV2 Recovery Keys (and/or access them via API) In fact, maybe you came here from there, watch out don’t loop! Continue!

The pain point is this: Keys are sent back to Jamf Pro (JSS) but then can only be gotten at manually/interactively through the web interface, not via API nor another method. For cases of mass migration to another JSS it sure would be nice to move those keys over rather than decrypt/re-encrypt. Well, I’ve got a few insights regarding this that I’d like to share that may help. ‘Cuz hey it’s 2020 and we’ve learned that hoarding is just silly.

Firstly, it should be pointed out that neither ye olde “Recovery Key Redirection” payload nor it’s replacement “Recovery Key Escrow” are needed to get keys to the JSS. There is another method and it’s what is used by the built-in “Filevault Encryption” policy payload to get the keys back to your JSS. Jamf references this method in this old script at their GitHub. I revamped the core bits a couple years ago in a (nearly 7 year old) feature request: Manually Edit FileVault 2 Recovery Key

Telling the JSS Your Secrets

The takeaway from that is to realize we have a way to explicitly send keys to the JSS by placing 2 XML files in the /Library/Application Support/JAMF/run folder: file_vault_2_id.xml and file_vault_2_recovery_key.xml. Also note, Jamf has updated the process for the better in the last two years: a jamf recon (or two) is no longer required to send the key and validate it, instead JamfDaemon will send it immediately when both the files are detected. Which is nice, but it’s the subsequent recon validations where we have an opportunity to get grabby.

Cold Lamping, Hard Linking

So here’s the fun part: When recon occurs there’s lots of file traffic in /Library/Application Support/JAMF/tmp all sorts of transient scripts hit this folder. What we can do is make hard links to these files as they come in so when the link is removed in tmp another exists elsewhere and the file remains (just in our new location). EAGrabber.sh does exactly that (and a little bit more)

EAGrabber.sh can be easily modified to narrow it’s focus to the FileVault 2 key only, deleting the rest. What you do with the key is up to you: Send it somewhere else for safe keeping or keep it on device temporarily for a migration to another Jamf console. A script on the new JSS could then put that key on-disk into file_vault_2_recovery_key.xml file which will then import and validate, no decrypt/recrypt necessary. Hope this helps.

Cuidado ¡ Achtung ! Alert

Jamf admins take note: Do you have hard coded passwords in your extension attributes or scripts? If so, then all your scripts are belong to us. Now, go read Obfuscation vs. Encryption from Richard Purves. Read it? OK, now consider what happens if you were to add a routine to capture the output of ps aww along with a hard-linking loop like in EAGraber.sh. If you are passing API credentials from policies via parameter, then ps can capture those parameters and even if you try and obscure them, if we’ve captured the script we can de-obfuscate them. This is a good reason to be really careful with what your API accounts can do. If you have an API account with Computer record Read rights that gets passed into a script via policy and you use LAPS, then captured API credentials could be used to harvest LAPS passwords via API. Keep this in mind and we’ll see if any meaningful changes will occur in recon and/or the script running process in the future (if you open a ticket you can reference PI-006270 regarding API credentials in the process list). In the meantime make API actions as short lived as possible and cross your fingers that only you, good and noble #MacAdmins read this blog. 🤞

secret origins: the jpt

On building a JSON tool for macOS without using Python, perl, or Ruby.

In my work as a Macintosh engineer and administrator I’ve noticed macOS has lacked a bundled tool for working with JSON at the command line. Where XML has its xpath, if your shell script needs some JSON chops, it’ll require an external binary like jq or something else scripted in Python, Ruby or perl using their JSON modules. The problem is, those runtimes have been slated for removal from a future macOS. So I took that as challenge to devise a method to query and modify JSON data within shell scripts, that didn’t use one of those deprecated scripting runtimes and didn’t require an external binary dependency either. Could I achieve robust and native JSON parsing on a Mac by simply “living off the land”?

Why not just re-install the runtimes when Apple deprecates them?

“Why limit yourself like this? Just re-install the runtimes and move on”, you may ask. Well, I’d like to think that limitations can inspire creativity but we should also consider there may be some other reasons why Apple is discontinuing the inclusion of those runtimes. Some may say, “Apple Silicon + macOS 11.0 is the perfect time for them to clean house”, to which I’d have to agree that’s a very good reason and likely a factor. Others could say they are looking to tighten the screws to keep out unsigned code: maybe, they do like to glue things shut! But really, I think it’s more akin to the web-plugins of yore like Java and Flash. Apple does not want to be the conduit for deploying 3rd party party runtimes which increase the attack surface of macOS. This seems like the most reasonable of the explanations. So, if you accept that Apple is attempting to reduce attack surface, why increase it by re-installing Python, Ruby, or perl, just so a transient script (like a Jamf Extension Attribute) can parse a JSON file? My answer to that, is you don’t. You play the hand you’re dealt. Game on!

Looking for truffles (in a very small back yard)

Despite having another project (shui) that can output and invoke Applescript from within a shell scripts for generating user interfaces, I definitely knew that Applescript was not the way to go. Apple however, added to the languages Open Scripting Architecture (OSA) supports back in 2014 with OS X Yosemite (10.10), they added Javascript along with a bridge to the Cocoa Objective-C classes and they called it JXA: Javascript for Automation. This seemed like a promising place, so I started playing with osascript and figured out how to load files and read /dev/stdin using JXA, and while looking for an answer for garbled input from stdin I came upon a Japanese blog that mentioned jsc the JavaScriptCore binary which resides in the /System/Library/Frameworks/ JavaScriptCore.framework. Arigato! Pay dirt! 🤑 jsc does exactly what we need it to do: It can interpret Javascript passed as an argument, can access the filesystem and read from /dev/stdin, and best of all is in non-Private System level Framework that exists all the way back to OS X 10.4! Just the kind of foundation on which to build the tool.

Homesteading jsc

The existence of jsc goes back all the way to OS X Tiger and it’s functionality has evolved over the years. In order to have a consistent experience in jpt from macOS 10.4 – 11.x+ a few polyfills had to be employed for missing functions, along with a few other workarounds regarding file loading, printing and exit codes (or lack thereof). Once those were addressed the jsc proved to be a highly optimized Javascript environment that’s blazingly fast. It spans 13 macOS releases and is even present in many Linux distros out-of-the-box (Ubuntu and CentOS) and can even be run on Windows when the Linux Subsytem is installed.

With the host environment sorted, I began working with the original JSONPath code by Stefan Goessner as the query language. I didn’t know about JSON Pointer yet so this strange beast was all I knew! It worked out really. I went full throttle into developing the “swiss army knife” of JSON tools. I really leaned into the Second System Effect, as described in the Mythical Man-month, it’s when you put every doodad, gizmo and doo-hicky in your 2nd product (my first simple JSON pretty-printer built in JS on jsc). Eventually though, after I reached a feature plateau, I came back around to the address the quirks of the original JSONPath code. I ended up rewriting signifigant chunks of JSONPath and released it as it’s own project: brunerd JSONPath. But I digress, let’s get back to the JSON Power Tool.

jpt: powers and abilities

At it’s most basic, jpt will format or “pretty print” any JSON given. jpt can also handle data retrieval from JSON document using either JSONPath or JSON Pointer syntax. JSONPath, while not a standard, is a highly expressive query language akin to XPath for XML, with poweful features like recursive search, filter expressions, slices, and unions. JSON Pointer on the other hand is narrower in focus, succint and easily expressed, and standardized but it does not offer any of the interogative features that I feel make JSONPath so intriguing. Finally, jpt can also modify values in a JSON document using standardized JSON Patch operations like: add, remove, replace, copy, move, and test, as well as the also standardized JSON Merge Patch operation. Altogether, the jpt can format, retrieve and alter JSON documents using only a bit of outer shell script plus a lot more Javascript on any macOS since 10.4! 😅

Where can I get the jpt?

Stop by the project’s GitHub page at: https://github.com/brunerd/jpt

There you will find the full source and also a minified version of the jpt for inclusion within your shell scripts. Since it’s never compiled you can always peer inside and learn from it, customize it, modify it, or just tinker around with it (usually the best teacher).

Future Plans

There will undoubtedly be continued work on the jpt. Surely there are less than optimal routines, un-idiomatic idioms, edge cases not found, and features yet to be realized. But as far as the core functionality goes though, it’s fairly feature complete in my opinion. Considering that one of my top 10 StrengthsFinder qualities is “Maximizer”, the odds are pretty good, I’ll keep honing the jpt‘s utility, size (smaller), and sharing more articles with examples on the kinds of queries and data alteration operations the jpt can so perform. Stay tuned!