jpt 1.0 text encoding fun

Besides JSON, jpt (the JSON power tool) can also output strings and numbers in a variety of encodings that the sysadmin or programmer might find useful. Let’s look at the encoding options from the output of jpt -h

% jpt -h
...
-T textual output of all data (omits property names and indices)
  Options:
	-e Print escaped characters literally: \b \f \n \r \t \v and \\ (escapes formats only)
	-i "<value>" indent spaces (0-10) or character string for each level of indent
	-n Print null values as the string 'null' (pre-encoding)

	-E "<value>" encoding options for -T output:

	  Encodes string characters below 0x20 and above 0x7E with pass-through for all else:
		x 	"\x" prefixed hexadecimal UTF-8 strings
		O 	"\nnn" style octal for UTF-8 strings
		0 	"\0nnn" style octal for UTF-8 strings
		u 	"\u" prefixed Unicode for UTF-16 strings
		U 	"\U "prefixed Unicode Code Point strings
		E 	"\u{...}" prefixed ES2016 Unicode Code Point strings
		W 	"%nn" Web encoded UTF-8 string using encodeURI (respects scheme and domain of URL)
		w 	"%nn" Web encoded UTF-8 string using encodeURIComponent (encodes all components URL)

		  -A encodes ALL characters
	
	  Encodes both strings and numbers with pass-through for all else:
		h 	"0x" prefixed lowercase hexadecimal, UTF-8 strings
		H 	"0x" prefixed uppercase hexadecimal, UTF-8 strings
		o 	"0o" prefixed octal, UTF-8 strings
		6 	"0b" prefixed binary, 16 bit _ spaced numbers and UTF-16 strings
		B 	"0b" prefixed binary, 8 bit _ spaced numbers and UTF-16 strings
		b 	"0b" prefixed binary, 8 bit _ spaced numbers and UTF-8 strings

		  -U whitespace is left untouched (not encoded)

Strings

While the above conversion modes will do both number and string types, these options will work only on strings (numbers and booleans pass-through). If you work with with shell scripts these techniques may be useful.

If you store shell scripts in a database that’s not using utf8mb4 table and column encodings then you won’t be able to include snazzy emojis to catch your user’s attention! In fact this WordPress install was so old (almost 15 years!) the default encoding was still latin1_swedish_ci, which an odd but surprisingly common default for many old blogs. Also if you store your scripts in Jamf (still in v10.35 as of this writing) it uses latin1 encoding and your 4 byte characters will get mangled. Below you can see in Jamf things look good while editing, fails once saved, and the eventual workaround is to use an coding like \x escaped hex (octal is an alternate)

Let’s use the red “octagonal sign” emoji, which is a stop sign to most everyone around the world, with the exception of Japan and Libya (thanks Google image search). Let’s look at some of the way πŸ›‘ can be encoded in a shell script

#reliable \x hex notation for bash and zsh
% jpt -STEx <<< "Alert πŸ›‘"
Alert \xf0\x9f\x9b\x91

#above string can be  in both bash and zsh
% echo $'Alert \xf0\x9f\x9b\x91'
Alert πŸ›‘

#also reliable, \nnn octal notation
% jpt -STEO <<< "Alert πŸ›‘"
Alert \360\237\233\221

#works in both bash and zsh
% echo $'Alert \360\237\233\221'
Alert πŸ›‘

#\0nnn octal notation
% jpt -STE0 <<< "Alert πŸ›‘"
Alert \0360\0237\0233\0221

#use with shell builtin echo -e and ALWAYS in double quotes
#zsh does NOT require -e but bash DOES, safest to use -e
% echo -e "Alert \0360\0237\0233\0221"
Alert πŸ›‘

#-EU code point for zsh only
% jpt -STEU <<< "Alert πŸ›‘"
Alert \U0001f6d1

#use in C-style quotes in zsh
% echo $'Alert \U0001f6d1'
Alert πŸ›‘

The -w/-W flags can encode characters for use in URLs

#web/percent encoded output in the case of non-URLs -W and -w are the same
% jpt -STEW <<< πŸ›‘  
%F0%9F%9B%91

#-W URL example (encodeURI)
jpt -STEW <<< http://site.local/page.php?wow=πŸ›‘
http://site.local/page.php?wow=%F0%9F%9B%91

#-w will encode everything (encodeURIComponent)
% jpt -STEw <<< http://site.local/page.php?wow=πŸ›‘
http%3A%2F%2Fsite.local%2Fpage.php%3Fwow%3D%F0%9F%9B%91

And a couple other oddballs…

#text output -T (no quotes), -Eu for \u encoding
#not so useful for the shell scripter
#zsh CANNOT handle multi-byte \u character pairs
% jpt -S -T -Eu <<< "Alert πŸ›‘"
Alert \ud83d\uded1

#-EE for an Javascript ES2016 style code point
% jpt -STEE <<< "Alert πŸ›‘"
Alert \u{1f6d1}

You can also \u encode all characters above 0x7E in JSON with the -u flag

#JSON output (not using -T)
% jpt <<< '"Alert πŸ›‘"'
"Alert πŸ›‘"

#use -S to treat input as a string without requiring hard " quotes enclosing
% jpt -S <<< 'Alert πŸ›‘'
"Alert πŸ›‘"

#use -u for JSON output to encode any character above 0x7E
% jpt -Su <<< 'Alert πŸ›‘'
"Alert \ud83d\uded1"

#this will apply to all strings, key names and values
% jpt -u <<< '{"πŸ›‘":"stop", "message":"Alert πŸ›‘"}' 
{
  "\ud83d\uded1": "stop",
  "message": "Alert \ud83d\uded1"
}

Whew! I think I covered them all. If there are newlines, tabs and other invisibles you can choose to output them or leave them encoded when you are outputting to text with -T

#JSON in, JSON out
jpt <<< '"Hello\n\tWorld"'
"Hello\n\tWorld"

#ANSI-C string in, -S to treat as string despite lack of " with JSON out
% jpt -S <<< $'Hello\n\tWorld' 
"Hello\n\tWorld"

#JSON in, text out: -T alone prints whitespace characters
% jpt -T <<< '"Hello\n\tWorld"'
Hello
	World

#use the -e option with -T to encode whitespace
% jpt -Te <<< '"Hello\n\tWorld"'
Hello\n\tWorld

Numbers

Let’s start simply with some numbers. First up is hex notation in the style of 0xhh and 0XHH. This encoding has been around since ES1, use the -Eh and -EH respectively to do so. All alternate output (i.e. not JSON) needs the -T option. In shell you can combine multiple options/flags together except only the last flag can have an argument, like -E does below.

#-EH uppercase hex
% jpt -TEH <<< [255,256,4095,4096] 
0xFFa
0x100
0xFFF
0x1000

#-Eh lowecase hex
% jpt -TEh <<< [255,256,4095,4096]
0xff
0x100
0xfff
0x1000

Next up are ye olde octals. Use the -Eo option to convert numbers to ye olde octals except using the more modern 0o prefix introduced in ES6

-Eo ES6 octals
% jpt -TEo <<< [255,256,4095,4096]    
0o377
0o400
0o7777
0o10000

Binary notation debuted in the ES6 spec, it used a 0b prefix and allows for _ underscore separators

#-E6 16 bit wide binary
% jpt -TE6 <<< [255,256,4095,4096]
0b0000000011111111
0b0000000100000000
0b0000111111111111
0b0001000000000000

#-EB 16 bit minimum width with _ separator per 8
% jpt -TEB <<< [255,256,4095,4096]
0b00000000_11111111
0b00000001_00000000
0b00001111_11111111
0b00010000_00000000

#-Eb 8 bit minimum width with _ separator per 8
% jpt -TEb <<< [15,16,255,256,4095,4096]
0b00001111
0b00010000
0b11111111
0b00000001_00000000
0b00001111_11111111
0b00010000_00000000

If you need to encode strings or numbers for use in scripting or programming, then jpt might be a handy utility for you and your Mac and if your *nix has jsc then it should work also. Check the jpt Releases page for Mac installer package download.

jpt 1.0, the JSON power tool in 2022

I’m happy to announce that jpt, the JSON power tool, has been updated to 1.0 and is available on my GitHub Releases page! It’s been over a year since the last release and I’ve been of working on, learning, and pondering what a really useful JSON tool could be and then working like heck to make it a reality.

jpt is a command line utility for formatting, querying, and modifying JSON. It can run on any version of macOS and most any Unix/Linux with jsc (JavaScriptCore) installed, otherwise it’s dependency free! For systems administrators and MacAdmins who need first class JSON tooling in their own shell scripts it can be embedded as a shell function in either bash or zsh. jpt was built to stand the test of time, needing only jsc and a shell, it will work on the newest macOS all the way back to OS X 10.4 Tiger!

There’s a year’s worth of bug fixes, improvements, and new features, let’s take a look at some of them.

Notable Features and Improvement

β€’Β jpt can now input and output multi-JSON documents like JSON Text Sequences (RFC 7464) plus other non-standard formats like NDJSON and JSON Lines and concatenated JSON. See: jpt 1.0 can deal with multiple JSON texts

β€’ Truncated JSON can be de detected in concatenated/lines JSON and repaired by closing up open strings, arrays and objects. See: Helping truncated JSON data with jpt 1.0

β€’ jpt can now fully rehabilitate JSON5 back to standard JSON. As I learned last year, there’s more mutant JSON than I realized! See: jpt 1.0 and JSON5 rehab

β€’ jpt can query JSON using both JSON Pointer, IETF RFC 6901 and JSONPath, a nascent draft RFC with powerful recursive search and filtering features.

β€’ jpt can manipulate JSON using JSONPatch, JSON Merge Patch, and some new merge modes not found anywhere else. See: merging JSON objects with jpt 1.0

β€’ shell scripters and programmers may find it useful that jpt has extensive string and number encoding capabilities. From classic encodings like hex, octal, and binary to more recent ones like URL/”percent” encoding and Unicode code points. See: jpt 1.0 text encoding fun

β€’ Need help finding visualizing JSON in new ways? jpt can output the structure of JSON in ways not seen, like JSONPath Object Literals which are simply the JSONPath, an equal sign, and the JSON value (e.g. $.ok="got it"). This simple and powerful declarative syntax can help find the exact path to a particular value and can also be used to quickly prototype a JSON object. See: jpt: see JSON differently

Try it out

There’s a lot to explore in jpt 1.0, download it from my GitHub with a macOS installer package in the Releases page. Check out my other jpt blog entries here at brunerd. As the IETF JSONPath standard takes shape I’ll be updating jpt to accomodate that standard. Until then, I hope this tool helps you in your JSON work, play, or research!


Decoding macOS automatic login details

In my previous post, Automating automatic login, we looked at how to create the /etc/kcpassword file used for automatic login by using only shell script and built-in command line tools. Why shell only? In preparation for the great scripting runtime deprecation yet to come, I say! Now it’s time to do the reverse for auto login. Let’s get those details back out! Who would need to do such a thing? Imagine a scenario where you the hapless Mac admin have inherited a bunch of Zoom Room Mac minis with auto-login enabled yet no one has documented the passwords used for them! If they are enrolled in Jamf there’s no need to guess what annoying l33t sp3@k password was used, let’s leverage our XOR’ing skills and knowledge of how kcpassword works to send those details back to Jamf.

To get the password back out of /etc/kcpassword we XOR the password again with the same cipher used to obfuscate it originally however but instead of padding it in multiples of 12, we will stop when a character is the same as the current cipher character. FYI when you XOR a value with itself the result is 00 but that’s an unnecessary operation, we can just compare the characters. VoilΓ‘, that’s it.

Here’s the gist of the kcpasswordDecode routine:

Now for something a bit more useful to those with Jamf or other management tools: getAutoLogin. It reports the auto login username, if set, and the decodes the /etc/kcpassword file, if present. Note that until macOS 12 Monterey /etc/kcpassword was not removed when Automatic Login was turned off in System Preferences! Here’s what getAutoLogin looks in the Jamf policy logs:

Plaintext passwords in your logs are probably not the best, but hey, how else you gonna figure out your dang Zoom Room passwords? After retrieving the credentials and storing somewhere more secure, like a password manager, make sure to Flush the policy logs! Thanks for reading, I hope this comes in handy or at the very least was informative and mildly entertaining. πŸ€“

Gist: kcpasswordDecode
Github: getAutoLogin

Automating automatic login for macOS

I recently had some Zoom Room Macs that needed some automation love and I thought I’d share how you can enable Automatic Login via script, perhaps you have several dozen and use Jamf or some other Mac management tool? Interested? Read on! Currently what’s out there are either standalone encoders only or part of larger packaging workflow to create a new user. The standalone encoders lacked some niceties like logging or account password verification and the package method added the required dependency of packaging if any changes were required. Above all, every script required Python, perl or Ruby, which are all on Apple’s hit list for removal in an upcoming OS release. For now macOS Monterey still has all of these runtimes but there will come a day when macOS won’t and will you really want to add 3rd party scripting runtimes and weaken your security by increasing attack surface when you can weaken your security using just shell? 😜 So for some fun, I re-implemented the /etc/kcpassword encoder in shell so it requires only awk, sed, and xxd, all out of the box installs. I also added some bells and whistles too.

Some of the features are:

  • If the username is empty, it will fully disable Automatic Login. Since turning it off via the System Preferences GUI does not remove the /etc/kcpassword file if it has been enabled (!)
  • Ensures the specified user exists
  • Verifies the password is valid for the specified user
  • Can handle blank passwords
  • Works on OS X 10.5+ including macOS 12 Monterey

For the Jamf admin the script is setAutomaticLogin.jamf.sh and for standalone usage get setAutomaticLogin.sh, both take a username and password, in that order and then enable Automatic Login if it all checks out. The difference with the Jamf script is that the first parameter is ${3} versus ${1} for the standalone version.

Also here’s a well commented Gist as a little show and tell for what the shell only version of the kcpassword encoder looks like. Enjoy!

Update: I meant to expound on how it didn’t seem that padding kcpassword to a length multiple of 12 was necessary, since it had been working fine on the versions I was testing with but then I tested on Catalina (thinking of this thread) and was proven wrong by Catalina. I then padded with 0x00 with HexFiend in a successful test but was reminded that bash can’t handle that character in strings, instead I padded with 0x01, which worked on some macOS versions but not others. Finally, I settled on doing on what macOS does somewhat, whereas it pads with the next cipher character then seemingly random data, I pad with the cipher characters for the length of the padding. This works for Catalina and other of macOS versions. πŸ˜…

Big Hat Tip to Pico over at MacAdmins Slack for pointing out this issue in pycreateuserpkg where it’s made clear that passwords with a lengths of 12 or multiples thereof, need another 12 characters of padding (or in some newer OSes at least one cipher character terminating the kcpassword data, thanks! πŸ‘

macOS Compatibility Fun!

Compatibility Questions

If you work with Macs and Jamf then you know every year there’s a new per OS Extension Attribute (EA) or Smart Group (SG) recipe to determine if macOS will run on your fleets hardware. However I asked myself: What if a single Extension Attribute script could fill the need, requiring only a periodic updating of Model IDs and the addition of new macOSes?

Then I also asked: Could this same script be re-purposed to output both text and CSV, not just for the script’s running host but for a list of Model IDs? And the answer was a resounding yes on all fronts!

EA Answers

So, my fellow Jamf admin I present to you macOSCompatibility.sh in its simplest form you just run the script and it will provide ultra-sparse EA output like: <result>10.14 10.15 11</result> this could then be used as a Smart Group criteria. Something like “macOS Catalina Compatible” would then match all Macs using LIKE 10.15 or “Big Sur Incompatible” would use NOT LIKE 11, of course care would be taken if you were also testing for 10.11 compatibility, however the versionsToCheck variable in the script can limit the default range to something useful and speeds things up the less version there are. I hope this helps Jamf admins who have vast unwieldy fleets where hardware can vary wildly across regions or departments,

CSV Answers

Now if you provide a couple arguments like so: ./macOSCompatibility.sh -c -v ALL ALL > ~/Desktop/macOSCompatibilityMatrix.csv you will get a pretty spiffy CSV that let’s you visualize which Mac models over the years have enjoyed the most and least macOS compatibility. This is my favorite mode, you can use it to assess the OS coverage of past Macs.

See macOSCompatibilityMatrix.csv for an example of the output. If you bring that CSV into Numbers or Excel you can surely liven it up with some Conditional Formatting! This is the barest of examples:

Can you spot the worst and best values?

Text Answers

If you don’t use the -c flag then it’ll just output in plain or text, like so: ./macOSCompatibility.sh -v ALL ALL

iMacPro1,1: 10.13 10.14 10.15 11
MacBook1,1: 10.4 10.5 10.6
MacBook2,1: 10.4 10.5 10.6 10.7
MacBook3,1: 10.5 10.6 10.7
MacBook4,1: 10.5 10.6 10.7
MacBook5,1: 10.5 10.6 10.7 10.8 10.9 10.10 10.11
MacBook6,1: 10.6 10.7 10.8 10.9 10.10 10.11 10.12 10.13
MacBook7,1: 10.6 10.7 10.8 10.9 10.10 10.11 10.12 10.13
MacBook8,1: 10.10 10.11 10.12 10.13 10.14 10.15 11
MacBook9,1: 10.11 10.12 10.13 10.14 10.15 11
MacBook10,1: 10.12 10.13 10.14 10.15 11
MacBookAir1,1: 10.5 10.6 10.7
MacBookAir2,1: 10.5 10.6 10.7 10.8 10.9 10.10 10.11

Wrapping Up

Now, it’s not totally perfect since some models shared Model IDs (2012 Retina and Non-Retina MacBook Pros for example) but for the most part the Intel Mac Model IDs were sane compared to the PPC hardware Model IDs: abrupt jumps, overlaps, and re-use across model familes. Blech! I’m glad Apple “got religion” for Model IDs (for the most part) when Intel CPUs came along. I did attempt to go back to 10.1-10.3 with PPC hardware but it was such a mess it wasn’t worth it. However testing Intel, Apple Silicon and VMs against macOS 10.4 – 11+ seems to have some real use and perhaps you think so too? Thanks for reading!

jpt + jamf uapi = backupJamf-scripts

Jamf UAPI: JSON Only

I am a creature of habit, no doubt, however sometimes you must get out of your comfort zone. The Jamf Universal API (UAPI) is one such case, it is JSON only and not XM. Those tried and true xpath snippets will no longer work with JSON, in fact what tool do you use with JSON? macOS really doesn’t have a good built-in JSON tool and if your scripts are client side do you really want to have jq as a dependency? Good thing I wrote a JSON parser you can embed in your scripts this summer! In fact, when I finished writing my JSON power tool jpt, I needed to find some practical examples to demonstrate its utility. Looking at the UAPI it’s clear some parts are still a work-in-progress, however the endpoint for scripts is actually really good. It gives you everything in one go almost like a database query. That should made backing up scripts a breeze!

backupJamf-scripts. boom.

If you’ve used the Classic API from Jamf, it is a 1:1 ratio of scripts to API calls: 2 scripts? 2 API calls via curl. 200 scripts? 200 API calls via curl. The new Universal API reduces that down to 1 call to get everything (plus one call to get the token), it’s super fast and I love it. Check out backupJamf-scripts.command in my newly minted jamfTools repo on GitHub for a working demostration of both the Jamf UAPI and jpt’s in-script JSON handling. I hope you like it!

jpt + jamf uapi = scripts downloading scripts

Jamf & FileVault 2: Tips & Tricks (and more)

Raiders of the Lost Feature Requests

So there’s this old feature request at Jamf Nation (stop me if you’ve heard this one…) it’s almost 6 years old: Add ability to report on FV2 Recovery Keys (and/or access them via API) In fact, maybe you came here from there, watch out don’t loop! Continue!

The pain point is this: Keys are sent back to Jamf Pro (JSS) but then can only be gotten at manually/interactively through the web interface, not via API nor another method. For cases of mass migration to another JSS it sure would be nice to move those keys over rather than decrypt/re-encrypt. Well, I’ve got a few insights regarding this that I’d like to share that may help. ‘Cuz hey it’s 2020 and we’ve learned that hoarding is just silly.

Firstly, it should be pointed out that neither ye olde “Recovery Key Redirection” payload nor it’s replacement “Recovery Key Escrow” are needed to get keys to the JSS. There is another method and it’s what is used by the built-in “Filevault Encryption” policy payload to get the keys back to your JSS. Jamf references this method in this old script at their GitHub. I revamped the core bits a couple years ago in a (nearly 7 year old) feature request: Manually Edit FileVault 2 Recovery Key

Telling the JSS Your Secrets

The takeaway from that is to realize we have a way to explicitly send keys to the JSS by placing 2 XML files in the /Library/Application Support/JAMF/run folder: file_vault_2_id.xml and file_vault_2_recovery_key.xml. Also note, Jamf has updated the process for the better in the last two years: a jamf recon (or two) is no longer required to send the key and validate it, instead JamfDaemon will send it immediately when both the files are detected. Which is nice, but it’s the subsequent recon validations where we have an opportunity to get grabby.

Cold Lamping, Hard Linking

So here’s the fun part: When recon occurs there’s lots of file traffic in /Library/Application Support/JAMF/tmp all sorts of transient scripts hit this folder. What we can do is make hard links to these files as they come in so when the link is removed in tmp another exists elsewhere and the file remains (just in our new location). EAGrabber.sh does exactly that (and a little bit more)

EAGrabber.sh can be easily modified to narrow it’s focus to the FileVault 2 key only, deleting the rest. What you do with the key is up to you: Send it somewhere else for safe keeping or keep it on device temporarily for a migration to another Jamf console. A script on the new JSS could then put that key on-disk into file_vault_2_recovery_key.xml file which will then import and validate, no decrypt/recrypt necessary. Hope this helps.

Cuidado Β‘ Achtung ! Alert

Jamf admins take note: Do you have hard coded passwords in your extension attributes or scripts? If so, then all your scripts are belong to us. Now, go read Obfuscation vs. Encryption from Richard Purves. Read it? OK, now consider what happens if you were to add a routine to capture the output of ps aww along with a hard-linking loop like in EAGraber.sh. If you are passing API credentials from policies via parameter, then ps can capture those parameters and even if you try and obscure them, if we’ve captured the script we can de-obfuscate them. This is a good reason to be really careful with what your API accounts can do. If you have an API account with Computer record Read rights that gets passed into a script via policy and you use LAPS, then captured API credentials could be used to harvest LAPS passwords via API. Keep this in mind and we’ll see if any meaningful changes will occur in recon and/or the script running process in the future (if you open a ticket you can reference PI-006270 regarding API credentials in the process list). In the meantime make API actions as short lived as possible and cross your fingers that only you, good and noble #MacAdmins read this blog. 🀞

jpt: jamf examples pt. 2

Over in the MacAdmins’ #bash channel I saw a I question regarding how to get the Sharing states of Bluetooth devices from system_profiler. The most succinct answer was to awk out the values:

system_profiler SPBluetoothDataType 2> /dev/null | awk '/State: / {print $2}'
Disabled
Disabled
Disabled

If you are using this for a Jamf Extension Attribute, I suppose it’ll do if you never want to allow any of them to be Enabled, but what if Internet Sharing was OK but not File Sharing? How would you match your Smart Group to multiple lines of unlabeled values? How would you match the first two but not the last two… and what if there was another USB Bluetooth device, that would add extra rows. Hmmm…

The answer for me, outputting the service name and the state on the same line. Since there isn’t a consistent line count from State: going back the service name, using something like grep -B n to include n lines of preceding data isn’t going to work.

      Services:
          Bluetooth File Transfer:
              Folder other devices can browse: ~/Public
              When receiving items: Accept all without warning
              State: Disabled
          Bluetooth File Exchange:
              Folder for accepted items: ~/Downloads
              When other items are accepted: Save to location
              When receiving items: Accept all without warning
              State: Disabled
          Bluetooth Internet Sharing:
              State: Disabled

So you know what I say the answer to that is? That’s right, jpt the JSON Power Tool! It can parse the -json output from system_profiler in a more structured way and it allows for the discovery of as many applicable Bluetooth devices might be on the system.

Here’s a sample run with Internet Sharing turned On as well as Bluetooth Sharing turned On

file_browsing: disabled
object_push: enabled
internet_sharing: enabled

File Browsing is set to “Never Allow” but File Receiving is in the affirmative (Accept and Open, Accept and Save, or Ask). The addition of labels gives us the ability to create a Smart Group to match specific services like “file_browing: enabled” or any other combination thereof (perhaps internet_sharing should always be enabled, who am I to say what your requirements are!).

About the jpt

The JSON Power Tool (jpt) is a parser/manipulator for JSON documents written in Javascript and shell and can run standalone or embedded in your scripts bash or zsh and all the way back to OS X 10.4 Tiger! Check it out at: https://github.com/brunerd/jpt

secret origins: the jpt

On building a JSON tool for macOS without using Python, perl, or Ruby.

In my work as a Macintosh engineer and administrator I’ve noticed macOS has lacked a bundled tool for working with JSON at the command line. Where XML has its xpath, if your shell script needs some JSON chops, it’ll require an external binary like jq or something else scripted in Python, Ruby or perl using their JSON modules. The problem is, those runtimes have been slated for removal from a future macOS. So I took that as challenge to devise a method to query and modify JSON data within shell scripts, that didn’t use one of those deprecated scripting runtimes and didn’t require an external binary dependency either. Could I achieve robust and native JSON parsing on a Mac by simply “living off the land”?

Why not just re-install the runtimes when Apple deprecates them?

“Why limit yourself like this? Just re-install the runtimes and move on”, you may ask. Well, I’d like to think that limitations can inspire creativity but we should also consider there may be some other reasons why Apple is discontinuing the inclusion of those runtimes. Some may say, “Apple Silicon + macOS 11.0 is the perfect time for them to clean house”, to which I’d have to agree that’s a very good reason and likely a factor. Others could say they are looking to tighten the screws to keep out unsigned code: maybe, they do like to glue things shut! But really, I think it’s more akin to the web-plugins of yore like Java and Flash. Apple does not want to be the conduit for deploying 3rd party party runtimes which increase the attack surface of macOS. This seems like the most reasonable of the explanations. So, if you accept that Apple is attempting to reduce attack surface, why increase it by re-installing Python, Ruby, or perl, just so a transient script (like a Jamf Extension Attribute) can parse a JSON file? My answer to that, is you don’t. You play the hand you’re dealt. Game on!

Looking for truffles (in a very small back yard)

Despite having another project (shui) that can output and invoke Applescript from within a shell scripts for generating user interfaces, I definitely knew that Applescript was not the way to go. Apple however, added to the languages Open Scripting Architecture (OSA) supports back in 2014 with OS X Yosemite (10.10), they added Javascript along with a bridge to the Cocoa Objective-C classes and they called it JXA: Javascript for Automation. This seemed like a promising place, so I started playing with osascript and figured out how to load files and read /dev/stdin using JXA, and while looking for an answer for garbled input from stdin I came upon a Japanese blog that mentioned jsc the JavaScriptCore binary which resides in the /System/Library/Frameworks/ JavaScriptCore.framework. Arigato! Pay dirt! πŸ€‘ jsc does exactly what we need it to do: It can interpret Javascript passed as an argument, can access the filesystem and read from /dev/stdin, and best of all is in non-Private System level Framework that exists all the way back to OS X 10.4! Just the kind of foundation on which to build the tool.

Homesteading jsc

The existence of jsc goes back all the way to OS X Tiger and it’s functionality has evolved over the years. In order to have a consistent experience in jpt from macOS 10.4 – 11.x+ a few polyfills had to be employed for missing functions, along with a few other workarounds regarding file loading, printing and exit codes (or lack thereof). Once those were addressed the jsc proved to be a highly optimized Javascript environment that’s blazingly fast. It spans 13 macOS releases and is even present in many Linux distros out-of-the-box (Ubuntu and CentOS) and can even be run on Windows when the Linux Subsytem is installed.

With the host environment sorted, I began working with the original JSONPath code by Stefan Goessner as the query language. I didn’t know about JSON Pointer yet so this strange beast was all I knew! It worked out really. I went full throttle into developing the “swiss army knife” of JSON tools. I really leaned into the Second System Effect, as described in the Mythical Man-month, it’s when you put every doodad, gizmo and doo-hicky in your 2nd product (my first simple JSON pretty-printer built in JS on jsc). Eventually though, after I reached a feature plateau, I came back around to the address the quirks of the original JSONPath code. I ended up rewriting signifigant chunks of JSONPath and released it as it’s own project: brunerd JSONPath. But I digress, let’s get back to the JSON Power Tool.

jpt: powers and abilities

At it’s most basic, jpt will format or “pretty print” any JSON given. jpt can also handle data retrieval from JSON document using either JSONPath or JSON Pointer syntax. JSONPath, while not a standard, is a highly expressive query language akin to XPath for XML, with poweful features like recursive search, filter expressions, slices, and unions. JSON Pointer on the other hand is narrower in focus, succint and easily expressed, and standardized but it does not offer any of the interogative features that I feel make JSONPath so intriguing. Finally, jpt can also modify values in a JSON document using standardized JSON Patch operations like: add, remove, replace, copy, move, and test, as well as the also standardized JSON Merge Patch operation. Altogether, the jpt can format, retrieve and alter JSON documents using only a bit of outer shell script plus a lot more Javascript on any macOS since 10.4! πŸ˜…

Where can I get the jpt?

Stop by the project’s GitHub page at: https://github.com/brunerd/jpt

There you will find the full source and also a minified version of the jpt for inclusion within your shell scripts. Since it’s never compiled you can always peer inside and learn from it, customize it, modify it, or just tinker around with it (usually the best teacher).

Future Plans

There will undoubtedly be continued work on the jpt. Surely there are less than optimal routines, un-idiomatic idioms, edge cases not found, and features yet to be realized. But as far as the core functionality goes though, it’s fairly feature complete in my opinion. Considering that one of my top 10 StrengthsFinder qualities is “Maximizer”, the odds are pretty good, I’ll keep honing the jpt‘s utility, size (smaller), and sharing more articles with examples on the kinds of queries and data alteration operations the jpt can so perform. Stay tuned!

Track and Tackle com.apple.macl

UPDATE: As of Big Sur (macOS 11) the com.apple.macl extended attribute can be removed using: xattr -rd com.apple.macl <file/folder>


Starting in macOS Catalina, an extended attribute (XA) named com.apple.macl is being added to the files and folder you work with. What does it do? How does it work? When does it get added? Where is the documentation?

All these are good questions, but there’s no official documentation. A good start is this assemblage of articles, here’s the highlights from that page and a few more…

Apple talked about this vaguely in their WWDC 2019 session 701, “Advances in macOS Security“, the Files and Folders fun starts at 21:21 but there’s nothing really about the implementation

Tom Bridge talked about com.apple.macl to the Penn State Macadmins back in August before Catalina’s release. He’s one of the first to notice this XA (and talk about it). Although, he says that curl will add this XA to downloads, it thankfully does not. Most likely it happened when he dragged it to Terminal or performed some other action on it.

Howard Oakley of the prolific Eclectic Light Company blog kicks the tires a bit more and finds some interesting quirks, as he always does.

Jeff Johnson though, really hits the nail on the head in illustrating what’s going on when you simply drag and drop a file or folder into Terminal and com.apple.macl is appended. He dropped this, December 18th and well… the holidays were around the corner and we all had better stuff to do!

It’s 2020 now and time to come back around to this. A very helpful post in the Apple Dev forums from an Apple employee, Quinn “the Eskimo”, decodes the data structure a bit more as well as what some of the conditions for having this XA appended are:

When a user selects a β€œprotected” file or folder in an NSOpenPanel in a non-sandboxed app on Catalina, consent is inferred and the app can access it.

It seems that 01 00 is a header of some form and … is a UUID associated with my test app.  I dug into how that UUID is set up and, well, it’s complex, and more of an implementation detail than I care to go into here on DevForums.

https://forums.developer.apple.com/thread/124121

Well, I would have loved the complex implementation detail! But I’ll tell you this: The app UUID in com.apple.macl is unique to EVERY computer. Jeff Johnson was right when he said: “The macl is effectively untraceable“.

However you can begin to see the shape of things when you write a tool to output the UUID in CSV like I have. Behold maclTrack.command

Run it in Terminal, give it files or folders as arguments, it’ll report on them, you can even specify a maximum depth for folders (-d) and silence reporting on items lacking com.apple.macl (-s), pipe it into tee to see the output and write it to a file.

This poor file got clobbered by multiple app UUIDs

Something interesting I noticed from using this tool is seeing the “Header” as Quinn called it, differ from 0100 to 0200. I believe 0200 is for drag and drop operations vs. a regular save which results in 0100.

Again, the app UUIDs you see on the files on your Mac will never be seen on another Mac ever – why? Because UUIDs are meant to be truly unique. Have you read the man page for uuidgen? They ain’t eff’in around!

 The uuidgen command generates a Universally Unique IDentifier (UUID), a 128-bit value guaranteed to be unique over both space and time.

Apple man page for uuidgen

So what’s the point of having a long lived XA that you can’t get rid of and that’s only useful on your local Mac? If it’s a permissions granting XA (the opposite of com.apple.quarantine which imposes restrictions) then why not let the user remove it? In some ways it’s a cross computer file tracking mechanism, albeit “anonymized”. Perhaps allowing it’s removal though (aka writing “nothing” to the XA) also allows for something to be written? This is where I could get in trouble speculating, it’s just a guess.

Perhaps you’ll forgive me that, since I have found a way to clear com.apple.macl without disabling SIP! Zip it. No, not the one spawned Finder’s “Compress” menu item, I mean /usr/bin/zip! I noticed command line zip would obliterate XAs during a project many years ago and it still does my friends! So behold maclTackle.command

This script has considerably less engineering to it because it’s a PoC and I’m really don’t want to take responsibility for something going awry should you try to clear com.apple.macl on your entire Desktop folder (no I haven’t tested that). But it will strip the XAs off a file that’s for sure! NOTE: USE A NON-IMPORTANT FILE with this script (make a duplicate). The script will overwrite the original (don’t like the behavior? It’s a script: Comment it out!)

🎢 Gonna wash that macl right outta my file… 🎢

Apple may come along and close this loophole now that it’s been pointed out or perhaps they won’t since they’d have to publish the modfication? Perhaps they’ll continue to purge “undesirable” binaries from macOS? They say they will stop shipping Python, perl and ruby in macOS some day and you know what xattr is written in? That’s right Python! Β―\_(ツ)_/Β―

Alright, come and get it while the gettin’s good! Thanks for reading!