Determining iCloud Drive and Desktop and Documents Sync Status in macOS

I’m on a roll with iCloud stuff. In this post I’d like to show you how you can determine if either iCloud Drive is enabled along with the “Desktop and Documents Folders” sync feature. While you can use MDM to turn these off, perhaps you like to know who you’d affect first! Perhaps the folks in your Enterprise currently using these features are in the C-Suite? I’m sure they’d appreciate a heads up before all their iCloud docs get removed from their Macs (when MDM disallowance takes affect it is swift and unforgiving).

iCloud Drive Status

When it comes to using on-disk artifacts to figure out the state of macOS I like to use the analogy of reading tea leaves. Usually it’s pretty straightforward but every now and then there’s something inscrutable and you have to take your best guess. For example iCloud Drive status is stored in your home folder at ~/Library/Preferences/MobileMeAccounts.plist but yet the Accounts key is an array. The question is how and why you could even have more than one iCloud account signed-in?! Perhaps a reader will tell me when and how you would ever have more than one? For now it seems inexplicable.

Update: It seems quite obvious now but as many folks did point out, of course you can add another iCloud account to Internet Accounts and it can sync Mail, Contacts, Calendars, Reminders, and Notes, just not iCloud Drive. Only one iCloud Drive user per user on a Mac. While I do have another AppleID I only use it for Media and Purchases and none of the other iCloud services. The code below stays the same as it is looking at all array entries. Aside: Boy, do I wish I could merge or transfer my old iTunes purchasing/media Apple ID with my main iCloud Apple ID! <weakly shakes fist at faceless Apple bureaucracy that for some reason hasn't solved the problem of merging Apple IDs>

Regardless of that mystery jpt can use the JSONPath query language to get us an answer in iCloudDrive_func.sh and the minified iCloudDrive_func.min.sh. Below is an edited excerpt:

#!/bin/bash
#Joel Bruner - iCloudDrive.func.sh - gets the iCloud Drive status for a console user

#############
# FUNCTIONS #
#############

function iCloudDrive()(

	#for brevity pretend we've pasted in the minified jpt function:
	#https://github.com/brunerd/jpt/blob/main/sources/jpt.min
	#for the full function see https://github.com/brunerd/macAdminTools/tree/main/Scripts

	consoleUser=$(stat -f %Su /dev/console)

	#if root grab the last console user
	if [ "${consoleUser}" = "root" ]; then
		consoleUser=$(/usr/bin/last -1 -t console | awk '{print $1}')
	fi

	userPref_json=$(sudo -u $consoleUser defaults export MobileMeAccounts - | plutil -convert json - -o -)

	#pref domain not found an empty object is returned
	if [ "${userPref_json}" = "{}" ]; then
		return 1
	else
		#returns the number paths that match
		matchingPathCount=$(jpt -r '$.Accounts[*].Services[?(@.Name == "MOBILE_DOCUMENTS" && @.Enabled == true)]' <<< "${userPref_json}" 2>/dev/null | wc -l | tr -d "[[:space:]]")
	
		if [ ${matchingPathCount:=0} -eq 0 ]; then
			return 1
		else
			return 0
		fi
	fi
)
########
# MAIN #
########

#example function usage, if leverages the return values
if iCloudDrive; then
	echo "iCloud Drive is ON"
	exit 0
else
	echo "iCloud Drive is OFF"
	exit 1
fi

The magic happens once we’ve gotten the JSON version of MobileMeAccount.plist I use jpt to see if there are any objects within the Accounts array with Services that have both a Name that matches MOBILE_DOCUMENTS and have an Enabled key that is set to true, the -r option on jpt tells it to output the the JSON Pointer path(s) the query matches. I could have used the -j option to output JSONPath(s) but either way a line is a line and that’s all we need. Altogether it looks like this: jpt -r '$.Accounts[*].Services[?(@.Name == "MOBILE_DOCUMENTS" && @.Enabled == true)]

Again because Accounts is an Array we have it look at all of them with $.Accounts[*]and in the off chance we get more than one we simply say if the number of matches is greater than zero then it’s on. This works very well in practice. This function could best be used as a JAMF Extension Attribute. I’ll leave that as a copy/paste exercise for the reader. Add it to your Jamf Pro EAs, let sit for 24-48 hours and check for results! ⏲ And while some of you might balk at a 73k Extensions Attribute, the execution time is on average a speedy .15s!

#!/bin/bash
#a pretend iCloudDrive Jamf EA 
#pretend we've pasted in the function iCloudDrive() above

if iCloudDrive; then
	result="ON"
else
	result="OFF"
fi

echo "<result>${result}</result>"

iCloud Drive “Desktop and Documents” status

Thankfully, slightly easier to determine yet devilishly subtle to discover, is determining the status of the “Desktop and Documents” sync feature of iCloud Drive. After searching in vain for plist artifacts, I discovered the clue is in the extended attributes of your Desktop (and/or Documents) folder! You can find the scripts at my GitHub here iCloudDriveDesktopSync_func.sh and the minified version iCloudDriveDesktopSync_func min.sh

#!/bin/bash
#Joel Bruner - iCloudDriveDesktopSync - gets the iCloud Drive Desktop and Document Sync Status for the console user

#############
# FUNCTIONS #
#############

#must be run as root
function iCloudDriveDesktopSync()(
	consoleUser=$(stat -f %Su /dev/console)

	#if root (loginwindow) grab the last console user
	if [ "${consoleUser}" = "root" ]; then
		consoleUser=$(/usr/bin/last -1 -t console | awk '{print $1}')
	fi

	#if this xattr exists then sync is turned on
	xattr_desktop=$(sudo -u $consoleUser /bin/sh -c 'xattr -p com.apple.icloud.desktop ~/Desktop 2>/dev/null')

	if [ -z "${xattr_desktop}" ]; then
		return 1
	else
		return 0
	fi
)

#example function usage, if leverages the return values
if iCloudDriveDesktopSync; then
	echo "iCloud Drive Desktop and Documents Sync is ON"
	exit 0
else
	echo "iCloud Drive Desktop and Documents Sync is OFF"
	exit 1
fi

The operation is pretty simple, it finds a console user or last user, then runs the xattr -p command as that user (anticipating this being run as root by Jamf) to see if the com.apple.icloud.desktop extended attribute exists on their ~/Desktop. In testing you’ll find if you toggle the “Desktop and Documents” checkbox in the iCloud Drive options, it will apply this to both of those folders almost immediately without fail. The function can be used in a Jamf Extension Attribute in the same way the iCloudDrive was above. Some assembly required. πŸ’ͺ

So, there you go! Another couple functions to read the stateful tea leaves of iCloud Drive settings. Very useful if you are about to disallow iCloud Drive and/or Desktop and Document sync via MDM but need to know who you are going to affect and let them know beforehand. Because I still stand by this sage advice: Don’t Be A Jerk. Thanks for reading you can find these script and more at my GitHub repo. Thanks for reading!

brunerd tools summer 2022 updates

Hey there, it’s summer 2022 and I wanted to share some updates on my shell tools for Mac admins.

No breaking changes or anything huge, but some nice incremental improvements all the same. When the JSONPath spec is finalized I’ll give jpt a more considerable under the hood improvement. For now though, I’m pretty happy with tweaking their behaviors to act like you’d expect (print help and exit if no input detected), to get out of your way when you need to debug your scripts (xtrace is now disabled inside them) and to try to understand what you mean when you ask them a question (key path queries now accepted). Because the tools and technologies we make should serve us, not the other way around. I hope these tools can save you time, energy, sanity or all the above!

β€’ jpt the “JSON Power Tool” has been updated to v1.0.2 with all the nice additions mentioned. If you don’t know already jpt can parse, query, and modify JSON every which way. It can be used as both a standalone utility as well as a function to be embedded into your shell scripts, without any other dependencies.

β€’ ljt the “little JSON tool” is jpt‘s smaller sibling weighing in at only 2k. While it lacks the transformational powers of jpt it can easily pluck a value out of JSON up to 2GB big and output results up to 720MB. It has been updated to v1.0.7 because dang it, even a small script can have just as many bugs as the big ones. Sometimes more if you’re not careful!

Both jpt and ljt now accept ye olde NextStep “keypath” expressions to help win over masochists who might still use plutil to work with JSON! 😁 (I kid, I kid!)

β€’ jse, the “JSON string encoder” does this one thing and does it well. It can be used in conjunction with jpt when taking in arbitrary data that needs encoding only. It’s a svelte 1.4k (in minified form) and runs considerably faster that jpt, so it’s ideal for multiple invocations, such as when preparing input data for a JSON Patch document. v1.0.2 is available here

β€’ shui lets you interact with your users with AppleScript but in the comfort of familiar shell syntax! You don’t need to know a line of AppleScript, although it does have the option to output it if you want to learn. Save time, brain cells and hair with shui, while you keep on trucking in shell script. It has been updated to v20220704.

P.S. All these tools still work in the macOS Ventura beta πŸ˜…

Improvisational jpt: Processing Apple Developer JSON transcript files

If you’ve downloaded the Developer app for the Mac there’s a trove of JSON transcripts cached in your home folder at ~/Library/Group Containers/group.developer.apple.wwdc/Library/Caches/Transcripts

Being curious I took a look at them using my JSON Power Tool jpt. In it’s default mode it will “pretty print” JSON or in Javascript parlance “stringify” them with a two space indent per nesting level. Inside it can be seen the transcripts are arrays of arrays inisde a uniquely named object. The 1st entry of the array is the time in seconds and the 2nd entry is the string we want.

Arrays of arrays

One of jpt’s cool features is that it supports the venerable yet nascent JSONPath query syntax. Using JSONPath we can use the recursive operator .. to go straight to the transcript object without needing determine the unique name of the parent object, then we want all the array within there [*]and inside those array we want the second entry of the 0 based array [1]. The query looks like this $..transcript[*][1]

Just text please

The query is single quoted so the shell doesn’t interpret the $ as the beginning of a variable name. The -T option for jpt it outputs text without quotes. The default output mode for jpt is JSON (double quoted strings). I added all sorts of other niceties to the script, as you’ll see below. The results are output to your ~/Desktop in a folder called Developer Transcripts

#!/bin/bash
: <<-LICENSE_BLOCK
Developer Transcript Extractor Copyright (c) 2022 Joel Bruner. Licensed under the MIT License. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
LICENSE_BLOCK

############
# VARIABLES #
############

destinationFolderName="Developer Transcripts"
destinationPath="${HOME}/Desktop/${destinationFolderName}"

#you'll need to download the Developer app and launch it: https://apps.apple.com/us/app/apple-developer/id640199958
transcriptPaths=$(find "${HOME}/Library/Group Containers/group.developer.apple.wwdc/Library/Caches/Transcripts/ByID" -name '*json')
contentsJSON="${HOME}/Library/Group Containers/group.developer.apple.wwdc/Library/Caches/contents.json"

########
# MAIN #
########

#either jpt should be installed or the function jpt.min can be pasted in here
if ! which jpt &>/dev/null; then
	echo "Please install jpt or embed jpt.min in this script: https://github.com/brunerd/jpt"
	exit 1
fi

#ensure the destination folder exists
[ ! -d "${destinationPath}" ] && mkdir "${destinationPath}"

#ignore spaces in file paths
IFS=$'\n'

#loop through each transcript json file
for transcriptPath in ${transcriptPaths}; do
	#id is just the file name without the path and extension
	id=$(cut -d. -f1 <<< "${transcriptPath##*/}")
	#a couple of nice-to-haves
	title=$(jpt -T '$.contents[?(@.id == "'"${id}"'")].title' "${contentsJSON}")
	description=$(jpt -T '$.contents[?(@.id == "'"${id}"'")].description' "${contentsJSON}")
	#change \ (disallowed in Unix) to : (Disallowed in Finder byt allowed in Unix)
	title=${title//\//:}
	url=$(jpt -T '$.contents[?(@.id == "'"${id}"'")].webPermalink' "${contentsJSON}")

	#"wwdc" always has the year in the id but not tech-talks or insights
	if ! grep -q -i wwdc <<< "$id"; then
		year="$(jpt '$.contents[?(@.id == "'"${id}"'")].originalPublishingDate' "${contentsJSON}" | date -j -r 1593018000 +"%Y")-"
		filename="${year}${id} - ${title}.txt"
	else
		filename="${id} - ${title}.txt"
	fi

	#put the ID and Title, the URL and Description at the top of the transcript
	echo "${id} - ${title}" > "${destinationPath}"/"${filename}"
	echo -e "${url}\n" >> "${destinationPath}"/"${filename}"
	echo -e "Description:\n${description}\n\nTranscript:" >> "${destinationPath}"/"${filename}"
	
	#append the transcript extract
	jpt -T '$..transcript[*][1]' "${transcriptPath}" >> "${destinationPath}"/"${filename}"

	#just echo out our progress
	echo "${destinationPath}"/"${filename}"
done

The final result is sortable folder of text files that you can easily QuickLook through.

Some serviceably readable contents!

So there you go! Some surprise JSON transcript files from the Apple Developer app, made me wonder how someone would turn them into human readable files. It turned out it was a fun and practical use of jpt and it’s support for JSONPath. You can download an installer package from the Releases page to try it out.

ljt, a little JSON tool for your shell script

ljt, the Little JSON Tool, is a concise and focused tool for quickly getting values from JSON and nothing else. It can be used standalone or embedded in your shell scripts. It requires only macOS 10.11+ or a *nix distro with jsc installed. It has no other dependencies.

You might have also seen my other project jpt, the JSON Power Tool. It too can be used standalone or embedded in a script however its features and size (64k) might be overkill in some case and I get it! I thought the same thing too after I looked at work of Mathew Warren, Paul Galow, and Richard Purves. Sometimes you don’t need to process JSON Text Sequences, use advanced JSONPath querying, modify JSON, or encode the output in a myriad of ways. Maybe all you need is something to retrieve a JSON value in your shell script.

Where jpt was an exercise in maximalism, ljt is one of essential minimalism – or at least as minimal as this CliftonStrengths Maximizer can go! πŸ€“ The minified version is mere 1.2k and offers a bit more security and functionality than a one-liner.

ljt features:
β€’ Query using JSON Pointer or JSONPath (canonical only, no filters, unions, etc)
β€’ Javascript code injection prevention for both JSON and the query
β€’ Multiple input methods: file redirection, file path, here doc, here string and Unix pipe
β€’ Output of JSON strings as regular text and JSON for all others
β€’ Maximum input size of 2GB and max output of 720MB (Note: functions that take JSON data as an environment variable or an argument are limited to a maximum of 1MB of data)
β€’ Zero and non-zero exit statuses for success and failure, respectively

Swing by the ljt Github page and check it out. There are two versions of the tool one is fully commented for studying and hacking (ljt), the other is a “minified” version without any comments meant for embedding into your shell scripts (ljt.min). The Releases page has a macOS pkg package to install and run ljt as a standalone utility.

Thanks for reading and happy scripting!

jpt 1.0 can deal with multiple JSON texts

jpt 1.0 can now deal with multiple JSON texts whether they are proper JSON Sequences or mutants like JSON Lines and NDJSON or just plain old concatenated JSON texts in one file. Even better it gives as good as it gets, it can also output in those formats as well. Let’s take a look!

Multiple JSON Text Options:
  -M "<value>" - options for working with multiple JSON texts
     S - Output JSON Text Sequences strictly conforming to RFC 7464 (default)
     N - Output newline delimited JSON texts
     C - Output concatenated JSON texts
     A - Gather JSON texts into an array, post-query and post-patching
     a - Gather JSON texts into an array, pre-query and pre-patching 

What is a JSON Text Sequence? It’s the standardized way (RFC 7464) to combine multiple JSON texts using the Record Separator (RS) control character (0x1E) as a delimiter.

The genius of JSON Text Sequences is that newlines are not made to be significant like they are in JSON Lines/NDJSON which introduce a fragility to what should be ignorable whitespace. These formats break the JSON specification. JSON sequences however are easily parsed and if needed more easily read by humans since the JSON texts do not need to be on a single line.

Let’s look at a simple JSON sequence. We can create the record separator character with the ANSI-C shell syntax $'\x1e' additional JSON-Seq requires that numbers be terminated with a newline (\n) for consistency I did that for each JSON text.

% jpt <<< $'\x1e1\n\x1e"a"\n\x1etrue\n'         
1
"a"
true

Now you might say that looks just like JSON Lines/NDJSON but what you don’t see are the RS characters, they are stripped out in Terminal. However if we send the output to a file or pipe it into bbedit you will see them appear as ΒΏ

RS (0x1e) characters in bbedit

With jpt 1.0 we can now choose to convert multiple JSON text (whether sequences, lines, or concatenated) into an array, either pre or post query and patching.

#-Ma and -MA without a query or patch are functionally the same
% jpt -Ma <<< $'\x1e1\n\x1e"a"\n\x1etrue\n'
[
  1,
  "a",
  true
]

Now let’s look at 3 separate JSON texts of arrays and see how -Ma and -MA differ

#an array of the 3 arrays is created pre-query
#thus /2 corresponds to the last JSON text
#-Ma is most useful to get a specific JSON text
% jpt -Ma /2 <<< $'[1,2,3]\n[4,5,6]\n[7,8,9]\n'
[
  7,
  8,
  9
]

#the query /2 is run first on all 3 texts
#then those results 3, 6, and 9 are gathered into an array
% jpt -MA /2 <<< $'[1,2,3]\n[4,5,6]\n[7,8,9]\n'
[
  3,
  6,
  9
]

One of thing about JSON Lines and NDJSON is they take otherwise ignorable whitespace like newline and make it something significant, the introduce a brittleness to the otherwise robust JSON spec. Let’s see how jpt handles it

#-MN will output NDJSON and correct the missing newline
% jpt -MN <<< $'{"a":1}\n{"b":2}{"c":3}'
{"a":1}
{"b":2}
{"c":3}

As you can see a simple line based NDJSON parser would have failed with the last object, jpt treats anything that doesn’t parse as a possibly concatenated JSON.

You can also output concatenated JSON… but why?! Maybe you have your reasons, if so, use -MC

#no RS characters with concatenated JSON
jpt -MC <<< $'{"a":1}\n{"b":2}{"c":3}'
{
  "a": 1
}
{
  "b": 2
}
{
  "c": 3
}

If you work with multiple JSON texts such as JSON logs then jpt might be a useful addition to your arsenal of tools. It can be downloaded from my jpt Releases page at Github.

jpt: see JSON differently

JSON is wonderfully efficient at encoding logical structure and hierarchy into a textual form, but sometimes our eyes and brains need some help making sense of that. Let’s see how jpt can help!

The Shape of JSON

JSON uses a small and simple palette of brackets and curly braces, to express nest-able structures like objects and arrays. However once inside those structures it’s hard to visualize the lineage of the members.

Using jpt however, one can get a better idea of what the shape of a JSON document is.

For example object-person.json:

{
  "person": {
    "name": "Lindsay Bassett",
    "nicknames": [
      "Lindy",
      "Linds"
    ],
    "heightInInches": 66,
    "head": {
      "hair": {
        "color": "light blond",
        "length": "short",
        "style": "A-line"
      },
      "eyes": "green"
    },
    "friendly":true
  }
}

Using jpt -L we can output what I am currently calling “JSONPath Object Literals”. With JPOL the entire path is spelled out for each item as well as it’s value expressed as a JSON object. You can also roundtrip JPOL right back into jpt and you’ll get the JSON again.

% jpt -L object-person.json
$={}
$["person"]={}
$["person"]["name"]="Lindsay Bassett"
$["person"]["nicknames"]=[]
$["person"]["nicknames"][0]="Lindy"
$["person"]["nicknames"][1]="Linds"
$["person"]["heightInInches"]=66
$["person"]["head"]={}
$["person"]["head"]["hair"]={}
$["person"]["head"]["hair"]["color"]="light blond"
$["person"]["head"]["hair"]["length"]="short"
$["person"]["head"]["hair"]["style"]="A-line"
$["person"]["head"]["eyes"]="green"
$["person"]["friendly"]=true

#roundtrip back into jpt (-i0 means zero indents/single line)
jpt -L object-person.json | jpt -i0
{"person":{"name":"Lindsay Bassett","nicknames":["Lindy","Linds"],"heightInInches":66,"head":{"hair":{"color":"light blond","length":"short","style":"A-line"},"eyes":"green"},"friendly":true}}

If you were to fire up jsc or some other JavaScript interpreter, declare $ as a variable using var $ and then copy and paste the output above and you will have this exact object.

Perhaps though you prefer single quotes? Add -q to use single quotes for the property names but values remain double quoted.

#-q will single quote JSONPath only, strings remain double quoted 

% jpt -qL object-person.json         
$={}
$['person']={}
$['person']['name']="Lindsay Bassett"
$['person']['nicknames']=[]
$['person']['nicknames'][0]="Lindy"
$['person']['nicknames'][1]="Linds"
$['person']['heightInInches']=66
$['person']['head']={}
$['person']['head']['hair']={}
$['person']['head']['hair']['color']="light blond"
$['person']['head']['hair']['length']="short"
$['person']['head']['hair']['style']="A-line"
$['person']['head']['eyes']="green"
$['person']['friendly']=true

#-Q will single quote both
% jpt -QL object-person.json
$={}
$['person']={}
$['person']['name']='Lindsay Bassett'
$['person']['nicknames']=[]
$['person']['nicknames'][0]='Lindy'
$['person']['nicknames'][1]='Linds'
$['person']['height in inches']=66
$['person']['head']={}
$['person']['head']['hair']={}
$['person']['head']['hair']['color']='light blond'
$['person']['head']['hair']['length']='short'
$['person']['head']['hair']['style']='A-line'
$['person']['head']['eyes']='green'
$['person']['friendly']=true

If you don’t like bracket notation use -d for dot notation (when the name permits). -Q will single quote both values and key names if using brackets. To save a few lines you can use -P to only print primitives (string, number, booleans, null) and will omit those empty object and array declarations. This too can roundtrip back into jpt, it will infer the structure type (object or array) based on the key type (string or number when in brackets).

% jpt -dQLP object-person.json       
$.person.name='Lindsay Bassett'
$.person.nicknames[0]='Lindy'
$.person.nicknames[1]='Linds'
$.person.heightInInches=66
$.person.head.hair.color='light blond'
$.person.head.hair.length='short'
$.person.head.hair.style='A-line'
$.person.head.eyes='green'
$.person.friendly=true

#roundtrip into jpt, structure types can be inferred
% jpt -dQLP object-person.json | jpt -i0    
{"person":{"name":"Lindsay Bassett","nicknames":["Lindy","Linds"],"heightInInches":66,"head":{"hair":{"color":"light blond","length":"short","style":"A-line"},"eyes":"green"},"friendly":true}}

Now perhaps we don’t care so much about the data but about the structure? Then we can use the -J (JSONPath) or -R (JSON Pointer) output combined with -K or -k for the key names only

#-K key names are double quoted
% jpt -JK /Users/brunerd/Documents/TestFiles/object-person.json
$
  "person"
    "name"
    "nicknames"
      0
      1
    "height in inches"
    "head"
      "hair"
        "color"
        "length"
        "style"
      "eyes"
    "friendly"

#-q/-Q can change the quoting to singles of course
#-C will show the constructor type of every element
% jpt -JKqC /Users/brunerd/Documents/TestFiles/object-person.json
$: Object
  'person': Object
    'name': String
    'nicknames': Array
      0: String
      1: String
    'height in inches': Number
    'head': Object
      'hair': Object
        'color': String
        'length': String
        'style': String
      'eyes': String
    'friendly': Boolean


#-k for unquotes key names, -C for constructor, -i1 for 1 space indents 

% jpt -JkC -i1 /Users/brunerd/Documents/TestFiles/object-person.json
$: Object
 person: Object
  name: String
  nicknames: Array
   0: String
   1: String
  height in inches: Number
  head: Object
   hair: Object
    color: String
    length: String
    style: String
   eyes: String
  friendly: Boolean

It’s been a lot of fun creating jpt and discovery new ways to express JSON using an emerging notation like JSONPath (which is going through the RFC process now). If you’d like to see JSON a different way download jpt and give it a try!

Helping truncated JSON data with jpt 1.0

jpt 1.0 now has the ability to detect and help truncation in strings, arrays and objects. The -D option can detect this in newline delimited and concatenated JSON and -H will help the JSON by closing up strings, arrays, and objects. For example, here’s two JSON text concatenated both with truncation:

jpt -i0 -DH <<< $'{"a":[1,2 {"b":[3,4,'
{"a":[1,2]}
{"b":[3,4,null]}

#heredoc to more clearly show the input data
% jpt -i0 -DH <<EOF
heredoc> {"a":[1,2
heredoc> {"b":[3,4,
heredoc> EOF
{"a":[1,2]}
{"b":[3,4,null]}

If there is a trailing comma, a null will be put there to signify the fact there was an additional value before truncation occured, in all cases one should assume some data has been lost. The -i0 option is for zero indents, single line JSON however it is not NDJSON. When multiple JSON texts are input, as above, JSON Sequence (RFC 7464) are the default output. JSON Sequences are simply a record separator (RS) character at the beginning of JSON text. RS characters do not appear in the Terminal. If you pipe the output to bbedit or redirect to a file you’ll see them. To get some commentary on the repairs, use the -c option:

jpt -i0 -cDH <<< $'{"a":[1,2 {"b":[3,4,'         
{"a":[1,2]}
{"b":[3,4,null]}
--> Original error: SyntaxError: JSON Parse error: Expected ']'
--> Byte 10: Truncation Detected
--> Truncation help for JSON Text (1), bytes: 1-10
--> Truncation help after byte 10, array end: ]
--> Truncation help after byte 10, object end: }
--> Truncation help for JSON Text (2), bytes: 11-21
--> Truncation help after byte 21, missing value and array end: null]
--> Truncation help after byte 21, object end: }
--> Bytes 1-10: JSON Text (1)
--> Bytes 11-21: JSON Text (2)

echo $?                                 
1

As you can see -c can get pretty verbose depending on the issues. If there is any output with -c the exit code of jpt will be 1, if there is nothing to fix it will exit 0. This can be useful if you need to know if the source JSON needed any help or not.

jpt 1.0 is there when your JSON needs some help, not just when it’s perfectly formatted. If that sounds cool give it a try, download the Mac .pkg at github.com/brunerd/jpt/releases

jpt 1.0 text encoding fun

Besides JSON, jpt (the JSON power tool) can also output strings and numbers in a variety of encodings that the sysadmin or programmer might find useful. Let’s look at the encoding options from the output of jpt -h

% jpt -h
...
-T textual output of all data (omits property names and indices)
  Options:
	-e Print escaped characters literally: \b \f \n \r \t \v and \\ (escapes formats only)
	-i "<value>" indent spaces (0-10) or character string for each level of indent
	-n Print null values as the string 'null' (pre-encoding)

	-E "<value>" encoding options for -T output:

	  Encodes string characters below 0x20 and above 0x7E with pass-through for all else:
		x 	"\x" prefixed hexadecimal UTF-8 strings
		O 	"\nnn" style octal for UTF-8 strings
		0 	"\0nnn" style octal for UTF-8 strings
		u 	"\u" prefixed Unicode for UTF-16 strings
		U 	"\U "prefixed Unicode Code Point strings
		E 	"\u{...}" prefixed ES2016 Unicode Code Point strings
		W 	"%nn" Web encoded UTF-8 string using encodeURI (respects scheme and domain of URL)
		w 	"%nn" Web encoded UTF-8 string using encodeURIComponent (encodes all components URL)

		  -A encodes ALL characters
	
	  Encodes both strings and numbers with pass-through for all else:
		h 	"0x" prefixed lowercase hexadecimal, UTF-8 strings
		H 	"0x" prefixed uppercase hexadecimal, UTF-8 strings
		o 	"0o" prefixed octal, UTF-8 strings
		6 	"0b" prefixed binary, 16 bit _ spaced numbers and UTF-16 strings
		B 	"0b" prefixed binary, 8 bit _ spaced numbers and UTF-16 strings
		b 	"0b" prefixed binary, 8 bit _ spaced numbers and UTF-8 strings

		  -U whitespace is left untouched (not encoded)

Strings

While the above conversion modes will do both number and string types, these options will work only on strings (numbers and booleans pass-through). If you work with with shell scripts these techniques may be useful.

If you store shell scripts in a database that’s not using utf8mb4 table and column encodings then you won’t be able to include snazzy emojis to catch your user’s attention! In fact this WordPress install was so old (almost 15 years!) the default encoding was still latin1_swedish_ci, which an odd but surprisingly common default for many old blogs. Also if you store your scripts in Jamf (still in v10.35 as of this writing) it uses latin1 encoding and your 4 byte characters will get mangled. Below you can see in Jamf things look good while editing, fails once saved, and the eventual workaround is to use an coding like \x escaped hex (octal is an alternate)

Let’s use the red “octagonal sign” emoji, which is a stop sign to most everyone around the world, with the exception of Japan and Libya (thanks Google image search). Let’s look at some of the way πŸ›‘ can be encoded in a shell script

#reliable \x hex notation for bash and zsh
% jpt -STEx <<< "Alert πŸ›‘"
Alert \xf0\x9f\x9b\x91

#above string can be  in both bash and zsh
% echo $'Alert \xf0\x9f\x9b\x91'
Alert πŸ›‘

#also reliable, \nnn octal notation
% jpt -STEO <<< "Alert πŸ›‘"
Alert \360\237\233\221

#works in both bash and zsh
% echo $'Alert \360\237\233\221'
Alert πŸ›‘

#\0nnn octal notation
% jpt -STE0 <<< "Alert πŸ›‘"
Alert \0360\0237\0233\0221

#use with shell builtin echo -e and ALWAYS in double quotes
#zsh does NOT require -e but bash DOES, safest to use -e
% echo -e "Alert \0360\0237\0233\0221"
Alert πŸ›‘

#-EU code point for zsh only
% jpt -STEU <<< "Alert πŸ›‘"
Alert \U0001f6d1

#use in C-style quotes in zsh
% echo $'Alert \U0001f6d1'
Alert πŸ›‘

The -w/-W flags can encode characters for use in URLs

#web/percent encoded output in the case of non-URLs -W and -w are the same
% jpt -STEW <<< πŸ›‘  
%F0%9F%9B%91

#-W URL example (encodeURI)
jpt -STEW <<< http://site.local/page.php?wow=πŸ›‘
http://site.local/page.php?wow=%F0%9F%9B%91

#-w will encode everything (encodeURIComponent)
% jpt -STEw <<< http://site.local/page.php?wow=πŸ›‘
http%3A%2F%2Fsite.local%2Fpage.php%3Fwow%3D%F0%9F%9B%91

And a couple other oddballs…

#text output -T (no quotes), -Eu for \u encoding
#not so useful for the shell scripter
#zsh CANNOT handle multi-byte \u character pairs
% jpt -S -T -Eu <<< "Alert πŸ›‘"
Alert \ud83d\uded1

#-EE for an Javascript ES2016 style code point
% jpt -STEE <<< "Alert πŸ›‘"
Alert \u{1f6d1}

You can also \u encode all characters above 0x7E in JSON with the -u flag

#JSON output (not using -T)
% jpt <<< '"Alert πŸ›‘"'
"Alert πŸ›‘"

#use -S to treat input as a string without requiring hard " quotes enclosing
% jpt -S <<< 'Alert πŸ›‘'
"Alert πŸ›‘"

#use -u for JSON output to encode any character above 0x7E
% jpt -Su <<< 'Alert πŸ›‘'
"Alert \ud83d\uded1"

#this will apply to all strings, key names and values
% jpt -u <<< '{"πŸ›‘":"stop", "message":"Alert πŸ›‘"}' 
{
  "\ud83d\uded1": "stop",
  "message": "Alert \ud83d\uded1"
}

Whew! I think I covered them all. If there are newlines, tabs and other invisibles you can choose to output them or leave them encoded when you are outputting to text with -T

#JSON in, JSON out
jpt <<< '"Hello\n\tWorld"'
"Hello\n\tWorld"

#ANSI-C string in, -S to treat as string despite lack of " with JSON out
% jpt -S <<< $'Hello\n\tWorld' 
"Hello\n\tWorld"

#JSON in, text out: -T alone prints whitespace characters
% jpt -T <<< '"Hello\n\tWorld"'
Hello
	World

#use the -e option with -T to encode whitespace
% jpt -Te <<< '"Hello\n\tWorld"'
Hello\n\tWorld

Numbers

Let’s start simply with some numbers. First up is hex notation in the style of 0xhh and 0XHH. This encoding has been around since ES1, use the -Eh and -EH respectively to do so. All alternate output (i.e. not JSON) needs the -T option. In shell you can combine multiple options/flags together except only the last flag can have an argument, like -E does below.

#-EH uppercase hex
% jpt -TEH <<< [255,256,4095,4096] 
0xFFa
0x100
0xFFF
0x1000

#-Eh lowecase hex
% jpt -TEh <<< [255,256,4095,4096]
0xff
0x100
0xfff
0x1000

Next up are ye olde octals. Use the -Eo option to convert numbers to ye olde octals except using the more modern 0o prefix introduced in ES6

-Eo ES6 octals
% jpt -TEo <<< [255,256,4095,4096]    
0o377
0o400
0o7777
0o10000

Binary notation debuted in the ES6 spec, it used a 0b prefix and allows for _ underscore separators

#-E6 16 bit wide binary
% jpt -TE6 <<< [255,256,4095,4096]
0b0000000011111111
0b0000000100000000
0b0000111111111111
0b0001000000000000

#-EB 16 bit minimum width with _ separator per 8
% jpt -TEB <<< [255,256,4095,4096]
0b00000000_11111111
0b00000001_00000000
0b00001111_11111111
0b00010000_00000000

#-Eb 8 bit minimum width with _ separator per 8
% jpt -TEb <<< [15,16,255,256,4095,4096]
0b00001111
0b00010000
0b11111111
0b00000001_00000000
0b00001111_11111111
0b00010000_00000000

If you need to encode strings or numbers for use in scripting or programming, then jpt might be a handy utility for you and your Mac and if your *nix has jsc then it should work also. Check the jpt Releases page for Mac installer package download.

jpt 1.0 and JSON5 rehab

JSON5 is not JSON nor is an IETF standard but jpt 1.0 can rehabilitate it back to standards compliant JSON. Let’s take the example from its web page.

{
  // comments
  unquoted: 'and you can quote me on that',
  singleQuotes: 'I can use "double quotes" here',
  lineBreaks: "Look, Mom! \
No \\n's!",
  hexadecimal: 0xdecaf,
  leadingDecimalPoint: .8675309, andTrailing: 8675309.,
  positiveSign: +1,
  trailingComma: 'in objects', andIn: ['arrays',],
  "backwardsCompatible": "with JSON",
}

Now let’s send it through jpt

% jpt json5_example.txt        
{
  "unquoted": "and you can quote me on that",
  "singleQuotes": "I can use \"double quotes\" here",
  "lineBreaks": "Look, Mom! No \\n's!",
  "hexadecimal": 912559,
  "leadingDecimalPoint": 0.8675309,
  "andTrailing": 8675309,
  "positiveSign": 1,
  "trailingComma": "in objects",
  "andIn": [
    "arrays"
  ],
  "backwardsCompatible": "with JSON"
}

That last line is a bit misleading. JSON5 texts are totally not backward compatible with JSON parsers. What they mean is that JSON5 parsers are backward compatible with JSON. FYI – Every single JSON5 addition is not compatible with JSON, I should know I had to write parsing rules and behaviors to deal with it! Let’s use the -c flag to comment on all the work that went into un-junking the above example

% jpt -c ./json5_example.txt 
{
  "unquoted": "and you can quote me on that",
  "singleQuotes": "I can use \"double quotes\" here",
  "lineBreaks": "Look, Mom! No \\n's!",
  "hexadecimal": 912559,
  "leadingDecimalPoint": 0.8675309,
  "andTrailing": 8675309,
  "positiveSign": 1,
  "trailingComma": "in objects",
  "andIn": [
    "arrays"
  ],
  "backwardsCompatible": "with JSON"
}
--> Original error: SyntaxError: JSON Parse error: Unrecognized token '/'
--> Byte 5: // line comment (JSON5)
--> Byte 19: unquoted object key name (JSON5)
--> Byte 29: single quoted string (JSON5)
--> Byte 63: unquoted object key name (JSON5)
--> Byte 77: single quoted string (JSON5)
--> Byte 113: unquoted object key name (JSON5)
--> Byte 138: \ escaped newline in string (JSON5)
--> Byte 153: unquoted object key name (JSON5)
--> Byte 164: 0x hex value (JSON5)
--> Byte 177: unquoted object key name (JSON5)
--> Byte 198: decimal point wihtout preceding digit (JSON5)
--> Byte 208: unquoted object key name (JSON5)
--> Byte 228: trailing decimal lacking mantissa (JSON5)
--> Byte 233: unquoted object key name (JSON5)
--> Byte 247: explicit plus + for value (JSON5)
--> Byte 253: unquoted object key name (JSON5)
--> Byte 268: single quoted string (JSON5)
--> Byte 282: unquoted object key name (JSON5)
--> Byte 290: single quoted string (JSON5)
--> Byte 296: trailing comma (JSON5)
--> Byte 336: trailing comma (JSON5)

Yeah there’s so much wrong with JSON5 vs JSON its not funny. It’s baffling that Apple is acknowledging it by adding JSON5 parsing to their frameworks! At least they don’t generate it! Despite not being funny, let’s try anyway, here’s some more JSON5 example to show some of the other ways it can lead you astray.

{
	"jpt_lineBreakHandling": "Extended parsing to ignore tabs \
	after escaped newlines.\nIf JSON5 is supposed to be human readable \
	shouldn't tabs be included in the escaping?",
	"leadingDecimalPoint": .8675309, 
	"andTrailingDecimal": 8675309.,
	"negativeLeadingDecimalPoint": -.1234, 

	//Infinity converts to null, use -I for string
	"+Infinity": +Infinity,
	"-Infinity": -Infinity,
	"Infinity": Infinity,
	//null is not a number now is it? done.
	"NaN": NaN,
}

Running this through jpt we’ll get pure JSON output

% jpt ./json5_examples_brunerd.txt 
{
  "jpt_lineBreakHandling": "Extended parsing to ignore tabs after escaped newlines.\nIf JSON5 is supposed to be human readable shouldn't tabs be included in the escaping?",
  "leadingDecimalPoint": 0.8675309,
  "andTrailingDecimal": 8675309,
  "negativeLeadingDecimalPoint": -0.1234,
  "+Infinity": null,
  "-Infinity": null,
  "Infinity": null,
  "NaN": null
}


#-I to convert to signed Infinity strings instead of null
#-8 to convert NaN to string "NaN" (-N was taken, like NaN 8 is like Infinity but not quite :)
% jpt -I8 ./json5_examples_brunerd.txt 
{
  "jpt_lineBreakHandling": "Extended parsing to ignore tabs after escaped newlines.\nIf JSON5 is supposed to be human readable shouldn't tabs be included in the escaping?",
  "leadingDecimalPoint": 0.8675309,
  "andTrailingDecimal": 8675309,
  "negativeLeadingDecimalPoint": -0.1234,
  "+Infinity": "+Infinity",
  "-Infinity": "-Infinity",
  "Infinity": "Infinity",
  "NaN": "NaN"
}

The fact that they added the data type of +/- Infinity and NaN really complicates the conversion since JSON doesn’t have equivalents. By default Infinity converts to null but -I will convert it to the string "Infinity" with signedness if specified. NaN is also converted to null by default but the -8 flag will convert it to the string "NaN". Why 8? Because it’s sort like infinity but it’s not, just like NaN.

If you have need to rehab JSON5 back to JSON then jpt might be able to help you out. It can run on Mac all the way back to OS X Tiger (10.4) all the way up to the newest macOS Monterey (12) and on other *nixes with jsc installed. Check the jpt Releases page

merging JSON objects with jpt 1.0

A cool new feature for jpt 1.0 is the ability perform additional merging operations beyond JSON Merge Patch (RFC 7396). To start with let’s check out a JSON Merge Patch operation, using the mergepatch operation:

#key "a" is overwritten with the -v value, "b" is removed because null
% jpt -i0 -o mergepatch -v '{"a":"🟒","b":null}' <<< '{"a":"πŸ”΄","b":"🟑"}'                                                                           
{"a":"🟒"}

# use -u to encode all characters above 0x7E with \u escaping
% jpt -u -i0 -o mergepatch -v '{"a":"🟒","b":null}' <<< '{"a":"πŸ”΄","b":"🟑"}'                                                                           
{"a":"\ud83d\udfe2"}

Only one key survives, since the Merge Patch spec says that any key with a null value will delete that key in the input JSON. Now perhaps you might want null to survive, if so, use the merge operation:

jpt -i0 -o merge -v '{"a":"🟒","b":null}' <<< '{"a":"πŸ”΄","b":"🟑"}' 
{"a":"🟒","b":null}

Now for the more advanced modes. -o merge0 will only merge in keys that do not exist. Existing keys remain untouched. To order the keys alphabetically use -O (the letter O)

% jpt -O -o merge0 -v '{"common":"🟒","uniqToMergeData":"🟒"}' <<< '{"common":"πŸ”΄","uniqToSource":"πŸ”΄"}'
{
  "common": "πŸ”΄",
  "uniqToMergeData": "🟒",
  "uniqToSource": "πŸ”΄"
}

As you can see only the uniqToMergeData key comes through, the other two keys common and uniqToSource do not change. Useful if you are upgrading an object to a newer format and want to retain existing key values and only add the new keys with empty or default values.

merge1 let’s us do the inverse of merge0 and merge only the keys that exist in the target JSON. Keys that do not exist do not merge over.

% jpt -o merge1 -v '{"uniqToMergeData":"🟒","common":"🟒"}' <<< '{"uniqToSource":"πŸ”΄","common":"πŸ”΄"}'
{
  "uniqToSource": "πŸ”΄",
  "common": "🟒"
}

If you like pictures (I sure do) and are a visual learner you can think of merge0 and merge1 like the Exclude and Intersect Shape Areas in Photoshop.

I hope these new merge modes could be useful to you, if you have a Mac (or *nix with jsc installed) download jpt from my GitHub Releases page and give it a go!