Author: brunerd

Automating automatic login for macOS

brunerd

I recently had some Zoom Room Macs that needed some automation love and I thought I’d share how you can enable Automatic Login via script, perhaps you have several dozen and use Jamf or some other Mac management tool? Interested? Read on! Currently what’s out there are either standalone encoders only or part of larger packaging workflow to create a new user. The standalone encoders lacked some niceties like logging or account password verification and the package method added the required dependency of packaging if any changes were required. Above all, every script required Python, perl or Ruby, which are all on Apple’s hit list for removal in an upcoming OS release. For now macOS Monterey still has all of these runtimes but there will come a day when macOS won’t and will you really want to add 3rd party scripting runtimes and weaken your security by increasing attack surface when you can weaken your security using just shell? 😜 So for some fun, I re-implemented the /etc/kcpassword encoder in shell so it requires only awk, sed, and xxd, all out of the box installs. I also added some bells and whistles too.

Some of the features are:

  • If the username is empty, it will fully disable Automatic Login. Since turning it off via the System Preferences GUI does not remove the /etc/kcpassword file if it has been enabled (!)
  • Ensures the specified user exists
  • Verifies the password is valid for the specified user
  • Can handle blank passwords
  • Works on OS X 10.5+ including macOS 12 Monterey

For the Jamf admin the script is setAutomaticLogin.jamf.sh and for standalone usage get setAutomaticLogin.sh, both take a username and password, in that order and then enable Automatic Login if it all checks out. The difference with the Jamf script is that the first parameter is ${3} versus ${1} for the standalone version.

Also here’s a well commented Gist as a little show and tell for what the shell only version of the kcpassword encoder looks like. Enjoy!

Update: I meant to expound on how it didn’t seem that padding kcpassword to a length multiple of 12 was necessary, since it had been working fine on the versions I was testing with but then I tested on Catalina (thinking of this thread) and was proven wrong by Catalina. I then padded with 0x00 with HexFiend in a successful test but was reminded that bash can’t handle that character in strings, instead I padded with 0x01, which worked on some macOS versions but not others. Finally, I settled on doing on what macOS does somewhat, whereas it pads with the next cipher character then seemingly random data, I pad with the cipher characters for the length of the padding. This works for Catalina and other of macOS versions. 😅

Big Hat Tip to Pico over at MacAdmins Slack for pointing out this issue in pycreateuserpkg where it’s made clear that passwords with a lengths of 12 or multiples thereof, need another 12 characters of padding (or in some newer OSes at least one cipher character terminating the kcpassword data, thanks! 👍

macOS Compatibility Fun!

brunerd

Compatibility Questions

If you work with Macs and Jamf then you know every year there’s a new per OS Extension Attribute (EA) or Smart Group (SG) recipe to determine if macOS will run on your fleets hardware. However I asked myself: What if a single Extension Attribute script could fill the need, requiring only a periodic updating of Model IDs and the addition of new macOSes?

Then I also asked: Could this same script be re-purposed to output both text and CSV, not just for the script’s running host but for a list of Model IDs? And the answer was a resounding yes on all fronts!

EA Answers

So, my fellow Jamf admin I present to you macOSCompatibility.sh in its simplest form you just run the script and it will provide ultra-sparse EA output like: <result>10.14 10.15 11</result> this could then be used as a Smart Group criteria. Something like “macOS Catalina Compatible” would then match all Macs using LIKE 10.15 or “Big Sur Incompatible” would use NOT LIKE 11, of course care would be taken if you were also testing for 10.11 compatibility, however the versionsToCheck variable in the script can limit the default range to something useful and speeds things up the less version there are. I hope this helps Jamf admins who have vast unwieldy fleets where hardware can vary wildly across regions or departments,

CSV Answers

Now if you provide a couple arguments like so: ./macOSCompatibility.sh -c -v ALL ALL > ~/Desktop/macOSCompatibilityMatrix.csv you will get a pretty spiffy CSV that let’s you visualize which Mac models over the years have enjoyed the most and least macOS compatibility. This is my favorite mode, you can use it to assess the OS coverage of past Macs.

See macOSCompatibilityMatrix.csv for an example of the output. If you bring that CSV into Numbers or Excel you can surely liven it up with some Conditional Formatting! This is the barest of examples:

Can you spot the worst and best values?

Text Answers

If you don’t use the -c flag then it’ll just output in plain or text, like so: ./macOSCompatibility.sh -v ALL ALL

iMacPro1,1: 10.13 10.14 10.15 11
MacBook1,1: 10.4 10.5 10.6
MacBook2,1: 10.4 10.5 10.6 10.7
MacBook3,1: 10.5 10.6 10.7
MacBook4,1: 10.5 10.6 10.7
MacBook5,1: 10.5 10.6 10.7 10.8 10.9 10.10 10.11
MacBook6,1: 10.6 10.7 10.8 10.9 10.10 10.11 10.12 10.13
MacBook7,1: 10.6 10.7 10.8 10.9 10.10 10.11 10.12 10.13
MacBook8,1: 10.10 10.11 10.12 10.13 10.14 10.15 11
MacBook9,1: 10.11 10.12 10.13 10.14 10.15 11
MacBook10,1: 10.12 10.13 10.14 10.15 11
MacBookAir1,1: 10.5 10.6 10.7
MacBookAir2,1: 10.5 10.6 10.7 10.8 10.9 10.10 10.11

Wrapping Up

Now, it’s not totally perfect since some models shared Model IDs (2012 Retina and Non-Retina MacBook Pros for example) but for the most part the Intel Mac Model IDs were sane compared to the PPC hardware Model IDs: abrupt jumps, overlaps, and re-use across model familes. Blech! I’m glad Apple “got religion” for Model IDs (for the most part) when Intel CPUs came along. I did attempt to go back to 10.1-10.3 with PPC hardware but it was such a mess it wasn’t worth it. However testing Intel, Apple Silicon and VMs against macOS 10.4 – 11+ seems to have some real use and perhaps you think so too? Thanks for reading!

jpt is unphased by macOS’s malformed JSON

brunerd

A funny thing happened on the way to writing a new blog entry. Instead of finding an example JSON file that ships with macOS, I stumbled upon a slow creep of malformed JSON files that started in OS X El Capitan and continues into Big Sur. While I can’t do much about deviant JSON files in macOS, I can control how jpt handles them. So like any good Teachable Moment™ and life lesson, I’ve turned lemons into lemonade and out of challenge comes innovation: jpt v0.9.6 is now nonplussed by these uncouth JSON files, with their comments and trailing commas. Like water off ducks back, jpt keeps on keepin’ on, unphased.

Finding the Misfits

To find these misfits I wrote made a script jsonParsingReportCSV-macOS. This will attempt to parse the JSON it finds using the json_pp binary that ships from Apple, the results are output as CSV and you can then easily sort it to spot troublemakers.

I’ve uploaded a couple annotated version of those searches along with the kind of deviation they exhibit: Malformed-macOS-10.15.csv and Malformed-macOS-11.1.csv

What’s their damage, man?

The two most prevalent kinds of JSON malformation I found in macOS were: trailing commas and comments. Trailing commas meaning the last item in an array or object should not have a comma after it. While Javascript has always allowed trailing commas in array literals and later in object literals (ES5), then in function parameters (ES2017), JSON has never allowed trailing commas nor comments. Neither are in the JSON spec (RFC8259).

The biggest group of offenders were the USD generated JSONs within USDKit (a private framework) and Model I/O. USD is interesting because of it’s lineage and purpose: It’s Pixar’s open source software for combining multiple 3D elements within a common scene. Or as Pixar’s page put it: “Universal Scene Description (USD) is the first publicly available software that addresses the need to robustly and scalably interchange and augment arbitrary 3D scenes that may be composed from many elemental assets.” Keep an eye on this framework 🕶

JSON Lines (Don’t Do It)

An outlier in the mix is defaultConfig.json within CoreAnalytics.framework, it is what some call “JSON Lines” or “concatenated JSON”, where each line is its own a JSON value or object. While doing some research on JSON Lines I came upon an intriguing comment in this post by rurban:

“The delimiters don’t get in the way, they protect you from MITM and other attacks. It’s a security feature, not a bug.”

Reini Urban on JSON vs. “concatenated JSON”

Taking a look at Reini’s GitHub and the level of his work, I’m going to give his statement some credence. To that end, I really didn’t want to add another output mode nor change query behavior to accommodate JSON Lines. My current compromise is to simply to attempt conversion of a JSON Lines file into an array: tack on commas and ensconce in square brackets. It Just Works™ if every line is truly a self contained JSON object. If you want to be made aware of this, use the -c option so jpt it will “complain” on stderr and exit with code 1.

The Rabbit Hole of CoreAnalytics

Within the CoreAnalytics private framework is defaultConfig.json, it is a collection of 4019 objects currently that describe the events and actions Apple is interested in collecting usage information on. They want to know if you’re actually using all those new features in CarPlay, Reminders, Photos, eGPU App usage, etc… If you want to see all the key names try this: jpt $..addTransform.name /System/Library/PrivateFrameworks/CoreAnalytics.framework/Versions/A/Resources/defaultConfig.json 

Do you actually use this stuff? Apple wants to know if it’s worth their while…

A Comment on JSON Comments

Comments in JSON has inspired debate far and wide, this post by getify sums it up very nicely:

A JSON encoder MUST NOT output comments. A JSON decoder MAY accept and ignore comments.

Douglas Crockford in some old Yahoo group that even Wayback can’t find anymore

So that’s just what jpt does now, it will never encode comments but it will strip out any and all comments it recognizes automatically, no switches needed. Which style comments? All of them (or most of them):

Commentkindlineage
# commentlineshell, Perl, PHP, Python, R, Ruby, MySQL
// commentlineC(99), Javascript, Java, Swift
/* comment */blockC, Java, Javascript, Swift
; commentlineassembly language (ASM)
-- commentlineAda, Applescript, Haskell, Lua, SQL
<!-- comment -->blockXML
*** comment ***linemacOS’s ionodecache.json

If you need a test file you can use my worstJSONEver.json. A most horrible JSON file if there ever was one! jpt soldiers on, use -c to see what it does to clean the input (-i0 for no level indents). It shows the initial parse error and each attempt at recovery.

Fixing the Misfits

There is no fixing, there is only acceptance. Since most of these files are in SIP protected areas of macOS so it will take an act of God, a well filed bug report, for Apple to correct these in a future update. But they usually only act if there’s a demonstrable bug, in this case it’s more of an observation about the condition of the JSON files than a clear issue. If someone is really bothered, they could disable SIP and go in and fix them, however future macOS updates will likely revert these changes. Since I haven’t tried this, who knows, macOS may get ornery and trip OS integrity protections, rendering your system un-bootable, YMMV 🤷‍♂️. The real question is if they impact how macOS works and I’m guessing they don’t. The apps and frameworks which use these off-spec JSON file must be doing something to correct the issues before parsing. Rest assured though, if you attempt to parse one of these files with jpt there will be no issues, it Just Works™. However if you would like to know it did something, use -c to “complain” about comments, commas, JSON Lines, and hard wrapping. It will send those notices to stderr and exit with a status of 1.

jpt: malformed JSON? Bring it on.

So after all this I don’t have a good example page for jpt’s multitude of features and usage (yet) but what I do have is a JSON Power Tool with more “dummy-proof” features than it did before. I’ll take that as a win! Check out the latest release of jpt at my Github.

jpt (JSON Power Tool) is a non-compiled tool that can manipulate and query JSON in a variety of ways, it runs on macOS (10.4 – 11.0) PPC/Intel/Apple Silicon; Linux with jsc; Windows with Linux subsystem for Windows and jsc. It can be used standalone or embedded within your shell scripts. I hope you find it useful. Thanks for reading!

jpt + jamf uapi = backupJamf-scripts

brunerd

Jamf UAPI: JSON Only

I am a creature of habit, no doubt, however sometimes you must get out of your comfort zone. The Jamf Universal API (UAPI) is one such case, it is JSON only and not XM. Those tried and true xpath snippets will no longer work with JSON, in fact what tool do you use with JSON? macOS really doesn’t have a good built-in JSON tool and if your scripts are client side do you really want to have jq as a dependency? Good thing I wrote a JSON parser you can embed in your scripts this summer! In fact, when I finished writing my JSON power tool jpt, I needed to find some practical examples to demonstrate its utility. Looking at the UAPI it’s clear some parts are still a work-in-progress, however the endpoint for scripts is actually really good. It gives you everything in one go almost like a database query. That should made backing up scripts a breeze!

backupJamf-scripts. boom.

If you’ve used the Classic API from Jamf, it is a 1:1 ratio of scripts to API calls: 2 scripts? 2 API calls via curl. 200 scripts? 200 API calls via curl. The new Universal API reduces that down to 1 call to get everything (plus one call to get the token), it’s super fast and I love it. Check out backupJamf-scripts.command in my newly minted jamfTools repo on GitHub for a working demostration of both the Jamf UAPI and jpt’s in-script JSON handling. I hope you like it!

jpt + jamf uapi = scripts downloading scripts

Jamf & FileVault 2: Tips & Tricks (and more)

brunerd

Raiders of the Lost Feature Requests

So there’s this old feature request at Jamf Nation (stop me if you’ve heard this one…) it’s almost 6 years old: Add ability to report on FV2 Recovery Keys (and/or access them via API) In fact, maybe you came here from there, watch out don’t loop! Continue!

The pain point is this: Keys are sent back to Jamf Pro (JSS) but then can only be gotten at manually/interactively through the web interface, not via API nor another method. For cases of mass migration to another JSS it sure would be nice to move those keys over rather than decrypt/re-encrypt. Well, I’ve got a few insights regarding this that I’d like to share that may help. ‘Cuz hey it’s 2020 and we’ve learned that hoarding is just silly.

Firstly, it should be pointed out that neither ye olde “Recovery Key Redirection” payload nor it’s replacement “Recovery Key Escrow” are needed to get keys to the JSS. There is another method and it’s what is used by the built-in “Filevault Encryption” policy payload to get the keys back to your JSS. Jamf references this method in this old script at their GitHub. I revamped the core bits a couple years ago in a (nearly 7 year old) feature request: Manually Edit FileVault 2 Recovery Key

Telling the JSS Your Secrets

The takeaway from that is to realize we have a way to explicitly send keys to the JSS by placing 2 XML files in the /Library/Application Support/JAMF/run folder: file_vault_2_id.xml and file_vault_2_recovery_key.xml. Also note, Jamf has updated the process for the better in the last two years: a jamf recon (or two) is no longer required to send the key and validate it, instead JamfDaemon will send it immediately when both the files are detected. Which is nice, but it’s the subsequent recon validations where we have an opportunity to get grabby.

Cold Lamping, Hard Linking

So here’s the fun part: When recon occurs there’s lots of file traffic in /Library/Application Support/JAMF/tmp all sorts of transient scripts hit this folder. What we can do is make hard links to these files as they come in so when the link is removed in tmp another exists elsewhere and the file remains (just in our new location). EAGrabber.sh does exactly that (and a little bit more)

EAGrabber.sh can be easily modified to narrow it’s focus to the FileVault 2 key only, deleting the rest. What you do with the key is up to you: Send it somewhere else for safe keeping or keep it on device temporarily for a migration to another Jamf console. A script on the new JSS could then put that key on-disk into file_vault_2_recovery_key.xml file which will then import and validate, no decrypt/recrypt necessary. Hope this helps.

Cuidado ¡ Achtung ! Alert

Jamf admins take note: Do you have hard coded passwords in your extension attributes or scripts? If so, then all your scripts are belong to us. Now, go read Obfuscation vs. Encryption from Richard Purves. Read it? OK, now consider what happens if you were to add a routine to capture the output of ps aww along with a hard-linking loop like in EAGraber.sh. If you are passing API credentials from policies via parameter, then ps can capture those parameters and even if you try and obscure them, if we’ve captured the script we can de-obfuscate them. This is a good reason to be really careful with what your API accounts can do. If you have an API account with Computer record Read rights that gets passed into a script via policy and you use LAPS, then captured API credentials could be used to harvest LAPS passwords via API. Keep this in mind and we’ll see if any meaningful changes will occur in recon and/or the script running process in the future (if you open a ticket you can reference PI-006270 regarding API credentials in the process list). In the meantime make API actions as short lived as possible and cross your fingers that only you, good and noble #MacAdmins read this blog. 🤞

jpt: jamf examples pt. 2

brunerd

Over in the MacAdmins’ #bash channel I saw a I question regarding how to get the Sharing states of Bluetooth devices from system_profiler. The most succinct answer was to awk out the values:

system_profiler SPBluetoothDataType 2> /dev/null | awk '/State: / {print $2}'
Disabled
Disabled
Disabled

If you are using this for a Jamf Extension Attribute, I suppose it’ll do if you never want to allow any of them to be Enabled, but what if Internet Sharing was OK but not File Sharing? How would you match your Smart Group to multiple lines of unlabeled values? How would you match the first two but not the last two… and what if there was another USB Bluetooth device, that would add extra rows. Hmmm…

The answer for me, outputting the service name and the state on the same line. Since there isn’t a consistent line count from State: going back the service name, using something like grep -B n to include n lines of preceding data isn’t going to work.

      Services:
          Bluetooth File Transfer:
              Folder other devices can browse: ~/Public
              When receiving items: Accept all without warning
              State: Disabled
          Bluetooth File Exchange:
              Folder for accepted items: ~/Downloads
              When other items are accepted: Save to location
              When receiving items: Accept all without warning
              State: Disabled
          Bluetooth Internet Sharing:
              State: Disabled

So you know what I say the answer to that is? That’s right, jpt the JSON Power Tool! It can parse the -json output from system_profiler in a more structured way and it allows for the discovery of as many applicable Bluetooth devices might be on the system.

Here’s a sample run with Internet Sharing turned On as well as Bluetooth Sharing turned On

file_browsing: disabled
object_push: enabled
internet_sharing: enabled

File Browsing is set to “Never Allow” but File Receiving is in the affirmative (Accept and Open, Accept and Save, or Ask). The addition of labels gives us the ability to create a Smart Group to match specific services like “file_browing: enabled” or any other combination thereof (perhaps internet_sharing should always be enabled, who am I to say what your requirements are!).

About the jpt

The JSON Power Tool (jpt) is a parser/manipulator for JSON documents written in Javascript and shell and can run standalone or embedded in your scripts bash or zsh and all the way back to OS X 10.4 Tiger! Check it out at: https://github.com/brunerd/jpt

jpt: jamf examples

brunerd

jpt has some practical applications for the Jamf admin

Sanitizing Jamf Reports

Let’s say you’ve exported an Advanced Search, it’s got some interesting data you’d like to share, however there is personal data in it. Rather than re-running the report, why not blank out or remove those fields?

Here’s a sample file: advancedcomputersearch-2-raw.json

This an excerpt from one of the computer record:

{
"name": "Deathquark",
"udid": "2ca8977b-05a1-4cf0-9e06-24c4aa8115bc",
"Managed": "Managed",
"id": 2,
"Computer_Name": "Deathquark",
"Last_Inventory_Update": "2020-11-04 22:01:09",
"Total_Number_of_Cores": "8",
"Username": "Professor Frink",
"FileVault_2_Status": "Encrypted",
"JSS_Computer_ID": "2",
"Number_of_Available_Updates": "1",
"Model_Identifier": "MacBookPro16,1",
"Operating_System": "Mac OS X 10.15.7",
"Model": "MacBook Pro (16-inch, 2019)",
"MAC_Address": "12:34:56:78:9A:BC",
"Serial_Number": "C02K2ND8CMF1",
"Email_Address": "frink@hoyvin-glavin.com",
"IP_Address": "10.0.1.42",
"FileVault_Status": "1/1",
"Processor_Type": "Intel Core i9",
"Processor_Speed_MHz": "2457",
"Total_RAM_MB": "65536"
}

Now let’s say the privacy standards for this place is GDPR on steroids and all personally identifiable information must be removed, including serials, IPs, UUIDs, almost everything (but you don’t want to run the report again because it took ages to get the output)!

Here’s what that command would look like: jpt -o replace -v '"REDACTED"' -p '$["advanced_computer_search"]["computers"][*]["name","udid","Username","Computer_Name","MAC_Address","Serial_Number","Email_Address","IP_Address"]' ./advancedcomputersearch-2-raw.json

Here’s what that same computer looks like in the resulting output (as well as all others in the document):

  {
    "name": "REDACTED",
    "udid": "REDACTED",
    "Managed": "Managed",
    "id": 2,
    "Computer_Name": "REDACTED",
    "Architecture_Type": "i386",
    "Make": "Apple",
    "Service_Pack": "",
    "Last_Inventory_Update": "2020-11-04 22:01:09",
    "Active_Directory_Status": "Not Bound",
    "Total_Number_of_Cores": "8",
    "Username": "REDACTED",
    "FileVault_2_Status": "Encrypted",
    "JSS_Computer_ID": "254",
    "Number_of_Available_Updates": "1",
    "Model_Identifier": "MacBookPro16,1",
    "Operating_System": "Mac OS X 10.15.7",
    "Model": "MacBook Pro (16-inch, 2019)",
    "MAC_Address": "REDACTED",
    "Serial_Number": "REDACTED",
    "Email_Address": "REDACTED",
    "IP_Address": "REDACTED",
    "FileVault_Status": "1/1",
    "Processor_Type": "Intel Core i9",
    "Processor_Speed_MHz": "2457",
    "Total_RAM_MB": "65536"
  }

What we did was use the -o replace operation with a -v <value> of the JSON string "REDACTED" to all the paths matched by the JSONPath union expression (the comma separated property names in brackets) of the -p option. JSON Pointer can only act on one value at a time, this is where JSONPath can save you time and really shines.

jpt is fast because WebKit’s JavascriptCore engine is fast, for instance there is a larger version of that search that has 15,999 computers, it took only 11 seconds to redact every record.

jpt -l $.advanced_computer_search.computers ./advancedcomputersearch.json
15999

time jpt -o replace -v '"REDACTED"' -p '$.advanced_computer_search.computers[*]["name","udid","Username","Computer_Name","MAC_Address","Serial_Number","Email_Address","IP_Address"]' ./advancedcomputersearch.json > /dev/null 

11.14s user 0.35s system 104% cpu 11.030 total

Put Smart Computer Groups on a diet

When you download Smart Groups via the API, you will also get an array of all the computers objects that match at that moment in time. If you just want to back up the logic or upload to another system, you don’t want all those computers in there.

Sample file: smartgroup-1-raw.json

{
  "computer_group": {
    "id": 12,
    "name": "All 10.15.x Macs",
    "is_smart": true,
    "site": {
      "id": -1,
      "name": "None"
    },
    "criteria": [
      {
        "name": "Operating System Version",
        "priority": 0,
        "and_or": "and",
        "search_type": "like",
        "value": "10.15",
        "opening_paren": false,
        "closing_paren": false
      }
    ],
    "computers": [
      {
        "id": 1,
        "name": "mac1",
        "mac_address": "12:34:56:78:9A:BC",
        "alt_mac_address": "12:34:56:78:9A:BD",
        "serial_number": "Z18D132XJTQD"
      },
      {
        "id": 2,
        "name": "mac2",
        "mac_address": "12:34:56:78:9A:BE",
        "alt_mac_address": "12:34:56:78:9A:BF",
        "serial_number": "Z39VM86X01MZ"
      }
    ]
  }
}

Lets’ remove those computers with jpt and the JSON Patch remove operartion:
jpt -o remove -p '$.computer_group.computers' ./smartgroup-1-raw.json

Since the target is a single property name, JSON Pointer can also be used:
jpt -o remove -p /computer_group/computers ./smartgroup-1-raw.json

{
  "computer_group": {
    "id": 12,
    "name": "All 10.15.x Macs",
    "is_smart": true,
    "site": {
      "id": -1,
      "name": "None"
    },
    "criteria": [
      {
        "name": "Operating System Version",
        "priority": 0,
        "and_or": "and",
        "search_type": "like",
        "value": "10.15",
        "opening_paren": false,
        "closing_paren": false
      }
    ]
  }
}

Further Uses

Using only JSON Patch replace and remove operations this JSON could have its id removed to prep it for an API POST on another JSS to create a new group or the name and value could be modified in a looping script to create multiple JSON files for every macOS version. The jpt is flexible enough to handle most anything you throw at it, whether you are using the standalone version or have embedded it in your scripts.

Stop by the jpt GitHub page to get your copy

secret origins: the jpt

brunerd

On building a JSON tool for macOS without using Python, perl, or Ruby.

In my work as a Macintosh engineer and administrator I’ve noticed macOS has lacked a bundled tool for working with JSON at the command line. Where XML has its xpath, if your shell script needs some JSON chops, it’ll require an external binary like jq or something else scripted in Python, Ruby or perl using their JSON modules. The problem is, those runtimes have been slated for removal from a future macOS. So I took that as challenge to devise a method to query and modify JSON data within shell scripts, that didn’t use one of those deprecated scripting runtimes and didn’t require an external binary dependency either. Could I achieve robust and native JSON parsing on a Mac by simply “living off the land”?

Why not just re-install the runtimes when Apple deprecates them?

“Why limit yourself like this? Just re-install the runtimes and move on”, you may ask. Well, I’d like to think that limitations can inspire creativity but we should also consider there may be some other reasons why Apple is discontinuing the inclusion of those runtimes. Some may say, “Apple Silicon + macOS 11.0 is the perfect time for them to clean house”, to which I’d have to agree that’s a very good reason and likely a factor. Others could say they are looking to tighten the screws to keep out unsigned code: maybe, they do like to glue things shut! But really, I think it’s more akin to the web-plugins of yore like Java and Flash. Apple does not want to be the conduit for deploying 3rd party party runtimes which increase the attack surface of macOS. This seems like the most reasonable of the explanations. So, if you accept that Apple is attempting to reduce attack surface, why increase it by re-installing Python, Ruby, or perl, just so a transient script (like a Jamf Extension Attribute) can parse a JSON file? My answer to that, is you don’t. You play the hand you’re dealt. Game on!

Looking for truffles (in a very small back yard)

Despite having another project (shui) that can output and invoke Applescript from within a shell scripts for generating user interfaces, I definitely knew that Applescript was not the way to go. Apple however, added to the languages Open Scripting Architecture (OSA) supports back in 2014 with OS X Yosemite (10.10), they added Javascript along with a bridge to the Cocoa Objective-C classes and they called it JXA: Javascript for Automation. This seemed like a promising place, so I started playing with osascript and figured out how to load files and read /dev/stdin using JXA, and while looking for an answer for garbled input from stdin I came upon a Japanese blog that mentioned jsc the JavaScriptCore binary which resides in the /System/Library/Frameworks/ JavaScriptCore.framework. Arigato! Pay dirt! 🤑 jsc does exactly what we need it to do: It can interpret Javascript passed as an argument, can access the filesystem and read from /dev/stdin, and best of all is in non-Private System level Framework that exists all the way back to OS X 10.4! Just the kind of foundation on which to build the tool.

Homesteading jsc

The existence of jsc goes back all the way to OS X Tiger and it’s functionality has evolved over the years. In order to have a consistent experience in jpt from macOS 10.4 – 11.x+ a few polyfills had to be employed for missing functions, along with a few other workarounds regarding file loading, printing and exit codes (or lack thereof). Once those were addressed the jsc proved to be a highly optimized Javascript environment that’s blazingly fast. It spans 13 macOS releases and is even present in many Linux distros out-of-the-box (Ubuntu and CentOS) and can even be run on Windows when the Linux Subsytem is installed.

With the host environment sorted, I began working with the original JSONPath code by Stefan Goessner as the query language. I didn’t know about JSON Pointer yet so this strange beast was all I knew! It worked out really. I went full throttle into developing the “swiss army knife” of JSON tools. I really leaned into the Second System Effect, as described in the Mythical Man-month, it’s when you put every doodad, gizmo and doo-hicky in your 2nd product (my first simple JSON pretty-printer built in JS on jsc). Eventually though, after I reached a feature plateau, I came back around to the address the quirks of the original JSONPath code. I ended up rewriting signifigant chunks of JSONPath and released it as it’s own project: brunerd JSONPath. But I digress, let’s get back to the JSON Power Tool.

jpt: powers and abilities

At it’s most basic, jpt will format or “pretty print” any JSON given. jpt can also handle data retrieval from JSON document using either JSONPath or JSON Pointer syntax. JSONPath, while not a standard, is a highly expressive query language akin to XPath for XML, with poweful features like recursive search, filter expressions, slices, and unions. JSON Pointer on the other hand is narrower in focus, succint and easily expressed, and standardized but it does not offer any of the interogative features that I feel make JSONPath so intriguing. Finally, jpt can also modify values in a JSON document using standardized JSON Patch operations like: add, remove, replace, copy, move, and test, as well as the also standardized JSON Merge Patch operation. Altogether, the jpt can format, retrieve and alter JSON documents using only a bit of outer shell script plus a lot more Javascript on any macOS since 10.4! 😅

Where can I get the jpt?

Stop by the project’s GitHub page at: https://github.com/brunerd/jpt

There you will find the full source and also a minified version of the jpt for inclusion within your shell scripts. Since it’s never compiled you can always peer inside and learn from it, customize it, modify it, or just tinker around with it (usually the best teacher).

Future Plans

There will undoubtedly be continued work on the jpt. Surely there are less than optimal routines, un-idiomatic idioms, edge cases not found, and features yet to be realized. But as far as the core functionality goes though, it’s fairly feature complete in my opinion. Considering that one of my top 10 StrengthsFinder qualities is “Maximizer”, the odds are pretty good, I’ll keep honing the jpt‘s utility, size (smaller), and sharing more articles with examples on the kinds of queries and data alteration operations the jpt can so perform. Stay tuned!

New Projects: jpt and shui, Now Available

brunerd

Between March and October 2020 I had some great ideas for command line Mac utilities the MacAdmin could apprecite and I had the time to devote to their realization. I’m excited to present these two open source projects, available on GitHub: jpt and shui. I hope they can add richness to your shell scripts’ presentation and capabilities without requiring additional external dependancies.

jpt – the “JSON Power Tool” is a Javascript and shell script polyglot that leverages jsc, the JavascriptCore binary that is standard on every Mac since 10.4 and since the jpt is purposefully written in ES5 to maintain maximum compatibility, why yes, this tool does run on both PPC and Intel Macs all the way back to OS X Tiger and then all the way forward to the latest 11.0 macOS Big Sur! Many Linux distros like CentOS and Ubuntu come with jsc pre-installed also, even Windows with the Linux Subsystem installed can run jsc and therefore can run the jpt!

What you can do with the jpt? Query JSON documents using either the simple yet expressive JSONPath syntax or the singular and precise JSON Pointer (RFC6901) syntax. The output mode is JSON but additional creative output modes can render JSONPaths, JSON Pointer paths, or even just the property names with their “constructor” types (try -KC with -J or -R) Textual output can be encoded in a variety of formats (hex/octal/URI encoding, Unicode code points, etc…), data can be modified using both JSON Patch (RFC6902) operations (add, replace, remove, copy, move, test) and also JSON Merge Patch (RFC7386) operations. JSON can be worked with in new ways, try -L for “JSONPath Object Literal” output to see what I mean. Or you simply feed jsc a file to pretty-print (stringify) to /dev/stdout. I’ll be writing more about this one for sure.
Github project page: jpt
Tagged blog posts: scripting/jpt

shui – first-class Applescript dialog boxes in your shell scripts without needing to remember esoteric Applescript phrasings! If you think it’s odd for code to have possessive nouns and are more comfortable in shell, you’re not alone. shui can be embedded in either bash or zsh scripts but it can also output Applescript if you really want to know how the sausage is made or want to embed in your script without shui. Hopefully shui will let you forget those awkward Applescript phrasing and focus on your shell script’s features and functionality. It uses osascript to execute the Applescript and launchctl to invoke osascript in the correct user context so user keyboard layouts are respected (vs. root runs). Check out the project page for demo videos and then give shui a try.
Project page: shui
Tagged blog posts: scripting/shui

Track and Tackle com.apple.macl

brunerd

UPDATE: As of Big Sur (macOS 11) the com.apple.macl extended attribute can be removed using: xattr -rd com.apple.macl <file/folder>

Starting in macOS Catalina, an extended attribute (XA) named com.apple.macl is being added to the files and folder you work with. What does it do? How does it work? When does it get added? Where is the documentation?

All these are good questions, but there’s no official documentation. A good start is this assemblage of articles, here’s the highlights from that page and a few more…

Apple talked about this vaguely in their WWDC 2019 session 701, “Advances in macOS Security“, the Files and Folders fun starts at 21:21 but there’s nothing really about the implementation

Tom Bridge talked about com.apple.macl to the Penn State Macadmins back in August before Catalina’s release. He’s one of the first to notice this XA (and talk about it). Although, he says that curl will add this XA to downloads, it thankfully does not. Most likely it happened when he dragged it to Terminal or performed some other action on it.

Howard Oakley of the prolific Eclectic Light Company blog kicks the tires a bit more and finds some interesting quirks, as he always does.

Jeff Johnson though, really hits the nail on the head in illustrating what’s going on when you simply drag and drop a file or folder into Terminal and com.apple.macl is appended. He dropped this, December 18th and well… the holidays were around the corner and we all had better stuff to do!

It’s 2020 now and time to come back around to this. A very helpful post in the Apple Dev forums from an Apple employee, Quinn “the Eskimo”, decodes the data structure a bit more as well as what some of the conditions for having this XA appended are:

When a user selects a “protected” file or folder in an NSOpenPanel in a non-sandboxed app on Catalina, consent is inferred and the app can access it.
…
It seems that 01 00 is a header of some form and … is a UUID associated with my test app.  I dug into how that UUID is set up and, well, it’s complex, and more of an implementation detail than I care to go into here on DevForums.

https://forums.developer.apple.com/thread/124121

Well, I would have loved the complex implementation detail! But I’ll tell you this: The app UUID in com.apple.macl is unique to EVERY computer. Jeff Johnson was right when he said: “The macl is effectively untraceable“.

However you can begin to see the shape of things when you write a tool to output the UUID in CSV like I have. Behold maclTrack.command

Run it in Terminal, give it files or folders as arguments, it’ll report on them, you can even specify a maximum depth for folders (-d) and silence reporting on items lacking com.apple.macl (-s), pipe it into tee to see the output and write it to a file.

This poor file got clobbered by multiple app UUIDs

Something interesting I noticed from using this tool is seeing the “Header” as Quinn called it, differ from 0100 to 0200. I believe 0200 is for drag and drop operations vs. a regular save which results in 0100.

Again, the app UUIDs you see on the files on your Mac will never be seen on another Mac ever – why? Because UUIDs are meant to be truly unique. Have you read the man page for uuidgen? They ain’t eff’in around!

 The uuidgen command generates a Universally Unique IDentifier (UUID), a 128-bit value guaranteed to be unique over both space and time.

Apple man page for uuidgen

So what’s the point of having a long lived XA that you can’t get rid of and that’s only useful on your local Mac? If it’s a permissions granting XA (the opposite of com.apple.quarantine which imposes restrictions) then why not let the user remove it? In some ways it’s a cross computer file tracking mechanism, albeit “anonymized”. Perhaps allowing it’s removal though (aka writing “nothing” to the XA) also allows for something to be written? This is where I could get in trouble speculating, it’s just a guess.

Perhaps you’ll forgive me that, since I have found a way to clear com.apple.macl without disabling SIP! Zip it. No, not the one spawned Finder’s “Compress” menu item, I mean /usr/bin/zip! I noticed command line zip would obliterate XAs during a project many years ago and it still does my friends! So behold maclTackle.command

This script has considerably less engineering to it because it’s a PoC and I’m really don’t want to take responsibility for something going awry should you try to clear com.apple.macl on your entire Desktop folder (no I haven’t tested that). But it will strip the XAs off a file that’s for sure! NOTE: USE A NON-IMPORTANT FILE with this script (make a duplicate). The script will overwrite the original (don’t like the behavior? It’s a script: Comment it out!)

🎶 Gonna wash that macl right outta my file… 🎶

Apple may come along and close this loophole now that it’s been pointed out or perhaps they won’t since they’d have to publish the modfication? Perhaps they’ll continue to purge “undesirable” binaries from macOS? They say they will stop shipping Python, perl and ruby in macOS some day and you know what xattr is written in? That’s right Python! ¯\_(ツ)_/¯

Alright, come and get it while the gettin’s good! Thanks for reading!