{"id":901,"date":"2020-12-17T18:56:43","date_gmt":"2020-12-17T23:56:43","guid":{"rendered":"https:\/\/www.brunerd.com\/blog\/?p=901"},"modified":"2020-12-17T23:53:34","modified_gmt":"2020-12-18T04:53:34","slug":"jpt-is-unphased-by-macoss-malformed-json","status":"publish","type":"post","link":"https:\/\/www.brunerd.com\/blog\/2020\/12\/17\/jpt-is-unphased-by-macoss-malformed-json\/","title":{"rendered":"jpt is unphased by macOS’s malformed JSON"},"content":{"rendered":"\n

A funny thing happened on the way to writing a new blog entry. Instead of finding an example JSON file that ships with macOS, I stumbled upon a slow creep<\/a> of malformed JSON files that started in OS X El Capitan and continues into Big Sur. While I can’t do much about deviant JSON files in macOS, I can<\/em> control how jpt<\/a> handles them. So like any good Teachable Moment\u2122 and life lesson, I’ve turned lemons into lemonade and out of challenge comes innovation: jpt v0.9.6<\/a> is now nonplussed by these uncouth JSON files, with their comments and trailing commas. Like water off ducks back, jpt keeps on keepin’ on, unphased.<\/p>\n\n\n\n

Finding the Misfits<\/h4>\n\n\n\n

To find these misfits I wrote made a script jsonParsingReportCSV-macOS<\/a><\/strong>. This will attempt to parse the JSON it finds using the json_pp binary that ships from Apple, the results are output as CSV and you can then easily sort it to spot troublemakers.<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

I’ve uploaded a couple annotated version of those searches along with the kind of deviation they exhibit: Malformed-macOS-10.15.csv<\/a><\/strong> and Malformed-macOS-11.1.csv<\/strong><\/a><\/p>\n\n\n\n

What’s their damage, man?<\/h4>\n\n\n\n

The two most prevalent kinds of JSON malformation I found in macOS were: trailing commas and comments. Trailing commas meaning the last item in an array or object should not have a comma after it. While Javascript<\/em> has always allowed trailing commas<\/a> in array literals and later in object literals (ES5), then in function parameters (ES2017), JSON has never<\/em> allowed trailing commas nor comments. Neither are in the JSON spec (RFC8259<\/a>).<\/p>\n\n\n\n

The biggest group of offenders were the USD generated<\/a> JSONs within USDKit (a private framework) and Model I\/O<\/a>. USD is interesting because of it’s lineage and purpose: It’s Pixar’s open source software for combining multiple 3D elements within a common scene. Or as Pixar’s page put it<\/a>: “Universal Scene Description (USD) is the first publicly available software that addresses the need to robustly and scalably interchange and augment arbitrary 3D scenes that may be composed from many elemental assets.<\/em>” Keep an eye<\/em> on this framework \ud83d\udd76<\/p>\n\n\n\n

JSON Lines (Don’t Do It<\/a>)<\/h4>\n\n\n\n

An outlier in the mix is defaultConfig.json<\/a> within CoreAnalytics.framework, it is what some call “JSON Lines<\/a>” or “concatenated JSON”, where each line is its own a JSON value or object. While doing some research on JSON Lines I came upon an intriguing comment in this post<\/a> by rurban<\/a>: <\/p>\n\n\n\n

“The delimiters don’t get in the way, they protect you from MITM and other attacks. It’s a security feature, not a bug.” <\/p>Reini Urban on JSON vs. “concatenated JSON”<\/cite><\/blockquote>\n\n\n\n

Taking a look at Reini’s GitHub<\/a> and the level of his work, I’m going to give his statement some credence. To that end, I really didn’t want to add another output mode nor change query behavior to accommodate JSON Lines. My current compromise is to simply to attempt conversion of a JSON Lines file into an array: tack on commas and ensconce in square brackets. It Just Works\u2122 if every line is truly a self contained JSON object. If you want to be made aware<\/em> of this, use the -c option so jpt it will “complain” on stderr and exit with code 1.<\/p>\n\n\n\n

The Rabbit Hole of CoreAnalytics<\/h4>\n\n\n\n

Within the CoreAnalytics<\/strong> private framework is defaultConfig.json<\/a>, it is a collection of 4019 objects currently that describe the events and actions Apple is interested in collecting usage information on. They want to know if you’re actually using all those new features in CarPlay, Reminders, Photos, eGPU App usage, etc… If you want to see all the key names try this: jpt $..addTransform.name \/System\/Library\/PrivateFrameworks\/CoreAnalytics.framework\/Versions\/A\/Resources\/defaultConfig.json <\/code><\/p>\n\n\n\n

\"\"<\/a>
Do you actually<\/em> use this stuff? Apple wants to know if it’s worth their while…<\/figcaption><\/figure>\n\n\n\n

A Comment on JSON Comments<\/h4>\n\n\n\n

Comments in JSON has inspired debate far and wide, this post<\/a> by getify sums it up very nicely: <\/p>\n\n\n\n

A JSON encoder MUST NOT output comments. A JSON decoder MAY accept and ignore comments.<\/strong><\/p>Douglas Crockford in some old Yahoo group that even Wayback can’t find anymore<\/cite><\/blockquote>\n\n\n\n

So that’s just what jpt<\/a> does now, it will never<\/em> encode comments but it will strip out any and all<\/span> comments it recognizes automatically, no switches needed. Which style comments<\/a>? All of them<\/span> (or most of them):<\/p>\n\n\n\n

Comment<\/td>kind<\/td>lineage<\/td><\/tr>
# comment<\/code><\/td>line<\/td>shell, Perl, PHP, Python, R, Ruby, MySQL<\/td><\/tr>
\/\/ comment<\/code><\/td>line<\/td>C(99), Javascript, Java, Swift<\/td><\/tr>
\/* comment *\/<\/code><\/td>block<\/td>C, Java, Javascript, Swift<\/td><\/tr>
; comment<\/code><\/td>line<\/td>assembly language (ASM)<\/td><\/tr>
-- comment<\/code><\/td>line<\/td>Ada, Applescript, Haskell, Lua, SQL<\/td><\/tr>
<!-- comment --><\/code><\/td>block<\/td>XML<\/td><\/tr>
*** comment ***<\/code><\/td>line<\/td>macOS’s ionodecache.json<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

If you need a test file you can use my worstJSONEver.json<\/a><\/strong>. A most horrible JSON file if there ever was one! jpt soldiers on, use -c to see what it does to clean the input (-i0 for no level indents). It shows the initial parse error and each attempt at recovery.<\/p>\n\n\n\n

\"\"<\/a><\/figure><\/div>\n\n\n\n

Fixing the Misfits<\/h4>\n\n\n\n

There is no fixing, there is only acceptance. Since most of these files are in SIP protected areas of macOS so it will take an act of God<\/s>, a well filed bug report, for Apple to correct these in a future update. But they usually only act if there’s a demonstrable bug, in this case it’s more of an observation about the condition of the JSON files than a clear issue. If someone is really bothered, they could disable SIP and go in and fix them, however future macOS updates will likely revert these changes. Since I haven’t tried this, who knows, macOS may get ornery and trip OS integrity protections, rendering your system un-bootable, YMMV \ud83e\udd37\u200d\u2642\ufe0f. The real question is if they impact how macOS works and I’m guessing they don’t. The apps and frameworks which use these off-spec JSON file must be doing something to correct the issues before parsing. Rest assured though, if you attempt to parse one of these files with jpt<\/a> there will be no issues, it Just Works\u2122. However if you would like to know it did something, use -c<\/strong> to “complain” about comments, commas, JSON Lines, and hard wrapping. It will send those notices to stderr and exit with a status of 1.<\/p>\n\n\n\n

\"\"<\/a><\/figure><\/div>\n\n\n\n

jpt: malformed JSON? Bring it on.<\/h4>\n\n\n\n

So after all this I don’t have a good example page for jpt’s multitude of features and usage (yet) but<\/em> what I do have is a JSON Power Tool with more “dummy-proof” features than it did before. I’ll take that as a win! Check out the latest release<\/a> of jpt<\/a> at my Github.<\/p>\n\n\n\n

jpt<\/a> (JSON Power Tool) is a non-compiled tool that can manipulate and query JSON in a variety of ways,<\/em> it runs on macOS (10.4 – 11.0) PPC\/Intel\/Apple Silicon; Linux with jsc; Windows with Linux subsystem for Windows and jsc. It can be used standalone or embedded within your shell scripts. I hope you find it useful. Thanks for reading!<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

A funny thing happened on the way to writing a new blog entry. Instead of finding an example JSON file that ships with macOS, I stumbled upon a slow creep of malformed JSON files that started in OS X El Capitan and continues into Big Sur. While I can’t do much about deviant JSON files […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3,37,4],"tags":[],"class_list":["post-901","post","type-post","status-publish","format-standard","hentry","category-apple","category-jpt","category-os-x"],"_links":{"self":[{"href":"https:\/\/www.brunerd.com\/blog\/wp-json\/wp\/v2\/posts\/901","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.brunerd.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.brunerd.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.brunerd.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.brunerd.com\/blog\/wp-json\/wp\/v2\/comments?post=901"}],"version-history":[{"count":17,"href":"https:\/\/www.brunerd.com\/blog\/wp-json\/wp\/v2\/posts\/901\/revisions"}],"predecessor-version":[{"id":930,"href":"https:\/\/www.brunerd.com\/blog\/wp-json\/wp\/v2\/posts\/901\/revisions\/930"}],"wp:attachment":[{"href":"https:\/\/www.brunerd.com\/blog\/wp-json\/wp\/v2\/media?parent=901"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.brunerd.com\/blog\/wp-json\/wp\/v2\/categories?post=901"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.brunerd.com\/blog\/wp-json\/wp\/v2\/tags?post=901"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}