Writing an efficient object previewer for JavaScript
In an earlier version of DataStation, previews of panel results were converted to JSON and then sliced to the first 100 characters. The point of these previews was to give the user feedback about the result of running a panel.
The code looked like this:
function previewObject(obj: any, length: number): string {
let str = JSON.stringify(obj, null, 2);
if (str.length > length) {
str = str.slice(0, length) + '...';
}
return str;
}
Performance implications
This was an easy enough solution and it worked alright at first. But as soon as you tried to preview large objects (at least a megabyte or so), the entire app would hang while calculating the preview.
This makes sense. It's a huge waste to convert the entire object to a JSON string just to show the first N characters.
My first instinct was to find a custom JSON stringifier library that would allow me to control the recursion depth. But when I couldn't find any, I started think about what "controlling the recursion depth" could mean.
The most obvious solution I could think of was showing a decreasing number of keys and array entries as you recurse through an object. The first version along these lines looked like this:
function previewObject(obj, limit = 10) {
if (!obj) {
return String(obj);
}
// Go down to zero at some point
const nextLimit = (limit / 2 < 1) ? 0 : (limit / 2);
if (!nextLimit) {
return "";
}
if (Array.isArray(obj)) {
const previewedChildren = obj.slice(0, limit).map(c => previewObject(c, nextLimit));
return '[ ' + previewedChildren.join(', ') + ' ]';
}
if (typeof obj === 'object') {
const keys = Object.keys(obj).slice(0, limit);
const previewedChildren = keys.map(k => '"' + k + '": ' + previewObject(obj[k], nextLimit));
return '{ ' + previewedChildren.join(', ') + ' }';
}
let str = obj;
if (typeof obj === 'string') {
str = '"' + str + '"';
}
return String(obj).slice(0, 200);
}
Some key aspects of this are that we use String(thing)
rather than thing.toString()
so that if
thing
is ever null
or
undefined
it will be properly stringified (and not
throw an exception).
We also need to check if obj
is an array
before checking if it is an object since all arrays
are objects in JavaScript. And we don't care about looking
at hasOwnProperty
in this case because the objects
we're previewing in DataStation are always simple data objects.
One nice property of this implementation
(unlike JSON.stringify
) is that this is circular-object
safe. If you pass a circular object to JSON.stringify
it will throw an exception. This will not, since the number of keys
looked at when recurse always goes to zero.
Finally, the most important part is that we're severely limiting the
number of keys we iterate over. If we had no limits this code would
be much slower than JSON.stringify
(which is
surprisingly fast). But again, this preview only needs to show a
small subset of the object being previewed. So showing only a few
keys is a great tradeoff.
When I replaced the JSON.stringify().slice()
implementation with the custom algorithm, there was an immediate
performance improvement. DataStation could handle many tens of
megabytes of data loaded into the app now.
Getting more user-friendly
After getting this custom preview working, I wanted to make it more friendly. I wanted to have key-value and array elements show up on a newline if the top-level object was an array or object. This would make better use of the horizontal space available in the preview box in DataStation.
I also wanted to show that keys, elements, and cutoff string values were modified if they indeed were. (It isn't very important that the output becomes not-JSON.)
Here is the current implementation (I won't say final since it's likely there'll be even more improvements as time goes on):
function unsafePreviewArray(
obj: any,
nKeys: number,
stringMax: number,
nextNKeys: number,
prefixChar: string,
joinChar: string
) {
const keys = obj.slice(0, nKeys);
const childPreview = keys.map(
(o: string) => prefixChar + unsafePreview(o, nextNKeys, stringMax)
);
if (obj.length > nKeys) {
childPreview.push(prefixChar + '...');
}
return ['[', childPreview.join(',' + joinChar), ']'].join(joinChar);
}
function unsafePreviewObject(
obj: any,
nKeys: number,
stringMax: number,
nextNKeys: number,
prefixChar: string,
joinChar: string
) {
const keys = Object.keys(obj);
keys.sort();
const firstKeys = keys.slice(0, nKeys);
const preview: Array = [];
firstKeys.forEach((k) => {
const formattedKey = `"${k.replaceAll('"', '\\"')}"`;
preview.push(
prefixChar +
formattedKey +
': ' +
unsafePreview(obj[k], nextNKeys, stringMax)
);
});
if (keys.length > nKeys) {
preview.push(prefixChar + '...');
}
return ['{', preview.join(',' + joinChar), '}'].join(joinChar);
}
function unsafePreview(
obj: any,
nKeys: number,
stringMax: number,
topLevel = false
): string {
if (!obj) {
return String(obj);
}
// Decrease slightly slower than (nKeys / 2) each time
const nextNKeys = nKeys < 1 ? 0 : Math.floor(nKeys * 0.6);
const joinChar = topLevel ? '\n' : ' ';
const prefixChar = topLevel ? ' ' : '';
if (Array.isArray(obj)) {
return unsafePreviewArray(
obj,
nKeys,
stringMax,
nextNKeys,
prefixChar,
joinChar
);
}
if (typeof obj === 'object') {
return unsafePreviewObject(
obj,
nKeys,
stringMax,
nextNKeys,
prefixChar,
joinChar
);
}
let res = String(obj).slice(0, stringMax);
if (String(obj).length > stringMax) {
res += '...';
}
if (typeof obj === 'string' && !topLevel) {
res = `"${res.replace('"', '\\"')}"`;
}
return res;
}
export function previewObject(obj: any, nKeys = 20, stringMax = 200): string {
try {
return unsafePreview(obj, nKeys, stringMax, true);
} catch (e) {
console.error(e);
return String(obj).slice(0, stringMax);
}
}
Which ends up producing previews that look like this:
Not too bad!
If you have a better solution or find a major issue, I'd love to hear about it!
Share
Have you ever wanted to preview large JavaScript objects? Have you run into performance issues trying to do so? Here's a walkthrough of a few ways to do so and how DataStation now does it efficiently and usefullyhttps://t.co/ekdqP2c45F pic.twitter.com/22l6ZY1DKa
— DataStation (@multiprocessio) July 15, 2021
With questions, criticism or ideas, email or Tweet me.
Also, check out DataStation and dsq.