Writing an efficient object previewer for JavaScript

What is DataStation?DataStation is an open-source data IDE for developers. It allows you to easily build graphs and tables with data pulled from SQL databases, logging databases, metrics databases, HTTP servers, and all kinds of text and binary files. Need to join or munge data? Write embedded scripts as needed in Python, JavaScript, Ruby, R, or Julia. All in one application.

In an earlier version of DataStation, previews of panel results were converted to JSON and then sliced to the first 100 characters. The point of these previews was to give the user feedback about the result of running a panel.

JSON.stringify()-based preview

The code looked like this:

function previewObject(obj: any, length: number): string {
  let str = JSON.stringify(obj, null, 2);
  if (str.length > length) {
    str = str.slice(0, length) + '...';
  }
  return str;
}

Performance implications

This was an easy enough solution and it worked alright at first. But as soon as you tried to preview large objects (at least a megabyte or so), the entire app would hang while calculating the preview.

This makes sense. It's a huge waste to convert the entire object to a JSON string just to show the first N characters.

My first instinct was to find a custom JSON stringifier library that would allow me to control the recursion depth. But when I couldn't find any, I started think about what "controlling the recursion depth" could mean.

The most obvious solution I could think of was showing a decreasing number of keys and array entries as you recurse through an object. The first version along these lines looked like this:

function previewObject(obj, limit = 10) {
  if (!obj) {                                                            
    return String(obj);
  }           

  // Go down to zero at some point                                       
  const nextLimit = (limit / 2 < 1) ? 0 : (limit / 2);
  if (!nextLimit) {
    return "";
  }

  if (Array.isArray(obj)) {
    const previewedChildren = obj.slice(0, limit).map(c => previewObject(c, nextLimit));
    return '[ ' + previewedChildren.join(', ') + ' ]';
  }

  if (typeof obj === 'object') {
    const keys = Object.keys(obj).slice(0, limit);
    const previewedChildren = keys.map(k => '"' + k + '": ' + previewObject(obj[k], nextLimit));
    return '{ ' + previewedChildren.join(', ') + ' }';
  }

  let str = obj;
  if (typeof obj === 'string') {
    str = '"' + str + '"';
  }

  return String(obj).slice(0, 200);
}

Some key aspects of this are that we use String(thing) rather than thing.toString() so that if thing is ever null or undefined it will be properly stringified (and not throw an exception).

We also need to check if obj is an array before checking if it is an object since all arrays are objects in JavaScript. And we don't care about looking at hasOwnProperty in this case because the objects we're previewing in DataStation are always simple data objects.

One nice property of this implementation (unlike JSON.stringify) is that this is circular-object safe. If you pass a circular object to JSON.stringify it will throw an exception. This will not, since the number of keys looked at when recurse always goes to zero.

Finally, the most important part is that we're severely limiting the number of keys we iterate over. If we had no limits this code would be much slower than JSON.stringify (which is surprisingly fast). But again, this preview only needs to show a small subset of the object being previewed. So showing only a few keys is a great tradeoff.

When I replaced the JSON.stringify().slice() implementation with the custom algorithm, there was an immediate performance improvement. DataStation could handle many tens of megabytes of data loaded into the app now.

Getting more user-friendly

After getting this custom preview working, I wanted to make it more friendly. I wanted to have key-value and array elements show up on a newline if the top-level object was an array or object. This would make better use of the horizontal space available in the preview box in DataStation.

I also wanted to show that keys, elements, and cutoff string values were modified if they indeed were. (It isn't very important that the output becomes not-JSON.)

Here is the current implementation (I won't say final since it's likely there'll be even more improvements as time goes on):

function unsafePreviewArray(
  obj: any,
  nKeys: number,
  stringMax: number,
  nextNKeys: number,
  prefixChar: string,
  joinChar: string
) {
  const keys = obj.slice(0, nKeys);
  const childPreview = keys.map(
    (o: string) => prefixChar + unsafePreview(o, nextNKeys, stringMax)
  );
  if (obj.length > nKeys) {
    childPreview.push(prefixChar + '...');
  }
  return ['[', childPreview.join(',' + joinChar), ']'].join(joinChar);
}

function unsafePreviewObject(
  obj: any,
  nKeys: number,
  stringMax: number,
  nextNKeys: number,
  prefixChar: string,
  joinChar: string
) {
  const keys = Object.keys(obj);
  keys.sort();
  const firstKeys = keys.slice(0, nKeys);
  const preview: Array = [];
  firstKeys.forEach((k) => {
    const formattedKey = `"${k.replaceAll('"', '\\"')}"`;
    preview.push(
      prefixChar +
        formattedKey +
        ': ' +
        unsafePreview(obj[k], nextNKeys, stringMax)
    );
  });

  if (keys.length > nKeys) {
    preview.push(prefixChar + '...');
  }

  return ['{', preview.join(',' + joinChar), '}'].join(joinChar);
}

function unsafePreview(
  obj: any,
  nKeys: number,
  stringMax: number,
  topLevel = false
): string {
  if (!obj) {
    return String(obj);
  }

  // Decrease slightly slower than (nKeys / 2) each time
  const nextNKeys = nKeys < 1 ? 0 : Math.floor(nKeys * 0.6);
  const joinChar = topLevel ? '\n' : ' ';
  const prefixChar = topLevel ? '  ' : '';

  if (Array.isArray(obj)) {
    return unsafePreviewArray(
      obj,
      nKeys,
      stringMax,
      nextNKeys,
      prefixChar,
      joinChar
    );
  }

  if (typeof obj === 'object') {
    return unsafePreviewObject(
      obj,
      nKeys,
      stringMax,
      nextNKeys,
      prefixChar,
      joinChar
    );
  }

  let res = String(obj).slice(0, stringMax);
  if (String(obj).length > stringMax) {
    res += '...';
  }

  if (typeof obj === 'string' && !topLevel) {
    res = `"${res.replace('"', '\\"')}"`;
  }

  return res;
}

export function previewObject(obj: any, nKeys = 20, stringMax = 200): string {
  try {
    return unsafePreview(obj, nKeys, stringMax, true);
  } catch (e) {
    console.error(e);
    return String(obj).slice(0, stringMax);
  }
}

Which ends up producing previews that look like this:

More efficient preview

Not too bad!

If you have a better solution or find a major issue, I'd love to hear about it!

Share

Have you ever wanted to preview large JavaScript objects? Have you run into performance issues trying to do so? Here's a walkthrough of a few ways to do so and how DataStation now does it efficiently and usefullyhttps://t.co/ekdqP2c45F pic.twitter.com/22l6ZY1DKa

— DataStation (@multiprocessio) July 15, 2021