Fifteen Thousand Columns

Most of what makes audience data hard is not its size. A hundred and seventy million people is a big table but not an interesting one. What makes it hard is its shape.

We spent time in 2025 working on audience assembly for a global advertising holding company. Their person records carried roughly fifteen thousand discrete behavioural attributes each — is a tennis shoe buyer, is not a tennis shoe buyer, and fourteen thousand nine hundred and ninety-eight others like it. Sparse, in the sense that any given attribute is true of maybe a fifth of the population rather than most of it.

The job was not to filter that table. Filtering is easy. The job was: take an arbitrary combination of attributes, pull back everyone who matches, and then report how many of those people match each of the other fifteen thousand attributes. Build an audience, then immediately learn what else is true about it.

That second half is the whole problem. It is also the thing an analyst most wants, because it is where the unexpected finding lives.

Why the obvious approaches don't work

A column store is superb at aggregating a narrow table. Fifteen thousand columns is not a narrow table, and the usual escape routes are all closed. You cannot pre-aggregate, because you cannot predict which combination someone will ask for. You cannot cache, because every audience definition is unique — nobody in this position runs the same query twice. What remains is scanning, and scanning is the thing you are billed for.

I want to be equally honest about our own side, because this is the part that usually gets left out of a write-up like this. Distributed search does not solve this naively either. We know, because we tried it first. We loaded the data the obvious way and it simply did not work — the underlying index cannot handle that many columns at all. Not slowly. At all.

So the interesting work was not "we pointed our database at it." It was figuring out what to do after the obvious thing failed.

What we actually did

Our chief architect built a representation specific to this shape of data: bitwise fields, serialised substrings, query facets, and multi-valued fields carrying the discrete attributes. Events were modelled as sub-objects with their own predictable discrete values, so that an audience could be defined by attributes and behaviour in the same query rather than in two passes.

None of those four techniques is exotic on its own. The combination is what made a fifteen-thousand-dimension facet tractable, and each of them has since been useful on projects that look nothing like this one.

The property worth understanding is where the cost actually sits. Returning counts across all fifteen thousand attributes does not cost fifteen thousand attributes' worth of work. It costs whatever is true of the people in your set. If five hundred attributes are set across that audience, we count five hundred things. If five thousand are, we count five thousand, and it takes about ten times as long.

Which means sparsity is not a detail of the data, it is the performance model. William put it better than I will: if people were rational about computation they would recognise those as equivalently efficient operations. They are not rational about computation. They are aware of how long they sat there.

What it did

Measured on their data, during the proof of concept:

Between sixty and nine hundred times faster depending on the query, on the workloads we tested.

The number I care about is the third row. Arbitrary attribute and event criteria, retrieve the audience, return statistics across every dimension — on the order of ten seconds. For most teams on most representations that is minutes, and often tens of minutes. It is the difference between a report you request and a question you ask.

Where it stops

This pattern requires discrete values. Booleans are ideal; small enumerated ranges work fine. Wide continuous values we do not have an answer for — they break the underlying index, and no amount of cleverness in the representation fixes that.

You can often bucket a continuous range into discrete chunks and use the pattern again. Sometimes that is the right modelling decision anyway. Sometimes it is a fudge that costs you more than it saves, and I would rather say so than sell you something that will disappoint you in month three.

If your data looks like this

Thousands to tens of thousands of discrete columns, sparse, across a lot of people, where every question is different from the last one — that is a shape we have solved, and we did not solve it by accident.

If your data is that shape, we can show you the same test on your own. If it is not, tell us anyway. The interesting problems have mostly been the ones that did not fit the pattern on the first try.