ElasticSearch Optimization Strategies at Scale
At Trestle, the extensive availability, coverage, and accuracy of identity data that we provide and the intelligent algorithms that are applied on top of it to serve the varying needs of our customers for different use cases is what differentiates us from the rest of the competitors.
The identity data is sourced from multiple providers. The core storage layer is an ElasticSearch cluster, which houses the main index containing this pre-processed, normalized, and formatted data to fit our custom mapping. During each ingestion run, around 2 terabytes of raw data, containing around 500 million entities of persons and businesses, is processed. On top of that, more than five billion “person to phone,” “person to business,” and “person to address” linkages are mapped. These datasets’ sheer volume and complexity highlight the massive scale of the data processing efforts.
The API products’ source of truth is this ElasticSearch index. Every request hits this index and serves the requested data queried by a combination of phone, address, name, and a few different parameters. The data ingestion process happens monthly when the raw data files obtained from the providers are processed and stored in the ElasticSearch cluster.
Recently, we took the initiative to optimize the ElasticSearch cluster, be it storage size reduction, cost savings by finding optimizations in the ingestion process, or improving the latency of the search queries. This document details the experiments and the findings of this initiative.
Reducing Total Nodes Count
The total size of the ElasticSearch index is ~1250GB with 36 shards. This means each shard would roughly be around 35GB in size. During the monthly ingestion process, 36 nodes were launched so that each shard would be created in one node. By isolating each shard on a separate node, the cluster avoids resource contention (CPU, memory, disk I/O) during high-throughput ingestion, maximizing parallelism and stability.
ElasticSearch recommends a shard size between 10-50GB. Reducing the total shard count to 24 would result in each shard being ~52GB, which is still around the recommended size. With this change, we would only need 24 nodes to complete the ingestion process at the same speed.
Impact
Reducing the total nodes resulted in thousands of dollars of savings for each ingestion run since the nodes used for the data ingestion process were high-specification instances.
What’s Next?
The primary reason for having a shard-per-node model was to maximize parallelism, but during the ingestion run, each node was not utilized to its fullest. So, there is more room for optimization here. We could start by having a node store two shards, reduce the total nodes to 12, and measure the total ingestion time and error rate. If it’s successful, it will result in even more cost savings, and this method can be continued until a sweet spot for “max shards per node” is reached without increasing the total ingestion time too much.
Reducing Shard Segment Count
Each shard is composed of a number of Lucene segments, which is where the documents are stored. On each search request, every segment is searched to find the relevant document. The lower the number of segments, the faster the search speed.
The main index has around ~30 segments per shard, each size between 5MB-5GB.
ElasticSearch should automatically take care of merging segments once they get too large. In our case, this doesn’t happen because once the ingestion is complete, there are no writes to the index, hence no change in the initial segment count.
Thus, the force_merge operation can also be performed post-ingestion to reduce segment size. This is an expensive operation and takes a while to complete.
The benefits of this operation with respect to size and performance are as follows,
- tiq_1609: without merging segments
- tiq_1609_kw: after merging segments
Merging segments resulted in higher throughput and ~25% lower p99 and p90 response times in all the tests.
Impact
This improvement helped with a critical milestone, aligning directly with our prioritization of low-latency performance for API endpoints.
What’s Next
Reduce the number of segments to 1 and run the same tests to see if it improves the performance further.
Index Mapping Modification
How the index data should be structured, its relationships, individual field types, index configurations, etc., is set in the mapping config during index creation. The goal was to analyze this mapping and find room for optimization. The mapping for “address” and related data looked like:
"ad" : {
"type" : "nested",
"properties" : {
"co" : {
"type" : "text",
"index" : false
},
"ct" : {
"type" : "text"
},
"dp" : {
"type" : "text"
},
"ds" : {
"type" : "text",
"index" : false
},
"f" : {
"type" : "text"
},
"fad" : {
"type" : "keyword"
}
...
}
}As seen above, a combination of text and keyword fields is used, and there are primary differences between these two types.
Text fields are analyzed during indexing, involving tokenization and normalization to enable partial matches and relevance-based searches, but this creates multiple inverted index entries, increasing storage. Keyword fields store values verbatim, skipping analysis, which optimizes them for exact matches, sorting, and aggregations while reducing storage overhead (single inverted index entry per value). Switching to keyword fields eliminates text-processing costs, improves query speed for exact matches, and compresses better for repetitive data but sacrifices partial-text search capabilities.
The search workload involved querying one to five fields at most. These fields were never queried partially. For example, there were no queries for only a part of a phone number, “+11237878791,” like “+11237”.
Although the phone number field was rightly using the keyword field, many other fields within the index were stored as “text” type but never queried for partially. As an example, the “city” field was stored as text type, so values like “New Orleans” will be split into two lowercase terms: [“new,” “Orleans”] (punctuation and spaces trigger splits). These terms are stored in the inverted index as individual tokens, but a query for a city “New” or “Orleans” will never be made.
So, the idea here was to change such fields to a “keyword” type and see if there was any decrease in the total index size.
The Change
Fields with the type “text,” which did not require partial matching, were converted to “keyword.”
Result
There was no significant reduction in the index size; this could be due to the following reasons:
- Data Distribution: Most values already contained only short strings, which didn’t undergo tokenization even as “text” fields.
- Compression Efficiency: ElasticSearch’s Lucene backend aggressively compresses inverted indices. Switching to a keyword might not yield measurable gains unless the dataset is massive or highly repetitive.
- Small ‘Unique Data’ Volume: If fields like the city had few “unique” values (e.g., repeated entries like “Seattle” or “New York”), the inverted index for text (even with tokenization) might have already been compressed efficiently in the old index. So, the relative storage difference between text and keyword is negligible for data with a low uniqueness quotient for these fields.
- doc_values Overhead: One thing that was not attempted was disabling “doc_values.” “Keyword” fields enable doc_values by default (for sorting/aggregations), which store data in a columnar format. If these fields were unused in aggregations, the doc_values added storage parity with the original text inverted index.
Conclusion
This experimentation has confirmed that fine-tuning an ElasticSearch cluster is a layered and nuanced process. Each adjustment must be evaluated against its potential performance benefits and operational costs. Our experiments taught us that a deep understanding of our data’s unique characteristics and access patterns is essential before rolling out optimizations. In cases where data is already efficiently configured, additional modifications may yield only incremental improvements. This tailored approach ensures that we invest our efforts where they make the most measurable impact, aligning performance gains with our broader operational goals.
