As customers evaluate us (or any data vendor), one key question is how to sample a test file to understand the impact of Trestle. We ask our customers a few related questions to ensure everyone is aligned during this evaluation phase, including:
- What does success look like with Trestle’s data for the pain point or use case? What are realistic expectations?
- What disposition data is available for correlation analysis?
- What are the records being tested? How recent are they, and how were they selected?
- How is the analysis performed? Is it a rule-based system, a machine-learning model, or something else?
What Does Success Look Like? #
First and foremost, be clear about the goals and make sure there are realistic expectations.
It is common to hear that customers want to reduce bad leads or sign-ups by 50%. This seems straightforward, but what if achieving this goal means rejecting 5% good leads as well? This happens because any rule selected or machine learning threshold picked to weed out bad records will invariably remove some good ones as well. Is that acceptable for the customer, and do they understand those trade-offs from the outset?
Another similar scenario is customers want to improve their conversion rates when they are outbound dialing. We can help reduce and remove bad phone numbers, which will have a direct impact on conversions based on the dials made only to good numbers. However, we cannot determine whether the call will be answered by the subscriber or go to voicemail, as several other factors contribute to these outcomes. That is not a service that we provide, and confusing our data attributes as a signal of intent will always result in a failed test, wasting time and resources.
Thus, understanding the goals, knowing the capabilities and pain points we can solve, and being aligned on this ensures that the test measures the right metrics and, irrespective of the eventual results, is an accurate measure of how well the pain point is being resolved for the customer.
Disposition Data #
Disposition data is a sample of records where the customer already knows the outcome — whether each record is good or bad. This known data allows for correlation analysis, comparing Trestle’s data attribute responses to the disposition labels. The goal is to evaluate how accurately Trestle can identify bad sign-ups or disconnected numbers.
The first question we ask is whether the disposition data is clean and whether the customer has confidence in it. For example, a label like “Good/Bad Number” is often too broad since a “Bad Number” could mean many things: disconnected, wrong subscriber, uninterested party, etc. It’s important that the disposition data includes detailed classifications and that the customer is clear on which disposition outcomes they want to analyze or align with Trestle’s data attributes.
Another important consideration is the accuracy of the disposition data. In outbound dialing, if the phone numbers being called are already flagged as spam, some calls may be blocked by the carrier before they even reach the subscriber. As a result, a number labeled as “disconnected” may not be disconnected. This is why confidence in the quality and accuracy of the disposition data is crucial for accurately assessing the value Trestle can provide.
What Should the Sample File Look Like? #
Because Trestle’s data is dynamic and updates frequently based on real-time signals, testing should happen as close to real time as possible. A live test is ideal, but since it requires integration work, an offline test may be easier and faster. In that case, it’s essential to use the most recent data file where the disposition data is available and accurate.
The sample file should also reflect the customer’s typical traffic. For example, pulling data from the last 7 or 30 days provides a more realistic view of how Trestle performs. This allows the customer to assess how well Trestle identifies bad leads and whether it introduces any false positives.
When selecting a sample file for evaluation, it’s important to avoid some common pitfalls that can skew Trestle’s results, positive or negative (or any data vendor’s performance).
- The customer tests a ‘few’ records manually. The number generally varies from 1 to 25, and it does not provide a true sense of our coverage, accuracy, or efficacy for the use case.
- The customer tests a file with 100% bad leads or sign-ups. The issue here is that we might identify all those bad leads, but the customer will not know if any of the good leads, which were not included, would have been rejected by Trestle as well.
- The customer tests a file that other data providers have already filtered, skewing results. Even if Trestle identifies further bad records to help improve conversion, customers cannot get a sense of potential good leads that might have been rejected (mistakenly) by the previous data vendor.
For a true data test, we recommend that customers have a statistically significant data set. We generally think that a file with 10,000-50,000 records will be sufficient for a statistically adequate sample set, especially for an offline data test. A 1,000-record sample might be on the lower side, while 100,000 might be on the higher side. It is also a balance between data recency (you do not want to use too old records where the data might have changed) and obtaining a sufficiently large sample set whose results will be statistically significant, providing enough confidence for a go/no-go decision. For a live test, customers typically run this for 30-90 days at scale to assess the overall impact across multiple metrics.
Analyzing Trestle Data and Impact #
Once the file is processed and Trestle data is available, the customer should consider a few things as part of the correlation analysis.
No vendor or data can find 100% of the bad records and approve all the good customers. This is why it’s important to start with realistic goals that are acceptable for the business. Ideally, this conversation should happen during the initial discussions with us.
Understanding our data attributes in sufficient detail and having the right rulesets to check the impact of the data. This is where the customer should collaborate with the Trestle team to determine which rules and features work best for their use case. Chances are, we have worked with a similar customer in the space and can provide pretty good guidance on the data that will be most useful. When using a machine learning model, ensure that attribute values are used appropriately or 1-hot encoded where necessary to ensure the feature representation in the model is accurate. The sample analysis below is a good example of what an ideal analysis should look like.
Finally, ensure that appropriate ROI analysis is done. If Trestle is finding 50% of bad leads, chances are that conversions are improving due to the prioritization of good leads, and the best salespeople are assigned to those good leads. Thus, do not just consider the dollar value saved by avoiding bad leads but also the incremental revenue generated by improved or additional conversions. How does that compare to Trestle costs? We generally see that Trestle adds 5-10X ROI ((Dollar Saved + Incremental Revenue)/ Trestle Costs) value for our customers. However, at scale, even a 3X ROI is a sufficient reason for the customer to build a business case for integrating Trestle. This is a place where collaborating with the Trestle team can benefit as they can help build this business case once the value is established.
Here’s a sample analysis from working with hundreds of customers. #
| Goal: Identify bad sign ups and improve conversions. Success criteria agreed upon: Reduce bad sign-ups by 20%, while keeping false positives below 1%. | ||||
| Assumptions: Bad sign-up cost to business: $25Every percentage point improvement in conversion: $1,000 | ||||
| Rules Identified: Option 1: Filter out phone.contact_grade == D or FOption 2: Prioritize only (phone.contact_grade == A) OR (phone.contact_grade==B or C AND email.age_score >70) | ||||
| Option | Bad Sign-ups Removed | False Positives | Conversion Rates improvement* | ROI |
| Option 1 | 25% | <1% | 3% | 5X |
| Option 2 | 67% | 3% | 7% | 11X |
To summarize: #
- Have clear and realistic goals for the data test, outlining what the success criteria look like, and ensure that the customer and Trestle are aligned.
- Build a realistic data sample that provides an accurate representation of Trestle’s overall capabilities. Testing only bad records or records approved by another vendor might not give an accurate picture of the actual impact of Trestle (or any data vendor).
- Have a statistically significant data sample that is recent and where the customer has enough confidence in the accuracy and fidelity of the disposition data.
- Ensure a thorough understanding of the Trestle data attributes and how they can be leveraged with appropriate rule sets or feature engineering to maximize value.
Finally, consult with our sales and sales engineering teams. Our team has been involved in hundreds of these evaluations and can provide the best guidance to ensure the test is conducted successfully and the customer has a clear understanding of the value Trestle can bring to the table.