ADR-009: Modernize Nouveau Search Architecture
Decision Analysis and Resolution: CouchDB Nouveau Search Architecture Modernization
Created by: SW360 Architecture Team Decision Date: July 2026 Status: Proposed
Table of Contents
- Context and Problem Statement
- Goal
- Key Principles
- Options Analysis
- Criteria for Making a Decision
- Technical Specification
1. Context and Problem Statement
The current SW360 search mechanism constructs queries by appending trailing
wildcards to every input token (term1* OR term2* OR term3*). While this
maximizes recall for unknown terms, it causes severe issues in production
environments:
- High Noise & Poor Precision: For multi-word queries (e.g.,
"My test project"), the default OR logic matches any document containing any word starting with those prefixes, returning hundreds of irrelevant entries (e.g., 340+ results) instead of the few actual matches. - Performance Penalties: Query-time
*wildcards bypass standard Lucene token analysis and force expensive term-expansion scans across inverted index segments. - Legacy Generic Concatenation: Global generic searches rely on manually
concatenating strings into custom
allfields, destroying field token boundaries and preventing field-specific relevance scoring. - Inefficient Cluster Pagination: Requesting page $N$ in the REST backend currently executes $N$ sequential HTTP requests to CouchDB to walk the bookmark chain. In a multi-node CouchDB cluster, sequential bookmark requests can land on different cluster nodes, creating unnecessary inter-node RPC overhead and potential bookmark state mismatches.
- Redundant Sorting Overhead: Search results sorted by UI columns undergo double-sorting—once in Lucene and again in Java memory via Comparator calls after document hydration.
We need a search architecture that brings “Google-like” precision (exact matches at #1, minimal noise) while preserving fast prefix matching, native catch-all search capabilities, stateless high-performance cluster pagination, and offloading multi-field sorting entirely to the search engine.
2. Goal
The end goal is to implement a modernized search architecture for SW360 that:
- Provides high precision and recall for user queries, minimizing irrelevant results.
- Reduces query execution time and resource consumption in CouchDB clusters.
- Supports stateless pagination and efficient sorting without redundant processing.
One of the biggest problem currently experienced by users is the noise in search results. For example, they created a project called “My test Project” and when searching for it next day, they get results like
- “A test project” (noise)
- “Aa project” (noise)
- “My test project” (actual match)
- “Test project” (noise)
On an environment with 10000+ projects, for a simple query, you might get 700+ results back and you’d have to scroll through them manually to find the actual match.
The goal is to reduce the noise in search results and provide a more accurate and efficient search experience for users, while also improving the performance of the search functionality in the SW360 application.
The major change is to move away from current "word1*" OR "word2*" style
queries to a more structured query approach that leverages CouchDB Nouveau’s
capabilities for Lucene-based full-text search. Improve the search query
construction to "word1" AND "word2*" style queries, reducing the irrelevant
results. At the same time, use Lucene’s scoring and ranking capabilities to
ensure that the most relevant results are returned first, improving the overall
user experience.
3. Key Principles
- Precision & Relevance: Exact phrase matches must automatically rank at position #1.
- Low Noise: Search for 3–4 word project names should return only matching documents rather than hundreds of partial-match hits.
- Native Catch-All Search: Utilize CouchDB Nouveau’s reserved
defaultfield pattern instead of manual string concatenation. - Stateless Cluster-Friendly Pagination: Calculate top-$N$ offsets
(
limit = pageNumber * pageSize) to execute 1 single request to CouchDB Nouveau per page jump, slicing the result page in Java memory. This eliminates multi-node bookmark dependencies and keeps the REST layer completely stateless. - Architectural Simplicity: Eliminate redundant Java-side in-memory sorting by utilizing native CouchDB Nouveau multi-field sorting.
- Query Performance (additional): Prefix searching should operate in $O(1)$ time rather than requiring expensive query-time wildcard expansions.
4. Options Analysis
4.1. Status Quo
Option 1: Status Quo (Query-Time Wildcards + Manual String Concatenation + $N$-Sequential Bookmarks + Java Re-Sorting)
- Retain trailing
*on every token, concatenate strings for generic searches, and rely on Java-side post-processing to re-rank and filter results.
4.2. Edge N-Gram Indexing with Tiered Conjunction Querying
Option 2: CouchDB Nouveau with JS Index-Time Edge N-Grams, Tiered Conjunction
(AND) Querying, Single-Query Top-$N$ Offset Slicing, and Native Sorting
- Index catch-all content under Nouveau’s reserved “
default” field name for field-less generic searches. - Generate prefix N-gram tokens directly inside the CouchDB Nouveau JavaScript
index()function, assigned to standard Lucene analyzers (whitespace/standard/keyword). - Use a 2-tier query structure enforcing
ANDconjunction logic with phrase boosting (^100). - Request
limit = pageNumber * pageSizein a single HTTP call to CouchDB and extract the target page sublist in Java memory (subList). - Index dedicated un-analyzed
stringfields for native Lucene column sorting and tie-breaking.
5. Criteria for Making a Decision
The single biggest pain point for the SW360 user community is the high noise in search results. The current search architecture returns hundreds of irrelevant results for multi-word queries. This is best when you don’t know what you are looking for, but remember only a fragment. However, when you know the exact thing you want and search for it, the results are so noisy that users generally give up on the results. Instead, they have started to share direct links to one another. The single most important criterion for making a decision is to reduce the noise in search results and provide a more accurate and efficient search experience for users. If in the process of achieving this, we could also improve the performance of the search functionality in the SW360 application, that would be a bonus. The following criteria were considered in making the decision:
- Precision & Relevance: The new search architecture must return exact phrase matches at the top of the results list.
- Low Noise: The new search architecture must return only matching documents for multi-word queries, rather than hundreds of partial-match hits.
5.1. Positive Consequences
- Native Generic Search: Utilizing Nouveau’s reserved “
default” field removes artificial string concatenation and preserves proper token boundaries for global search bar inputs. - Instant Prefix Lookup: JS-generated Edge N-Grams turn prefix searches into exact inverted index term lookups ($O(1)$ time).
- 100% CouchDB Nouveau Compliant: Uses standard named analyzer strings
(
"standard","whitespace","keyword") supported by Nouveau’sdefault_analyzerandfield_analyzersoptions. - Cluster-Safe Stateless Pagination: Single top-$N$ requests eliminate $N$-sequential bookmark calls and work seamlessly across load-balanced multi-node CouchDB clusters without needing sticky sessions or external caching layers.
- Noise Reduction: Switching default multi-word queries from loose
ORto strictANDconjunction reduces result sets from 300+ entries down to exact matches. - Exact Match Launching: Tier 1 phrase boosting (
^100) guarantees that exact matches always rocket to position #1. - Zero Java Re-sorting: CouchDB Nouveau returns results pre-sorted by Lucene, removing duplicate Java Comparator sorting overhead.
5.2. Negative Consequences
- Index Size Growth: Pre-computing N-Gram tokens increases disk usage for CouchDB Nouveau index segments (typically 15–25% increase).
- Re-indexing Required: Existing CouchDB databases must re-index documents against the new Nouveau design document.
6. Technical Specification
6.1. CouchDB Nouveau Design Document Definition
Note: In CouchDB Nouveau, default_analyzer and field_analyzers map field
names to built-in Lucene analyzer names ("standard", "whitespace",
"keyword", "english", etc.). Custom token filters like edge_ngram are
generated directly within the JavaScript index() function.
{
"_id": "_design/sw360_search",
"nouveau": {
"indexes": {
"projects": {
"default_analyzer": "standard",
"field_analyzers": {
"default": "standard",
"default_ngram": "whitespace",
"name_exact": "standard",
"name_ngram": "whitespace",
"version_ngram": "keyword",
"tag_ngram": "whitespace"
},
"index": "function(doc) {
if (doc.type === 'project') {
// Helper function to generate Edge N-Grams in JavaScript
function emitEdgeNGrams(fieldName, text, minGram, maxGram) {
if (!text) return;
var words = text.toLowerCase().split(/\\s+/);
for (var i = 0; i < words.length; i++) {
var word = words[i];
var limit = Math.min(word.length, maxGram);
for (var len = minGram; len <= limit; len++) {
index('text', fieldName, word.substring(0, len));
}
}
}
// --- 1. RESERVED 'DEFAULT' FIELD (CATCH-ALL GENERIC SEARCH) ---
if (doc.name) {
index('text', 'default', doc.name);
emitEdgeNGrams('default_ngram', doc.name, 2, MAX_GRAM_BOUND);
}
if (doc.description) {
index('text', 'default', doc.description);
}
if (doc.version) {
index('text', 'default', doc.version);
}
if (doc.tag) {
index('text', 'default', doc.tag);
}
// --- 2. SPECIFIC FIELD PREFIX & EXACT FIELDS ---
if (doc.name) {
index('text', 'name_exact', doc.name);
emitEdgeNGrams('name_ngram', doc.name, 2, 25);
index('string', 'name_sort', doc.name.toLowerCase());
}
if (doc.version) {
var ver = doc.version.toLowerCase();
emitEdgeNGrams('version_ngram', ver, 1, ver.length);
index('string', 'version_sort', ver);
}
if (doc.tag) {
emitEdgeNGrams('tag_ngram', doc.tag, 2, 10);
}
// --- 3. EXACT MATCH ONLY FIELDS ---
if (doc.license) {
index('string', 'license', doc.license);
}
// --- 4. SORTABLE NUMERIC / DATE FIELDS ---
if (doc.createdOn) {
index('double', 'created_on', doc.createdOn);
}
}
}"
}
}
}
}
6.2. Reference Java Implementation
The following service utility constructs Nouveau queries, supports reserved
default generic searches, and provides simple offset limit calculation for
single-query page extraction:
package org.eclipse.sw360.datahandler.search;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public class SW360NouveauSearchBuilder {
private static final int MAX_GRAM_CAP = 25;
/**
* Constructs a tiered Lucene query string for a target field.
*/
public static String buildFieldQuery(String fieldExact, String fieldNgram, String rawInput) {
if (rawInput == null || rawInput.trim().isEmpty()) {
return "*:*";
}
String trimmed = rawInput.trim();
// Exact phrase search if quoted
if (trimmed.startsWith("\"") && trimmed.endsWith("\"") && trimmed.length() > 2) {
String phrase = trimmed.substring(1, trimmed.length() - 1);
// + name_exact:"My test project"
return "+" + fieldExact + ":\"" + sanitizeLuceneChars(phrase) + "\"";
}
String[] tokens = Arrays.stream(trimmed.split("\\s+"))
.map(SW360NouveauSearchBuilder::sanitizeLuceneChars)
.filter(s -> !s.isEmpty())
.toArray(String[]::new);
if (tokens.length == 0) {
return "*:*";
}
// Single-word query pattern
if (tokens.length == 1) {
String token = capToken(tokens[0].toLowerCase());
// (name_exact:my^100 OR name_ngram:my)
return String.format("(%s:%s^100 OR %s:%s)", fieldExact, token, fieldNgram, token);
}
// Multi-word tiered query pattern
String fullPhrase = String.join(" ", tokens);
String phraseClause = fieldExact + ":\"" + fullPhrase + "\"^100";
StringBuilder andClause = new StringBuilder();
andClause.append("(");
for (int i = 0; i < tokens.length; i++) {
if (i > 0) {
andClause.append(" AND ");
}
if (i == tokens.length - 1) {
// Last token uses n-gram for prefix matching
andClause.append(fieldNgram).append(":").append(capToken(tokens[i].toLowerCase()));
} else {
// Earlier tokens use exact match
andClause.append(fieldExact).append(":").append(tokens[i].toLowerCase());
}
}
andClause.append(")");
/*
* (
* name_exact:"My test project"^100
* OR
* (
* name_exact:my AND name_exact:test AND name_ngram:project
* )
* )
*/
return String.format("(%s OR %s)", phraseClause, andClause.toString());
}
/**
* Constructs generic "search anything" bar queries targeting reserved 'default' fields.
*/
public static String buildGenericQuery(String genericInput) {
return buildFieldQuery("default", "default_ngram", genericInput);
}
/**
* Combines multiple explicit field criteria into a composite query.
*/
public static String buildCompositeQuery(String nameInput, String version, String tag, String vendor) {
List<String> clauses = new ArrayList<>();
if (nameInput != null && !nameInput.trim().isEmpty()) {
clauses.add("+(" + buildFieldQuery("name_exact", "name_ngram", nameInput) + ")");
}
if (version != null && !version.trim().isEmpty()) {
clauses.add("+version_ngram:" + capToken(sanitizeLuceneChars(version.trim().toLowerCase())));
}
if (tag != null && !tag.trim().isEmpty()) {
clauses.add("+tag_ngram:" + capToken(sanitizeLuceneChars(tag.trim().toLowerCase())));
}
if (vendor != null && !vendor.trim().isEmpty()) {
clauses.add("+vendor:\"" + sanitizeLuceneChars(vendor.trim()) + "\"");
}
if (clauses.isEmpty()) {
return "*:*";
}
/*
* +(name_exact:"My test project"^100 OR (name_exact:my AND name_exact:test AND name_ngram:project))
* AND
* +version_ngram:1.0.0
* AND
* +tag_ngram:department
* AND
* +vendor:"eclipse"
*/
return String.join(" AND ", clauses);
}
/**
* Maps UI column sort requests directly to Nouveau sort parameters.
* Uses more than 1 tiebreaker field to ensure deterministic sort order for identical values.
*/
public static List<String> buildSortParameters(String uiColumn, boolean isAscending) {
String direction = isAscending ? "" : "-";
if (uiColumn == null || uiColumn.isEmpty() || "score".equalsIgnoreCase(uiColumn)) {
return List.of("relevance", "name_sort", "version_sort");
}
return switch (uiColumn.toLowerCase()) {
case "name" -> List.of(direction + "name_sort", "relevance", "version_sort");
case "version" -> List.of(direction + "version_sort", "name_sort", "relevance");
case "vendor" -> List.of(direction + "vendor_sort", "name_sort", "relevance");
case "createdon", "date" -> List.of(direction + "created_on", "name_sort");
default -> List.of(direction + uiColumn, "relevance");
};
}
/**
* Calculates the top-N limit to fetch from CouchDB Nouveau in a single query.
*/
public static int calculateFetchLimit(int pageNumber, int pageSize) {
int page = Math.max(1, pageNumber);
int size = Math.max(1, pageSize);
return page * size;
}
/**
* Extracts the requested page sublist in Java memory from the single top-N result set.
*/
public static <T> List<T> extractPageSublist(List<T> allHits, int pageNumber, int pageSize) {
if (allHits == null || allHits.isEmpty()) {
return List.of();
}
int page = Math.max(1, pageNumber);
int size = Math.max(1, pageSize);
int startIndex = (page - 1) * size;
if (startIndex >= allHits.size()) {
return List.of();
}
int endIndex = Math.min(startIndex + size, allHits.size());
return allHits.subList(startIndex, endIndex);
}
private static String capToken(String token) {
return token.length() > MAX_GRAM_CAP ? token.substring(0, MAX_GRAM_CAP) : token;
}
private static String sanitizeLuceneChars(String input) {
if (input == null) return "";
return input.replaceAll("([+\\-!\\(\\)\\{\\}\\[\\]\\^\"\\~\\*\\?\\:\\\\/])", "\\\\$1")
.replaceAll("&&", "\\\\&&")
.replaceAll("\\|\\|", "\\\\||")
;
}
}
6.3. Native Nouveau Sorting Matrix
| User Action | Nouveau “sort” Parameter | Description |
|---|---|---|
| Default Search | ["relevance", "name_sort", "version_sort"] | Ranks exact phrase matches (#1) by BM25 score, tie-breaking alphabetically. |
| Sort by Name (asc) | ["name_sort", "relevance", "version_sort"] | Pure alphabetical sort with score as secondary tie-breaker. |
| Sort by Version (desc) | ["-version_sort", "name_sort", "relevance"] | Descending version sort with multi-field tie-breaking. |
| Sort by Created Date (desc) | ["-created_on", "name_sort"] | Sorts newest first. |
This program and the accompanying materials are made available under the terms of the Eclipse Public License 2.0 which is available at https://www.eclipse.org/legal/epl-2.0/