Skip to main content

Stopwords API

Stopwords are common words that are filtered out from search queries to improve search relevance and performance. Words like "the", "a", "an", "and", etc. are typically stopwords.

Overview​

Stopwords are words that don't add significant meaning to search queries. Filtering them out helps:

  • Improve search performance
  • Reduce index size
  • Focus on meaningful terms
  • Improve search relevance

List Stopwords​

Get all stopwords for a collection.

Endpoint: GET /collections/{name}/stopwords

Example Request:

curl http://localhost:9200/collections/products/stopwords

Example Response:

{
  "stopwords": [
    "the",
    "a",
    "an",
    "and",
    "or",
    "but",
    "in",
    "on",
    "at",
    "to",
    "for",
    "of",
    "with",
    "by"
  ]
}

List All Stopwords​

List all stopword records known to the server.

Endpoint: GET /stopwords

curl http://localhost:9200/stopwords

Use /stopwords/global when you only need global stopwords.

Global Stopwords​

Global stopwords apply to all collections. They are merged with collection-specific stopwords at query time.

List global stopwords

GET /stopwords/global

curl http://localhost:9200/stopwords/global

Add a global stopword

POST /stopwords/global

curl -X POST http://localhost:9200/stopwords/global \
  -H "Content-Type: application/json" \
  -d '{"word":"the"}'

Delete a global stopword

DELETE /stopwords/global/{word}

curl -X DELETE http://localhost:9200/stopwords/global/the

Add Stopword​

Add a stopword to a collection.

Endpoint: POST /collections/{name}/stopwords

Request Body:

{
  "word": "stopword"
}

Example Request:

curl -X POST http://localhost:9200/collections/products/stopwords \
  -H "Content-Type: application/json" \
  -d '{
    "word": "the"
  }'

Example Response:

{
  "message": "Stopword added successfully",
  "word": "the"
}

Delete Stopword​

Remove a stopword from a collection.

Endpoint: DELETE /collections/{name}/stopwords/{word}

Example Request:

curl -X DELETE http://localhost:9200/collections/products/stopwords/the

Example Response:

{
  "message": "Stopword deleted successfully",
  "word": "the"
}

Common Stopwords​

Here are some common English stopwords you might want to add:

{
  "stopwords": [
    "a", "an", "and", "are", "as", "at", "be", "by", "for", "from",
    "has", "he", "in", "is", "it", "its", "of", "on", "that", "the",
    "to", "was", "will", "with"
  ]
}

Local vs Global Stopwords​

  • Local (collection) stopwords: stored under /collections/{name}/stopwords and only affect that collection.
  • Global stopwords: stored under /stopwords/global and apply to every collection.
  • Scope preference: controlled by query_settings.lexical_scope_preference in hlquery.conf.

Scope Preference​

Stopwords are a set of terms, so precedence is handled at the scope level:

<query_settings
     enable_stopwords="true"
     lexical_scope_preference="merge">
ValueBehavior
mergeDefault. Global and local stopwords are combined.
localIf a collection has local stopwords, hlquery uses those for that collection instead of the global stopword set. If no local stopwords exist, global stopwords are used.
globalIf global stopwords exist, hlquery uses them instead of collection stopwords. If no global stopwords exist, collection stopwords are used.

Example:

  • Global stopwords: the, and, demo
  • Collection stopwords: product, sku

With merge, all five terms are filtered.

With local, only product and sku are filtered for that collection.

With global, only the, and, and demo are filtered.

How Stopwords Work​

Stopwords are applied to lexical and hybrid searches. Vector-only searches do not use stopwords.

When a user searches, stopwords are automatically filtered out:

Example:

  • User query: "the best laptop"
  • After stopword filtering: "best laptop"
  • Search matches documents containing: "best" and "laptop"

Best Practices​

  1. Language-specific: Use stopwords appropriate for your content language
  2. Domain-specific: Consider domain-specific stopwords (e.g., "product" in e-commerce)
  3. Test impact: Verify stopwords improve search results
  4. Don't over-filter: Avoid removing words that might be important in context
  5. Update regularly: Review and update stopwords as needed

Use Cases​

  • English content: Filter common English words
  • Technical content: Filter common technical terms that don't add meaning
  • E-commerce: Filter generic product terms
  • Multilingual: Use language-specific stopwords

Bulk Operations​

To add multiple stopwords, make multiple POST requests or use a script:

#!/bin/bash
STOPWORDS=("the" "a" "an" "and" "or" "but")

for word in "${STOPWORDS[@]}"; do
  curl -X POST http://localhost:9200/collections/products/stopwords \
    -H "Content-Type: application/json" \
    -d "{\"word\": \"$word\"}"
done

Next Steps​