Engines like Pexels provide images that one would more likely expect in a
category named ``stock_images``.
By classifying engines like Pexels more precisely, we enable a more specific
handling of them, which also helps to avoid mixing them with the more general
image search on the internet. [1]
We leave these engines *enabled* by default, take them out of ``images`` and
move them into the specialization ``stock_images``.
This makes it possible, on the one hand, for the admin to configure a tab in the
UI, and on the other hand, the user can always select the group directly using
the search syntax ``!stock_images ...``.
What we need to keep in mind:
1. Technically speaking, there are no subcategories in the strict
sense. Organizationally, the *subcategories* are derived from the
`categories_as_tabs`.
2. From an admin’s perspective, `categories_as_tabs` is the tool for
structuring their content.
3. In the context of “enabled/disabled by default”: We should avoid having
the admin have to activate many individual engines when they want to
structure their content (UI tabs).
So if we leave an engine enabled by default and **move** it from `images` to a
new category, the admin only needs to add the new category to
`categories_as_tabs` to restructure their content. If the new category is also
part of the translations (see `searx/searxng.msg` in this PR), this UI structure
would also be available internationally.
[1] https://github.com/searxng/searxng/pull/6690#issuecomment-5646291679
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Code style somewhat unified, type annotations added, ``LegacyResult`` replaced
by ``Image``, but no functional modifications.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Queries like `!mvw hercule poirot` usually yield no results if the `description`
is not also included in the search.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Additional fields:
add resolution, thumbnail and author to the image result
Types:
use EngineResults and add type hints from internall APIs
Disabled engine:
as of 09/2026: The availability of https://openverse.org/ is poor, and queries
and image requests often time out. However, these issues also occur when using
a WEB browser to search on openverse.org or view images.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Switching to internal APIs such as ``EngineResults`` and
adding more type hints. No functional changes.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The ``mkt`` argument does no longer exists and the ``async`` argument has been
renamed to ``mmasync``.
For still unknown reasons (IP based?), some users had to observe that the title
was missing [1]. I myself was not able to reproduce this error, however the
evaluation of the title was additionally expanded by the attribute value of the
``<a title=".."`` element.
Related:
- [1] https://github.com/searxng/searxng/pull/6690#issuecomment-5636811572
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [fix] engine: pinterest - fix empty titles and complete modernization
The titles from the fields ``title`` and ``grid_title`` are mostly empty or have
short strings without meaningful content. Various fields for the title are now
being queried, which have more informative value.
As part of the bug fix, the engine was completely revised and modernized.
Related:
- https://github.com/searxng/searxng/pull/6690#issuecomment-5631113121
Co-authored-by: @vojkovic
The previous implementation ran into an error if the search term contained words
other than just the location (ValueError was raised).
To test engine use search terms like:
!ddw weather berlin germany
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [mod] show img_load_error.svg if thumbnail fails to load (#6368)
Brave and a few other engines occasionally provide dead URLs for the thumbnails,
which looks pretty ugly in the UI. As already with the image search, in case of
an error img_load_error.svg should be displayed here.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [build] /static (#6368)
---------
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The GSA headers that were introduced in PR #5644 unfortunately no longer
work (#6359).
The Google engines
- google.py
- google_videos.py
do not work anymore either, but we'll leave it in the code for now:
- In google.py, central functions like get_google_info(..) are provided, which
are also used by other modules.
- We will probably need a Google HTML (and video) engine again very soon.
Related:
- https://github.com/searxng/searxng/issues/6359
- https://github.com/searxng/searxng/pull/6364
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
So far, any change to any file triggered a new build, and changes to the
doc-strings (in ./searx) were not taken into account. With this patch, a build
is only triggered when files with the specified file extensions change.
The watch has been expanded to ./searx so that changes to doc strings also
trigger a new build.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
TBH: Some of the parameters such as engines or plugins still theoretically exist
in SearXNG, but we will no longer support them. They are usually accessed via
cookies and should not pollute the API / it is these supposed features that
hinder SearXNG in further development.
Closes: https://github.com/searxng/searxng/issues/6003
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [mod] update_engine_descriptions.py - revision of the script (#6309)
The script was a bit outdated, type hints were missing, and the strings were
converted to double quotation marks
The `descriptions.items()` are sorted to avoid assigning the references to
seen_descriptions randomly (should reduce unnecessary diffs in the future).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [data] update searx.data - update_engine_descriptions.py (#6309)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
---------
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The old property should still be supported for a transitional period; the
reasons for this can be seen from the discussion in [1] / the further procedure
is also discussed there.
[1] https://github.com/searxng/searxng/issues/6261
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the past, the engine option ``language_support`` was not consistently
maintained; with this patch, a ValueError is now thrown if an engine has
languages in its traits but language_support is not set to True.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Drop outdated engine attributes: supported_languages, language_aliases
Complete, normalize and document the type definitions for the engine-module and
engine-class.
For the ``engine.about`` section of the configuration, a type check is performed
based on structure ``searx.enginelib.EngineAbout``.
The property ``engine.about.language`` no longer exists; existing values have
been migrated to ``engine.language``.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [mod] template images.html: reformatted for readability (no func change)
In preparation for upcoming changes, the template is being reformatted for
better readability; no functional changes are being made.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [mod] image results: add list of alternative formats
To test alternatives formats apply patch from below, query ``!flaticon bmw`` and
open the detail view for the image.
diff --git a/searx/engines/flaticon.py b/searx/engines/flaticon.py
index 06b6a8e25..d88388705 100644
--- a/searx/engines/flaticon.py
+++ b/searx/engines/flaticon.py
@@ -8,7 +8,7 @@ from urllib.parse import urlencode
import typing as t
-from searx.result_types import EngineResults
+from searx.result_types import EngineResults, ImageRef
if t.TYPE_CHECKING:
from searx.extended_types import SXNG_Response
@@ -61,6 +61,14 @@ def response(resp: "SXNG_Response"):
thumbnail_src=_fix_url(result["png"]),
img_src=_fix_url(result["png512"]),
author=result["team_name"],
+ formats=[
+ ImageRef(label="PNG 100x100", url="https://example.org/test.png", subtype="png"),
+ ImageRef(label="SVG", url="https://example.org/test.svg", subtype="svg+xml"),
+ ImageRef(url="https://example.org/test.jpg", subtype="jpeg"),
+ ImageRef(url="https://example.org/test.bmp", subtype="bmp"),
+ ImageRef(url="https://example.org/test.ico", subtype="x-icon"),
+ ImageRef(url="https://example.org/test.tif", subtype="tiff"),
+ ],
)
)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
---------
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The initialization of the DB schema ("base schema") has so far been done on
demand, which causes race conditions with competing threads and processes.
The DDL statements for creating the "base schema" are now executed as part of
the initialization of the app.
Further improvements were made to harden the database applications:
- Wikidata & Radio-Browser engine perform their initialization only once (so far
the initialization was carried out in each thread/process).
- If multiple processes try to set DB's WAL mode when opening the DB at the same
time, this usually leads to another race condition, which is now also caught.
Related:
- https://github.com/searxng/searxng/issues/6181#issuecomment-4586705Closes: #6181
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
On the first page of the WEB search, there are, among other things, sections for
videos and news. The video results from these sections should not be used as
results in the WEB search of SearXNG.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Exceptions in the execution of the callback must be caught / ignored and logged
on the ERROR log.
To test, apply this patch to provoke a ValueError exception::
diff --git a/searx/data/tracker_patterns.py b/searx/data/tracker_patterns.py
index ed4415bce..695ed05d2 100644
--- a/searx/data/tracker_patterns.py
+++ b/searx/data/tracker_patterns.py
@@ -114,6 +114,7 @@ class TrackerPatternsDB:
Returns bool ``True`` to use URL unchanged (``False`` to ignore URL).
If URL should be modified, the returned string is the new URL to use.
"""
+ raise ValueError("test callback exceptions")
new_url = url
parsed_new_url = urlparse(url=new_url)
Start a `make run` instance and query for example `amazon` .. have a look at the
ERROR log:
ERROR searx.result_types: filter_urls (field 'url'): ignore ValueError('test callback exceptions') from callback searx/data/tracker_patterns.py:117
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The implementation is normalized, type annotations are applied, and the results
are freed from the HTML markup (which is partially present).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the result-list, the ``number_of_results`` indicate the number of hits in the
Index, they do not indicate how many results are in the answer.
In the past, search engines such as google or ddg had an indication on the first
page of a search term of how many hits there were for this term in total in
their index.
This info was added up in SearXNG and delivered under ``number_of_results``.
Nowadays the search engines no longer indicate how many hits there are in the
index and so this field in SearXNG is also superfluous.
- https://github.com/searxng/searxng/issues/2457#issuecomment-2566181574
- https://github.com/searxng/searxng/issues/2987
- https://github.com/searxng/searxng/issues/5034
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
[mod] normalize variable name for the max number of results per request
In the past, we have used different names for the variable that specifies the
maximum number of hits in the outgoing request.
- ``page_size``
- ``number_of_results``
- ``nb_per_page``
Since *page_size* is the most accurate term and is also used in the XPath
engines, all other engines are adjusted accordingly within this
patch .. documentation adjusted accordingly.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Previously, `update_engine_traits.py` would fetch traits for all engines, which
is very slow and by side-effect touches engine data that are unrelated to the
engine you're currently working on.
To be faster with developing `update_engine_traits.py` supports now engine
arguments.
To test, jump into the developer environment and run the script::
$ ./manage dev.env
(dev.env)$ ./searxng_extra/update/update_engine_traits.py --help
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The readability can be improve with the following two changes:
Slightly increase the content width to make the engine list more readable::
.content {
width: 52em; /* instead of 46em */
}
To avoid word wrap in bang::
p code.literal {
text-wrap: nowrap;
}
Sugested-by: https://github.com/searxng/searxng/pull/3408#issuecomment-2094082144
The type checker in my IDE shut down after over 500 errors / after this
patch there are still 125 criticisms, however its an improvement and a better
starting point.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The AOL engines deliver no results or too many incorrect results; in #5972 it is
reported that the AOL-images do not work, and on AOL-web the language selection
seems to depend more on the IP than on the selected language / Locally, the
language selection works for me, on the public server only English results are
displayed, which significantly worsens the result list.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
PR #5947 upgraded the pygments dependency, this commit updates
the static files ::
$ make themes.all
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Karmasearch seem to crash when searching with long queries >= 100 characters.
The returned JSON is exactly this `["",[]]`, which will crash when trying to
access `resp.json()["results"]`
Close: https://github.com/searxng/searxng/issues/5911
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The end-of-life (EOL) of a Python release is reached after 5 years, with the
last three years dedicated to security fixes. [1]
Unfortunately, this doesn't apply to common libraries (dependencies): bug fixes
are often only included in major releases, and minor releases with corresponding
security fixes are rarely offered.
To make matters worse, these dependencies often prematurely discontinue their
support for older Python releases (because, for example, they want/need to use
new Python features).
If we want to offer secure software, we are faced with the dilemma of either
supporting old Python releases and accepting that there are security bugs in the
dependencies, or ending support for older Python versions before the actual EOL
of the Python release.
[1] https://devguide.python.org/versions/
Closes: https://github.com/searxng/searxng/issues/5869
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Submitting an issue on GitHub isn't a end user task .. most issue reports coming
from this form are not filled out, since end users are not familiar with the
needs of a developer community.
Closes: https://github.com/searxng/searxng/issues/5820
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The online engines emulate a request as it would come from a web browser, which
is why the HTTP headers in the default settings should also be set the way a
standard web browser would set them.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The default settings for the suspend times were previously 24 hours and 3 hours,
respectively. Based on my experience, these defaults are too high; most engines
handle suspend times of 3 minutes or 1 hour (captcha) without any problems.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Package typer-slim does nothing other than depend on typer. The only reason this
package exists is as a migration path for old projects that used to depend on
typer-slim, so that they can get the latest version of typer.
Install instead:
pip install typer
Package typer-slimis deprecated and will stop receiving any updates and
published versions.
[1] https://pypi.org/project/typer-slim/
Closes: https://github.com/searxng/searxng/issues/5742
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Unit test fails::
Traceback (most recent call last):
File "/share/searxng/local/py3/lib/python3.10/site-packages/parameterized/parameterized.py", line 620, in standalone_func
return func(*(a + p.args), **p.kwargs, **kw)
File "/share/searxng/tests/unit/test_locales.py", line 121, in test_locale_optimized_territory
self.assertEqual(locales.match_locale(locale, locale_list), expected_locale)
AssertionError: 'fr-CH' != 'fr-BE'
- fr-CH
+ fr-BE
With the `babel` update from 2.17.0 to 2.18.0 the population DB has been
updated (the test was implemented for the old values).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
For unknown locales, the return value of::
locales.get_locale(params['searxng_locale'])
is None which cuase the following issue::
ERROR searx.engines.presearch : exception : 'NoneType' object has no attribute 'territory'
Traceback (most recent call last):
File "search/processors/online.py", line 256, in search
search_results = self._search_basic(query, params)
File "search/processors/online.py", line 231, in _search_basic
self.engine.request(query, params)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
File "engines/presearch.py", line 153, in request
request_id, cookies = _get_request_id(query, params)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
File "engines/presearch.py", line 140, in _get_request_id
if l.territory:
^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'territory'
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Markus Heiser
