- parse s1search suggestions
- thereby also fixes that some queries would crash the engine if using `https://search.nation.online` as base url because it tried to parse the suggestions as real results
### How to test this PR locally?
- set `base_url = "https://search.nation.online"` and `inactive: false`
- !s1 rust
Add support for xprivo.com. Xprivo has a rate-limit of smth like 20 requests
per day without an API key. However, if you use a proxy like the IPv6 rotating
proxy [1] you basically get unlimited requests.
[1] https://github.com/vojkovic/http-proxy-ipv6-pool-docker
Many engines use the JSON results from Brave (e.g. Brave, BraveAPI, Tusksearch
and more), so the result parsing logic is always the same. This PR deduplicates
this code.
Icons are provided for the main categories known to us.
The distinction between CATEGORY_GROUPS and CATEGORY_NAMES is abandoned in favor
of the latter.
HINT: we need single-color icons; transparent background and the color must be
`currentColor` so that the icons can be used in light/dark themes.
The following SVGs from the ionicons had to be revised (use ``currentColor``).
- client/simple/src/svg/ionicons/logo-rss.svg
- client/simple/src/svg/ionicons/radio-outline.svg
- client/simple/src/svg/ionicons/color-palette-outline.svg
For the following categories, I did not find any usable icons in Ionicons; new
SVGs were created for these categories.
- onions: client/simple/src/svg/ionicons/tor-browser.svg
- lyrics: client/simple/src/svg/ionicons/song-lyric-music.svg [1]
- packages: client/simple/src/svg/ionicons/package_2.svg [1]
[1] Based on Google's Material icons: https://fonts.google.com/icons
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
We previously set the 'cookie' cookie value with a timestamp
from 2024 to `yq=...`. The cookie is not actually used in Yandex,
instead we probably wanted to set the `yq` cookie instead. But
since the engine works without it, it can be dropped.
Also simplifies the request method a bit to be easier to read.
This has only been needed because we were flagged by HTTP2 fingerprinting.
Now, since we use curl cffi, we can bypass the fingerpinting, so we can
use HTTP2 just fine without getting blocked.
Currently, many engines that provide video results call the same
`get_embedded_stream_url` method for setting the `iframe_src`. There
is no value in that because this logic is engine-specific and probably
many video engines forgot to implement this.
With these changes, it's done automatically, so engine implementors
don't have to worry about setting an `iframe_src` (unless the engine
explicitly has a field for it).
The results were previously HTML-escaped for some reason.
That doesn't really make much sense because they never got
unescaped anywhere.
For example, `A&B` gets escaped to `A&B` and never unescaped,
so the autocompletion frontend shows `A&B`, which isn't user-friendly
at all.
Also, it doesn't make sense to escape the full JSON instead
of only escaping the suggestion texts individually because that
makes it even more unclear.
The code used here has always been "bad" because `about` shouldn't be used
as data source, but the engine probably broke when type checks / dataclasses
for the about parameter in engines has been added with
<https://github.com/searxng/searxng/pull/6258>.
Error log:
```
WARNING searx.engines.public domain im: ErrorContext('searx/engines/public_domain_image_archive.py', 143, '\'url\': _clean_url(f"{about[\'website\']}/images/{result[\'objectID\']}"),', 'TypeError', None, ("'EngineAbout' object is not subscriptable",)) False
ERROR searx.engines.public domain im: exception : 'EngineAbout' object is not subscriptable
Traceback (most recent call last):
File "/home/bnyro/Projects/searxng/searx/search/processors/online.py", line 253, in search
search_results = self._search_basic(query, params)
File "/home/bnyro/Projects/searxng/searx/search/processors/online.py", line 239, in _search_basic
return self.engine.response(response)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/home/bnyro/Projects/searxng/searx/engines/public_domain_image_archive.py", line 143, in response
'url': _clean_url(f"{about['website']}/images/{result['objectID']}"),
~~~~~^^^^^^^^^^^
TypeError: 'EngineAbout' object is not subscriptable
```
Qwant now requires a `datadome` cookie that it returns
in the first search response as a `Set-Cookie`.
This cookie has to be sent for all requests, otherwise they
will be blocked.
This means that now, the first search request is blocked (results in CAPTCHA),
and only the subsequent searches work (same happens on the Qwant website for me).
However, I don't think it's worth repeating the same
search request multiple times very quickly because
that also makes us more suspicious.
Some search engines encode domains that contain special characters
in the IDN format, i.e. by using Punycode (https://en.wikipedia.org/wiki/Punycode).
This means that domains formatted like that are not human readable
and thus very unintuive.
Also, as only some engines use Punycode, this often causes a result
to appear multiple times, once with a Punycode domain (e.g. xn--allestrungen-9ib.de),
and once with a normal domain (e.g. allestörungen.de).
Always formatting the domains nicely has the caveat that the official sites
are harder to distinguish from malicious clones that just swapped out some
characters by using Punycode, e.g. replaced `a` with `á`. I don't think
that will be much of an issue though - I don't think Punycode results
previously stopped users from clicking the link, and I also think that
most search engines filter out such bad results or don't even have them
indexed.
Changes:
- the `embed.js` request now requires a user agent header
- we include a user agent in the "actual" request (I dropped it by accident)
- we only send the first 4 decimal places of the location
instead of 7+ (not required, but harder to detect)
`engines` only contains active engines. Since `google` was set to
inactive, it's no longer part of the , making the engine crash.
With this workaround, we directly load the engine traits from the data
dictionary instead of going the intermediate step with the Google engine.
We're now shuffling the order of query arguments, as
it seems that Qwant actively blocks the query order of
SearXNG.
tgp seems to be short for "test group" (look at the XHR requests the
browser sends to "https://www.qwant.com/action/ui") - its actual value
doesn't matter, as long as it's sent and at the correct position in the
query params and doesn't change too frequently
FindFiles.net is a specialized file search engine designed to help you search
files online with precision. Unlike traditional search engines that mainly index
web pages, FindFiles focuses on finding real files on the internet - including
PDFs, documents, archives, videos, datasets, and more. [1]
[1] https://findfiles.net
Bnyro