Free from ghcr.io/searxng/cache! Now we save and load images without polluting
this shared (and useless) layer cache, which also makes it easy to download
artifacts for local debugging.
I also removed this nonsense about virtualenv normalization, this will be
properly fixed by using lockfiles on py dependencies.
A missing condition allowed an attacker to mask their IP address from the
server. This exploit can only be carried under specific client & server
environments.
Reported-by: Raffaele Forte <raffaele@backbox.org>
Signed-off-by: Ivan Gabaldon <igabaldon@inetol.net>
* [enh] engines: rework bing engine
Only Bing-Web has been reworked.
Some features now require JavaScript (paging and time-range results).
Cookies no longer work, parameters such as `cc`, `ui`, ... alter the results.
The engine only appears to use the locale from `Accept-Language` header properly.
The rest of Bing's child engines (Bing-Image, Bing-Video, ...) seem to benefit
from using `mkt` param in conjunction with the `Accept-Language` header
override, although Bing-Web does not (?)
* [enh] explicit mkt
* [fix] engines: bing_videos.py
https://github.com/searxng/searxng/pull/5793#pullrequestreview-3881883250
* [mod] client/simple: client plugins
Defines a new interface for client side *"plugins"* that coexist with server
side plugin system. Each plugin (e.g., `InfiniteScroll`) extends the base
`ts Plugin`. Client side plugins are independent and lazy‑loaded via `router.ts`
when their `load()` conditions are met. On each navigation request, all
applicable plugins are instanced.
Since these are client side plugins, we can only invoke them once DOM is fully
loaded. E.g. `Calculator` will not render a new `answer` block until fully
loaded and executed.
For some plugins, we might want to handle its availability in `settings.yml`
and toggle in UI, like we do for server side plugins. In that case, we extend
`py Plugin` instancing only the information and then checking client side if
[`settings.plugins`](https://github.com/inetol/searxng/blob/1ad832b1dc33f3f388da361ff2459b05dc86a164/client/simple/src/js/toolkit.ts#L134)
array has the plugin id.
* [mod] client/simple: rebuild static
I've narrowed the permissions and rotated the token for the deploy account on
DockerHub registry. I replaced the secret ref in GitHub so that it's available
organization wide. No further actions are necessary.
Notes on the developer environment with MISE have been added to the Quickstart
Guide. We migrated `.tool-versions` to replace it with `mise` [1], so we can
now remove `asdf` related documentation.
[1] https://github.com/searxng/searxng/pull/5253
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
Docker buildx outputs the following error:
variable expansion is not supported for --from, define a new stage with FROM
using ARG from global scope as a workaround.
Also force BuildKit extension to be installed, legacy build is no longer
supported.
Closes https://github.com/searxng/searxng/issues/5219
We can leverage the immutable nature of containers to add additional
optimizations.
No debugging or tinkering inside containers, so stripping all unused symbols
inside `venv` should be fine. We are also going to compile the bytecode
ourselves to modify some parameters related to reproducibility.
With these small changes, we have reduced the `venv` layer size by 10MB~
We get some good stuff without uvloop, 13MB~ less of dependencies, 3
minutes of build time for armv7 saved, and we are one step closer to NT
compatibility. Although it's true that theoretically the raw performance
have worsened on network side (we only used uvloop for that), the latest
cpython versions have been improving on asyncio performance.
This commit replaces `pip` in container builds with `uv` pip compat
with a 1:1 parity. The only thing that changes is the installation speed of the
wheels, which seems to be considerably faster, although I haven't been able to
properly quantify this yet.
uv also gives us more tools to manage the cache. We can revert the prior cache
changes in `container.yml` as we won't have duplicated wheels anymore.
Set minor versioning for most of the packages that iterate fast or we know won't
cause problems, and fixed versioning for the rest.
Packages going into bundles should be placed in "dependencies".
The inspection of prod bundles is necessary. Although it does not make a lot of
sense right now, it will be useful in later PR and will give us a reference to
start with.
TypeScript is a superset of JavaScript, converting the entire theme to
TypeScript allows us to receive much more feedback on possible issues made in
package updates or our own typos, furthermore, it allows to transpile properly
to lower specs. This PR couldn't be done in smaller commits, a lot of work
needed to make everything *work properly*:
- A browser baseline has been set that requires minimum **Chromium 93, Firefox
92 and Safari 15** (proper visuals/operation on older browser versions is not
guaranteed)
- LightningCSS now handles minification and prefix creation for CSS.
- All hardcoded polyfills and support for previous browser baseline versions
have been removed.
- Convert codebase to TypeScript.
- Convert IIFE to ESM, handling globals with IIFE is cumbersome, ESM is the
standard for virtually any use of JS nowadays.
- Vite now builds the theme without the need for `vite-plugin-static-copy`.
- `searxng.ready` now accepts an array of conditions for the callback to be
executed.
- Replace `leaflet` with `ol` as there were some issues with proper Vite
bundling.
- Merged `head` with `main` script, as head was too small now.
- Add `assertElement` to properly check the existence of critical DOM elements.
- `searxng.on` renamed to `searxng.listen` with some handling improvements.
All actions are pulled using the version hash, versions are handled by
dependabot, and we'll have control over which actions get updated.
Replaces Trivy scanner with Docker Scout, we have recently begun analyzing the
images there, and the action will keep us in sync about the problems on GHCS
dashboard.
Due to current limitations of `actions/cache`, the cache cannot be overwritten.
In our case, we need to accumulate cached wheels from different architectures.
To solve this, we simply delete the key before storing the cache again.
Building the container currently does not work properly.
When rebuilding several times with `make container`, `version_frozen.py`
is recreated, which wouldn't be an issue if the file’s timestamp was constant.
Now, when creating `version_frozen.py`, it will have the same timestamp as the
commit when it was created. (`version_frozen.py` is moved to a dedicated layer).
Reusing "builder" cache when building "dist" could be slow
(CD reports 2 seconds, but locally I've seen it take up to 10 seconds),
so the Dockerfile is now split and we save a couple steps
by importing the "builder" image directly.
The last changes made it possible to remove the layer cache in "builder",
since the overhead is now greater than building the layers from scratch.
Until now, all "dist" layers were squashed into a single layer,
which in most cases is a good idea
(except for storage/delivery pricing/overhead), but in our case,
since we manage the entire pipeline, we can ignore this
and share layers between builds.
This means (for example) that if we change files unrelated to the container
in several consecutive commits (documentation changes), we don't have to push
the entire image to registry, but only the different layers
(`version_frozen.py` in this example).
The same applies when pulling, as only the layers that have changed
compared to the local layers will be downloaded (that's the theory,
we'll see if this works as expected or if we need to tweak something else).
With this change, the "latest" tag will be visually higher (on registry tag list). Right now, it appears under the "DOCKER_TAG" manifest tag, which can be confusing.
The path to static should be relative (If sxng is served under "/sxng", the static route passed to the client won't be "/sxng/static/..." as expected but "/static/...")
Closes https://github.com/searxng/searxng/issues/5042
While looking at ways to better handle static files, I saw a package that replaces Flask `static_folder` functionality. Not only it's considerably faster, but already includes the capability to serve sidecars without having to intercept. This also replaces the uWSGI folder mapping functionality.
Closes https://github.com/searxng/searxng/issues/4977
I've seen that by default Granian sets a `Cache-Control` header for 1 day IF `GRANIAN_STATIC_PATH_MOUNT` is set. This option is not a hard requirement, but it's set because I found to be faster when serving the static files.
Another thing is that by removing `GRANIAN_STATIC_PATH_MOUNT`, the headers set by the application are present again in the static files, which was not the case before.
Related https://github.com/searxng/searxng/pull/5004
This is one of various PR to refactor the simple theme internally.
Replace eslint tool with Biome. I have been using this for quite some time, and
it will help us to have more consistent and valid code without extending on
other third party plugins.
Removes unused dependencies.
This is a poorly designed instruction, which is hardcoded and cannot be easily modified or maintained on a rolling release sw like ours. This *should* be set in the SearXNG Docker Compose template, not in the image itself.
The OCI format is now used since we no longer have the HEALTHCHECK on the Dockerfile.
Closes https://github.com/searxng/searxng/issues/4906
Closes https://github.com/searxng/searxng/issues/4722
The action does not take into account all cases of how an image is stored, causing errors like the ones below on image pull. I exclude `base` until I find a solution.
*Error: internal error: unable to copy from source ...: initializing source ...: reading manifest ... in ghcr.io/searxng/base: manifest unknown*
I'm not too pleased to reverse this, but issues like https://github.com/searxng/searxng/issues/4792 have not been foreseen, and we can't just turn away. It has become apparent over the last weeks that there are still quite a few people with an incompatible CPU or having SearXNG on some random VM provider who can't (or won't) modify the configuration of their machines to expose the features needed for x86_64v2 march.
As I don't want to trash the work with apko and base images, I thought about trying building Alpine again now that we have all the container related workflow refactored.
There will still be the discussion of whether to use musl and its drawbacks, but right now I don't know any other alternatives.
The nice part of this is that both Dockerfiles (mainline and legacy) can now be unified under the same umbrella again.
Closes https://github.com/searxng/searxng/issues/4792
Closes https://github.com/searxng/searxng/issues/4753
Temporarily remove the -e flag from set to prevent entrypoint.sh from stopping execution if any command returns a non-zero status. This doesn't solve anything but relaxes the script checks.
Related https://github.com/searxng/searxng/issues/4818
That entrypoint is prone to screw things up, especially with permission handling. The new script handles initialization better and fixes some issues like delayed settings update via ENVs and timestamp overwriting, also adjusts what should be copied into the container.
Related https://github.com/searxng/searxng/pull/4721#issuecomment-2850272129
The wolfi-base metapackage includes busybox, ca-certificates-bundle and the package manager. The change is to make the use of base-builder image more flexible.
Instead of using Wolfi base images from cgr.dev and making that mess on the Dockerfile, why don't we build the base images ourselves from Wolfi repos with apko? The intention of this is to simplify the main Dockerfile and avoid having to patch the base image every time, it also simplifies some steps like image ownership management and provides extremely fast builds.
Ivan Gabaldon