* Improve ES search
The new performance is very good.
Some examples on my 1.5gb vm:
INFO[0153] Query 'shingeki' took 6 milliseconds.
INFO[0125] Query 'アニメ' took 17 milliseconds.
INFO[0102] Query 'shingeki -kyojin horrible ' took 12 milliseconds
Looking at the criteria we wanted here:
https://pad.riseup.net/p/i8DrilHDWRvf, it meets:
1. Fast: sub-100ms for a typical query, sub-50ms is good and sub-20ms is
optimal
2. Prefix match: "horrible" finds horriblesubs
3. Substring match? "アニメ" finds "TVアニメ"
4. Position-independent terms ("shingeki kyojin" finds the same as
"kyojin shingeki")
5. Works with short term lengths correctly and fast (no in "kyoukai no
kanata", 04 in "horrible shingeki 04" etc)
7. (nice to have) search negation: shingeki kyojin -horriblesubs
* Use match_all query instead of *, fix *
* Fix error messages with ES results
* Add lsof for debugging
* Add torrents table variable to index sukebei
* Use elasticsearch alias for hotswapping index
* Increase max open files, increase ES heap size
* Add reindex script and reindex triggers
We use a table to store the actions happened to the torrents table.
When the torrents table is INSERTED/UPDATED/DELETED, the trigger kicks
in and an entry is made to the reindex_torrents table.
The reindex_nyaapantsu.py script is then used to query the
reindex_torrents table and apply the correct reindex action to
elasticsearch. The entries are then removed for reindex_torrents table.
* Reindex every 5 minutes as cronjob
* TorrentJSON.ID is uint now, fix weird page sorting
The bug was that ES would sort by ID in a weird manner because the id
was a string. The id is now a uint.
* Resolved the conflict for future merging
* Update mapping to be similar to TorrentJSON
* Implement ES search for TorrentParam
* Add seeders/leechers/completed to es index
* Fix filter, use analyzer
* Use ES for the search route
* Add upload_id filtering with ES
* Create/update ES index on torrent upload/update
* Delete from ES index on Delete
* Use ES everywhere, fallback to postgres query
Use Elasticsearch to search the index whenever a call to searchByQuery
is made. Big cleanup needed, but _it werks_.
* Only fetch ids from ES, nothing else
* Use ColumnUpdate instead of Save
* Add FIXME/info to search
* Template needs []TorrentJSON not []Torrent
* Use ModeratorDir variable
* Rename cookieHelper to cookie_helper for consistency
* Use named constant instead of literals
* Fix ability to upload when uploads are disabled
The old code let people upload under the right conditions when uploads
were disabled. (ie: User is banned and config.AdminAreStillAllowedTo is
false)
* Increase timeout (fixes#517)
* Fix inconsistent indentation *.{js, css} (fix#583)
* Fix negative page
Temporary fix. The issue was that going to a negative page caused the
sql query to have a negative offset. This caused an error in the
database query.
We need to cleanup this code, but this will work for now.
* Fix wrong PG_DATA directory due to upgrade to 9.6
* Add server status link to FAQ
* Fix failing tests
* Clarify group_vars/all and hosts doc
* Add a wrapper to protect /mod route
* Fix login page not showing form errors
* Keep naming consistent
* Remove execute bit from files
* Default to DefaultLanguage (without passing it to the func)
* Remove commented code
* Use Content-Type to get language json
* Lines of 400 characters is dumb
* Update new repo in README
* Remove useless script since we fallback to a defaultlang
* Fix fallback language panic
* Fix uninitialized MaxPerPage when not in querystr
The issue was that the req.MaxPerPage was not set (default to 0) when
the query string didn't include "max". This makes the server query the
whole db since the resulting limit is 0.
* Fix creating empty torrents (only worked once)
* Lines of 400 characters is still dumb
Creates the user and the database provided by the environment variable.
These are currently duplicated in postgres.env so we might want to find
a way to have them in only one place.
I tried my best at keeping the pg_hba.conf file secure for the server,
but I am no expert so it'd be great if someone could check it out.
tomleb
tomleb
Eliot Whalan
alucard0134
PantsuDev