When a user had hidden torrents, they were still listed on his user page even if it's not the actual user who is viewing it.
That's why I added a new function for searching which filter out hidden torrents.
Now when a user go to his own page (or a moderator), he can see all his torrents.
However if another person visit the page, only non hidden torrents will be listed.
I added the hidden parameter in ES database, it permits also to use the search instead of SQL when listing torrents on the user page. Less load time, go with the idea that we should use ES everywhere when needed.
* Increase max_result_window to 5000000
This is until we switch to the scroll API. We can't access pages after
~2000 without increasing it.
* Increase jvm heap ram
* Prevent elasticsearch from being swapped
* Add flags for torrents
Add a new field, .Language, to the Torrent model, and a new package,
torrentLanguages, which maps languages to flags. Added also a flag icon pack
from googlei18n/region-flags, with (mostly) public domain flags from Wikipedia.
* Optimize flags
* Use FlagSprites CSS instead of .png files
* Only use flags for languages we support
* Add test for CSS flags
Ensure that we have all the flags for the languages we support.
* Add AdditionalLanguages field to config
This allows us to support additional languages for new uploaded torrents,
even if we have no translation for it.
* Minor CSS fix
* Add "other" and "multiple" torrent languages
Also removed the TorrentLanguage struct, as it wasn't much useful.
* Fix test
* Add colspan=2 to category when language is empty
Also hide the language column if empty.
* Add lang field to search.
Hopefully it works with Elasticsearch as well, but I haven't tested
(lol Java)
* Add language field to ES index and settings
* Add language column to JS template
* Add keyword type to language ES field
* Remove 'raw' from keyword
* Set "simple" analyzer on language
* Document .Language field on Torrent model
* Improve ES search
The new performance is very good.
Some examples on my 1.5gb vm:
INFO[0153] Query 'shingeki' took 6 milliseconds.
INFO[0125] Query 'アニメ' took 17 milliseconds.
INFO[0102] Query 'shingeki -kyojin horrible ' took 12 milliseconds
Looking at the criteria we wanted here:
https://pad.riseup.net/p/i8DrilHDWRvf, it meets:
1. Fast: sub-100ms for a typical query, sub-50ms is good and sub-20ms is
optimal
2. Prefix match: "horrible" finds horriblesubs
3. Substring match? "アニメ" finds "TVアニメ"
4. Position-independent terms ("shingeki kyojin" finds the same as
"kyojin shingeki")
5. Works with short term lengths correctly and fast (no in "kyoukai no
kanata", 04 in "horrible shingeki 04" etc)
7. (nice to have) search negation: shingeki kyojin -horriblesubs
* Use match_all query instead of *, fix *
* Fix error messages with ES results
* Add lsof for debugging
* Add torrents table variable to index sukebei
* Use elasticsearch alias for hotswapping index
* Increase max open files, increase ES heap size
* Add reindex script and reindex triggers
We use a table to store the actions happened to the torrents table.
When the torrents table is INSERTED/UPDATED/DELETED, the trigger kicks
in and an entry is made to the reindex_torrents table.
The reindex_nyaapantsu.py script is then used to query the
reindex_torrents table and apply the correct reindex action to
elasticsearch. The entries are then removed for reindex_torrents table.
* Reindex every 5 minutes as cronjob
* TorrentJSON.ID is uint now, fix weird page sorting
The bug was that ES would sort by ID in a weird manner because the id
was a string. The id is now a uint.
* Resolved the conflict for future merging
* Update mapping to be similar to TorrentJSON
* Implement ES search for TorrentParam
* Add seeders/leechers/completed to es index
* Fix filter, use analyzer
* Use ES for the search route
* Add upload_id filtering with ES
* Create/update ES index on torrent upload/update
* Delete from ES index on Delete
* Use ES everywhere, fallback to postgres query
Use Elasticsearch to search the index whenever a call to searchByQuery
is made. Big cleanup needed, but _it werks_.
* Only fetch ids from ES, nothing else
* Use ColumnUpdate instead of Save
* Add FIXME/info to search
* Template needs []TorrentJSON not []Torrent
* Use ModeratorDir variable
* Rename cookieHelper to cookie_helper for consistency
* Use named constant instead of literals
* Fix ability to upload when uploads are disabled
The old code let people upload under the right conditions when uploads
were disabled. (ie: User is banned and config.AdminAreStillAllowedTo is
false)
* Increase timeout (fixes#517)
* Fix inconsistent indentation *.{js, css} (fix#583)
* Fix negative page
Temporary fix. The issue was that going to a negative page caused the
sql query to have a negative offset. This caused an error in the
database query.
We need to cleanup this code, but this will work for now.
* Fix wrong PG_DATA directory due to upgrade to 9.6
* Add server status link to FAQ
* Fix failing tests
* Clarify group_vars/all and hosts doc
* Add a wrapper to protect /mod route
* Fix login page not showing form errors
Creates the user and the database provided by the environment variable.
These are currently duplicated in postgres.env so we might want to find
a way to have them in only one place.
I tried my best at keeping the pg_hba.conf file secure for the server,
but I am no expert so it'd be great if someone could check it out.
