* Add flags for torrents
Add a new field, .Language, to the Torrent model, and a new package,
torrentLanguages, which maps languages to flags. Added also a flag icon pack
from googlei18n/region-flags, with (mostly) public domain flags from Wikipedia.
* Optimize flags
* Use FlagSprites CSS instead of .png files
* Only use flags for languages we support
* Add test for CSS flags
Ensure that we have all the flags for the languages we support.
* Add AdditionalLanguages field to config
This allows us to support additional languages for new uploaded torrents,
even if we have no translation for it.
* Minor CSS fix
* Add "other" and "multiple" torrent languages
Also removed the TorrentLanguage struct, as it wasn't much useful.
* Fix test
* Add colspan=2 to category when language is empty
Also hide the language column if empty.
* Add lang field to search.
Hopefully it works with Elasticsearch as well, but I haven't tested
(lol Java)
* Add language field to ES index and settings
* Add language column to JS template
* Add keyword type to language ES field
* Remove 'raw' from keyword
* Set "simple" analyzer on language
* Document .Language field on Torrent model
* Improve ES search
The new performance is very good.
Some examples on my 1.5gb vm:
INFO[0153] Query 'shingeki' took 6 milliseconds.
INFO[0125] Query 'アニメ' took 17 milliseconds.
INFO[0102] Query 'shingeki -kyojin horrible ' took 12 milliseconds
Looking at the criteria we wanted here:
https://pad.riseup.net/p/i8DrilHDWRvf, it meets:
1. Fast: sub-100ms for a typical query, sub-50ms is good and sub-20ms is
optimal
2. Prefix match: "horrible" finds horriblesubs
3. Substring match? "アニメ" finds "TVアニメ"
4. Position-independent terms ("shingeki kyojin" finds the same as
"kyojin shingeki")
5. Works with short term lengths correctly and fast (no in "kyoukai no
kanata", 04 in "horrible shingeki 04" etc)
7. (nice to have) search negation: shingeki kyojin -horriblesubs
* Use match_all query instead of *, fix *
* Fix error messages with ES results
* Add lsof for debugging
* Add torrents table variable to index sukebei
* Use elasticsearch alias for hotswapping index
* Increase max open files, increase ES heap size
* Add reindex script and reindex triggers
We use a table to store the actions happened to the torrents table.
When the torrents table is INSERTED/UPDATED/DELETED, the trigger kicks
in and an entry is made to the reindex_torrents table.
The reindex_nyaapantsu.py script is then used to query the
reindex_torrents table and apply the correct reindex action to
elasticsearch. The entries are then removed for reindex_torrents table.
* Reindex every 5 minutes as cronjob
* TorrentJSON.ID is uint now, fix weird page sorting
The bug was that ES would sort by ID in a weird manner because the id
was a string. The id is now a uint.
* Resolved the conflict for future merging
* Update mapping to be similar to TorrentJSON
* Implement ES search for TorrentParam
* Add seeders/leechers/completed to es index
* Fix filter, use analyzer
* Use ES for the search route
* Add upload_id filtering with ES
* Create/update ES index on torrent upload/update
* Delete from ES index on Delete
* Use ES everywhere, fallback to postgres query
Use Elasticsearch to search the index whenever a call to searchByQuery
is made. Big cleanup needed, but _it werks_.
* Only fetch ids from ES, nothing else
* Use ColumnUpdate instead of Save
* Add FIXME/info to search
* Template needs []TorrentJSON not []Torrent
