Albirew/nyaa-pantsu
Albirew/nyaa-pantsu
Archivé
1
0
Bifurcation 0
Code Activité
Ce dépôt a été archivé le 2022-05-07. Vous pouvez voir ses fichiers ou le cloner, mais pas ouvrir de ticket ou de demandes d'ajout, ni soumettre de changements.
nyaa-pantsu/deploy/ansible/roles/elasticsearch/files/elasticsearch_settings.yml

78 lignes
1,5 Kio
YAML
Brut Vue normale Historique

Create main directory and add elasticsearch configuration
2017-05-17 21:49:42 +02:00
---
settings:
analysis:
analyzer:
Improve search and fix '*' in search box (#871) * Improve ES search The new performance is very good. Some examples on my 1.5gb vm: INFO[0153] Query 'shingeki' took 6 milliseconds. INFO[0125] Query 'アニメ' took 17 milliseconds. INFO[0102] Query 'shingeki -kyojin horrible ' took 12 milliseconds Looking at the criteria we wanted here: https://pad.riseup.net/p/i8DrilHDWRvf, it meets: 1. Fast: sub-100ms for a typical query, sub-50ms is good and sub-20ms is optimal 2. Prefix match: "horrible" finds horriblesubs 3. Substring match? "アニメ" finds "TVアニメ" 4. Position-independent terms ("shingeki kyojin" finds the same as "kyojin shingeki") 5. Works with short term lengths correctly and fast (no in "kyoukai no kanata", 04 in "horrible shingeki 04" etc) 7. (nice to have) search negation: shingeki kyojin -horriblesubs * Use match_all query instead of *, fix *
2017-06-01 00:38:29 +02:00
# Don't use ngram for search otherwise 'horribleexample' would match
# 'horriblesubs'
nyaapantsu_search_analyzer:
tokenizer: standard
Create main directory and add elasticsearch configuration
2017-05-17 21:49:42 +02:00
filter:
- standard
- lowercase
Improve search and fix '*' in search box (#871) * Improve ES search The new performance is very good. Some examples on my 1.5gb vm: INFO[0153] Query 'shingeki' took 6 milliseconds. INFO[0125] Query 'アニメ' took 17 milliseconds. INFO[0102] Query 'shingeki -kyojin horrible ' took 12 milliseconds Looking at the criteria we wanted here: https://pad.riseup.net/p/i8DrilHDWRvf, it meets: 1. Fast: sub-100ms for a typical query, sub-50ms is good and sub-20ms is optimal 2. Prefix match: "horrible" finds horriblesubs 3. Substring match? "アニメ" finds "TVアニメ" 4. Position-independent terms ("shingeki kyojin" finds the same as "kyojin shingeki") 5. Works with short term lengths correctly and fast (no in "kyoukai no kanata", 04 in "horrible shingeki 04" etc) 7. (nice to have) search negation: shingeki kyojin -horriblesubs * Use match_all query instead of *, fix *
2017-06-01 00:38:29 +02:00
char_filter:
- dash_to_underscore
nyaapantsu_index_analyzer:
tokenizer: standard
filter:
- standard
- lowercase
- e_ngram_filter
char_filter:
- dash_to_underscore
filter:
e_ngram_filter:
Revert "Add playbook to download /build latest nyaa, fix k-on elasticsearch issue (#821)" This reverts commit da1e323825c58c022ff47696ffb398aa79cb26f5.
2017-05-29 00:24:04 +02:00
type: edge_ngram
Use minimum 1 ngram (ie: for Gantz O)
2017-06-01 03:20:58 +02:00
min_gram: 1
max_gram: 16
Improve search and fix '*' in search box (#871) * Improve ES search The new performance is very good. Some examples on my 1.5gb vm: INFO[0153] Query 'shingeki' took 6 milliseconds. INFO[0125] Query 'アニメ' took 17 milliseconds. INFO[0102] Query 'shingeki -kyojin horrible ' took 12 milliseconds Looking at the criteria we wanted here: https://pad.riseup.net/p/i8DrilHDWRvf, it meets: 1. Fast: sub-100ms for a typical query, sub-50ms is good and sub-20ms is optimal 2. Prefix match: "horrible" finds horriblesubs 3. Substring match? "アニメ" finds "TVアニメ" 4. Position-independent terms ("shingeki kyojin" finds the same as "kyojin shingeki") 5. Works with short term lengths correctly and fast (no in "kyoukai no kanata", 04 in "horrible shingeki 04" etc) 7. (nice to have) search negation: shingeki kyojin -horriblesubs * Use match_all query instead of *, fix *
2017-06-01 00:38:29 +02:00
char_filter:
dash_to_underscore:
type: pattern_replace
pattern: "([^\\s]+)-(?=[^\\s]+)"
replacement: "$1_"
Create main directory and add elasticsearch configuration
2017-05-17 21:49:42 +02:00
index:
number_of_shards: 1
number_of_replicas: 0
Add reindexing every 5 minutes, and a bunch of other things (#852) * Fix error messages with ES results * Add lsof for debugging * Add torrents table variable to index sukebei * Use elasticsearch alias for hotswapping index * Increase max open files, increase ES heap size * Add reindex script and reindex triggers We use a table to store the actions happened to the torrents table. When the torrents table is INSERTED/UPDATED/DELETED, the trigger kicks in and an entry is made to the reindex_torrents table. The reindex_nyaapantsu.py script is then used to query the reindex_torrents table and apply the correct reindex action to elasticsearch. The entries are then removed for reindex_torrents table. * Reindex every 5 minutes as cronjob
2017-05-31 04:22:12 +02:00
max_result_window: 30000
Create main directory and add elasticsearch configuration
2017-05-17 21:49:42 +02:00
mappings:
torrents:
properties:
Elasticsearch integration (WIP) (#730) * Update mapping to be similar to TorrentJSON * Implement ES search for TorrentParam * Add seeders/leechers/completed to es index * Fix filter, use analyzer * Use ES for the search route * Add upload_id filtering with ES * Create/update ES index on torrent upload/update * Delete from ES index on Delete * Use ES everywhere, fallback to postgres query Use Elasticsearch to search the index whenever a call to searchByQuery is made. Big cleanup needed, but _it werks_. * Only fetch ids from ES, nothing else * Use ColumnUpdate instead of Save * Add FIXME/info to search * Template needs []TorrentJSON not []Torrent
2017-05-26 01:48:14 +02:00
# TODO Consistent ID's type in TorrentJSON
id:
TorrentJSON.ID is uint now, fix weird page sorting (#769) * TorrentJSON.ID is uint now, fix weird page sorting The bug was that ES would sort by ID in a weird manner because the id was a string. The id is now a uint. * Resolved the conflict for future merging
2017-05-27 03:54:41 +02:00
type: long
Elasticsearch integration (WIP) (#730) * Update mapping to be similar to TorrentJSON * Implement ES search for TorrentParam * Add seeders/leechers/completed to es index * Fix filter, use analyzer * Use ES for the search route * Add upload_id filtering with ES * Create/update ES index on torrent upload/update * Delete from ES index on Delete * Use ES everywhere, fallback to postgres query Use Elasticsearch to search the index whenever a call to searchByQuery is made. Big cleanup needed, but _it werks_. * Only fetch ids from ES, nothing else * Use ColumnUpdate instead of Save * Add FIXME/info to search * Template needs []TorrentJSON not []Torrent
2017-05-26 01:48:14 +02:00
name:
Create main directory and add elasticsearch configuration
2017-05-17 21:49:42 +02:00
type: text
Improve search and fix '*' in search box (#871) * Improve ES search The new performance is very good. Some examples on my 1.5gb vm: INFO[0153] Query 'shingeki' took 6 milliseconds. INFO[0125] Query 'アニメ' took 17 milliseconds. INFO[0102] Query 'shingeki -kyojin horrible ' took 12 milliseconds Looking at the criteria we wanted here: https://pad.riseup.net/p/i8DrilHDWRvf, it meets: 1. Fast: sub-100ms for a typical query, sub-50ms is good and sub-20ms is optimal 2. Prefix match: "horrible" finds horriblesubs 3. Substring match? "アニメ" finds "TVアニメ" 4. Position-independent terms ("shingeki kyojin" finds the same as "kyojin shingeki") 5. Works with short term lengths correctly and fast (no in "kyoukai no kanata", 04 in "horrible shingeki 04" etc) 7. (nice to have) search negation: shingeki kyojin -horriblesubs * Use match_all query instead of *, fix *
2017-06-01 00:38:29 +02:00
analyzer: nyaapantsu_index_analyzer
fields:
raw:
type: keyword
Create main directory and add elasticsearch configuration
2017-05-17 21:49:42 +02:00
category:
Elasticsearch integration (WIP) (#730) * Update mapping to be similar to TorrentJSON * Implement ES search for TorrentParam * Add seeders/leechers/completed to es index * Fix filter, use analyzer * Use ES for the search route * Add upload_id filtering with ES * Create/update ES index on torrent upload/update * Delete from ES index on Delete * Use ES everywhere, fallback to postgres query Use Elasticsearch to search the index whenever a call to searchByQuery is made. Big cleanup needed, but _it werks_. * Only fetch ids from ES, nothing else * Use ColumnUpdate instead of Save * Add FIXME/info to search * Template needs []TorrentJSON not []Torrent
2017-05-26 01:48:14 +02:00
type: text
Create main directory and add elasticsearch configuration
2017-05-17 21:49:42 +02:00
sub_category:
Elasticsearch integration (WIP) (#730) * Update mapping to be similar to TorrentJSON * Implement ES search for TorrentParam * Add seeders/leechers/completed to es index * Fix filter, use analyzer * Use ES for the search route * Add upload_id filtering with ES * Create/update ES index on torrent upload/update * Delete from ES index on Delete * Use ES everywhere, fallback to postgres query Use Elasticsearch to search the index whenever a call to searchByQuery is made. Big cleanup needed, but _it werks_. * Only fetch ids from ES, nothing else * Use ColumnUpdate instead of Save * Add FIXME/info to search * Template needs []TorrentJSON not []Torrent
2017-05-26 01:48:14 +02:00
type: text
Create main directory and add elasticsearch configuration
2017-05-17 21:49:42 +02:00
status:
Fix typo in elasticsearch_settings.yml file
2017-05-17 22:25:30 +02:00
type: long
Elasticsearch integration (WIP) (#730) * Update mapping to be similar to TorrentJSON * Implement ES search for TorrentParam * Add seeders/leechers/completed to es index * Fix filter, use analyzer * Use ES for the search route * Add upload_id filtering with ES * Create/update ES index on torrent upload/update * Delete from ES index on Delete * Use ES everywhere, fallback to postgres query Use Elasticsearch to search the index whenever a call to searchByQuery is made. Big cleanup needed, but _it werks_. * Only fetch ids from ES, nothing else * Use ColumnUpdate instead of Save * Add FIXME/info to search * Template needs []TorrentJSON not []Torrent
2017-05-26 01:48:14 +02:00
hash:
type: text
date:
type: date
uploader_id:
type: long
downloads:
type: long
seeders:
type: long
leechers:
type: long
completed:
type: long
filesize:
type: long
Country flags (language) for torrents. (#970) * Add flags for torrents Add a new field, .Language, to the Torrent model, and a new package, torrentLanguages, which maps languages to flags. Added also a flag icon pack from googlei18n/region-flags, with (mostly) public domain flags from Wikipedia. * Optimize flags * Use FlagSprites CSS instead of .png files * Only use flags for languages we support * Add test for CSS flags Ensure that we have all the flags for the languages we support. * Add AdditionalLanguages field to config This allows us to support additional languages for new uploaded torrents, even if we have no translation for it. * Minor CSS fix * Add "other" and "multiple" torrent languages Also removed the TorrentLanguage struct, as it wasn't much useful. * Fix test * Add colspan=2 to category when language is empty Also hide the language column if empty. * Add lang field to search. Hopefully it works with Elasticsearch as well, but I haven't tested (lol Java) * Add language field to ES index and settings * Add language column to JS template * Add keyword type to language ES field * Remove 'raw' from keyword * Set "simple" analyzer on language * Document .Language field on Torrent model
2017-06-12 01:14:26 +02:00
language:
type: text
analyzer: simple
Copier le lien permanent