Albirew/nyaa-pantsu
Archivé
1
0
Bifurcation 0
Graphe des révisions

34 commits

Auteur SHA1 Message Date
tomleb
45da57fd0d Playbook to install latest nyaa straight from github (#843)
* Playbook to install latest nyaa straight from github

* Backup old nyaa dir before pulling latest
2017-05-30 21:48:20 -05:00
tomleb
360b35a08f Add reindexing every 5 minutes, and a bunch of other things (#852)
* Fix error messages with ES results

* Add lsof for debugging

* Add torrents table variable to index sukebei

* Use elasticsearch alias for hotswapping index

* Increase max open files, increase ES heap size

* Add reindex script and reindex triggers

We use a table to store the actions happened to the torrents table.
When the torrents table is INSERTED/UPDATED/DELETED, the trigger kicks
in and an entry is made to the reindex_torrents table.

The reindex_nyaapantsu.py script is then used to query the
reindex_torrents table and apply the correct reindex action to
elasticsearch. The entries are then removed for reindex_torrents table.

* Reindex every 5 minutes as cronjob
2017-05-30 21:22:12 -05:00
tomleb
0d70ed3038 Dump database in format usable by pg_restore 2017-05-30 21:20:22 -05:00
Eliot Whalan
5bcda5c9a1
Revert "Add playbook to download /build latest nyaa, fix k-on elasticsearch issue (#821)"
This reverts commit da1e323825.
2017-05-29 08:24:04 +10:00
tomleb
da1e323825 Add playbook to download /build latest nyaa, fix k-on elasticsearch issue (#821)
* Install nyaa from latest github commit

* Add install playbook, fix k-on search
2017-05-29 07:54:24 +10:00
tomleb
d6c50f5640 TorrentJSON.ID is uint now, fix weird page sorting (#769)
* TorrentJSON.ID is uint now, fix weird page sorting

The bug was that ES would sort by ID in a weird manner because the id
was a string. The id is now a uint.

* Resolved the conflict for future merging
2017-05-27 11:54:41 +10:00
tomleb
4d87ba3b3e Decrease idle time on pgpool 2017-05-26 16:15:00 -04:00
tomleb
456eac68ec Fix wrong ram amount for pgpool 2017-05-26 08:56:50 -04:00
tomleb
90356b22bc Install firewalld before starting it
Firewalld was installed on vagrant centos by default but not on server.
2017-05-26 08:56:49 -04:00
tomleb
9c81c85325 Index seeders,leechers and completed into es 2017-05-26 08:56:49 -04:00
tomleb
f22d11b35d Elasticsearch integration (WIP) (#730)
* Update mapping to be similar to TorrentJSON

* Implement ES search for TorrentParam

* Add seeders/leechers/completed to es index

* Fix filter, use analyzer

* Use ES for the search route

* Add upload_id filtering with ES

* Create/update ES index on torrent upload/update

* Delete from ES index on Delete

* Use ES everywhere, fallback to postgres query

Use Elasticsearch to search the index whenever a call to searchByQuery
is made. Big cleanup needed, but _it werks_.

* Only fetch ids from ES, nothing else

* Use ColumnUpdate instead of Save

* Add FIXME/info to search

* Template needs []TorrentJSON not []Torrent
2017-05-26 09:48:14 +10:00
alucard0134
66fe20bab8 Update backup.sh 2017-05-22 19:05:50 -05:00
tomleb
7eee47b0d3 Fix bug, remove literals (#629)
* Use ModeratorDir variable

* Rename cookieHelper to cookie_helper for consistency

* Use named constant instead of literals

* Fix ability to upload when uploads are disabled

The old code let people upload under the right conditions when uploads
were disabled. (ie: User is banned and config.AdminAreStillAllowedTo is
false)

* Increase timeout (fixes #517)

* Fix inconsistent indentation *.{js, css} (fix #583)

* Fix negative page

Temporary fix. The issue was that going to a negative page caused the
sql query to have a negative offset. This caused an error in the
database query.

We need to cleanup this code, but this will work for now.

* Fix wrong PG_DATA directory due to upgrade to 9.6

* Add server status link to FAQ

* Fix failing tests

* Clarify group_vars/all and hosts doc

* Add a wrapper to protect /mod route

* Fix login page not showing form errors
2017-05-20 09:10:16 +10:00
tomleb
d757a9006a Fetch index name from environment 2017-05-17 19:05:58 -04:00
tomleb
94af9997e0 Fix typo in elasticsearch_settings.yml file 2017-05-17 19:05:58 -04:00
tomleb
4609176785 Create main directory and add elasticsearch configuration 2017-05-17 19:05:58 -04:00
tomleb
79e7d147af Upgrade to postgresql 9.6 2017-05-17 19:05:58 -04:00
tomleb
a6d34ef582 Start add elasticsearch 2017-05-17 19:05:58 -04:00
tomleb
f40de4c8dc Go get requires git installed for github repos 2017-05-17 19:05:58 -04:00
tomleb
b6c7955925 Add firewalld service 2017-05-17 19:05:58 -04:00
tomleb
bd8cf8cfcd Install golang 1.8.1 with ansible 2017-05-17 19:05:58 -04:00
tomleb
24fe8f2689 Add extensions support for postgresql 2017-05-17 19:05:58 -04:00
tomleb
d8b82b52e0 Distribute the database with daily backups
Daily cron job that dump the database, xz it, sign it with GPG,
create a torrent file and seed it with rtorrent.
2017-05-17 19:05:58 -04:00
tomleb
510ef57a0a Make sure postgresql uses pgpool's port 2017-05-17 19:05:58 -04:00
tomleb
c54b60594b Tune pgpool variables, we're going fast 2017-05-08 23:42:04 -04:00
tomleb
a29d612678 Fix auth problem with pgpool 2017-05-08 23:42:04 -04:00
tomleb
67947cc4aa Add pgpool, pg_hba.conf needs to be modified 2017-05-08 23:42:04 -04:00
tomleb
aaf3bd05d3 Use rtorrent to seed the torrents (fails on systemd) 2017-05-08 23:42:04 -04:00
tomleb
dff098da6a Create .torrents file from .xz and .sig 2017-05-08 23:42:04 -04:00
tomleb
7456fc8a03 Daily backup of db zipped and signed 2017-05-08 23:42:04 -04:00
tomleb
aa91d89bce Add .pgpass file to avoid entering db password 2017-05-08 23:42:04 -04:00
tomleb
1abb2fa7fe Remove TODO 2017-05-08 23:42:04 -04:00
tomleb
55c7252327 Automate postgresql setup
Creates the user and the database provided by the environment variable.
These are currently duplicated in postgres.env so we might want to find
a way to have them in only one place.

I tried my best at keeping the pg_hba.conf file secure for the server,
but I am no expert so it'd be great if someone could check it out.
2017-05-08 23:42:04 -04:00
tomleb
b178f12bab Start working on server automation with ansible 2017-05-08 23:42:04 -04:00