This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
package models
2017-05-05 14:20:51 +02:00
import (
2017-05-27 00:45:18 +02:00
"context"
This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
"errors"
2017-05-09 16:42:12 +02:00
"fmt"
2017-05-05 14:20:51 +02:00
"html/template"
2017-05-15 23:45:47 +02:00
"path/filepath"
2017-05-27 00:45:18 +02:00
"reflect"
2017-05-05 14:20:51 +02:00
"strconv"
"strings"
"time"
2017-05-27 00:45:18 +02:00
2017-05-26 01:48:14 +02:00
elastic "gopkg.in/olivere/elastic.v5"
This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
"net/http"
2017-05-27 00:45:18 +02:00
"net/url"
2017-05-26 01:48:14 +02:00
"github.com/NyaaPantsu/nyaa/config"
2017-07-14 17:53:37 +02:00
"github.com/NyaaPantsu/nyaa/utils/cache"
2017-07-02 16:54:55 +02:00
"github.com/NyaaPantsu/nyaa/utils/format"
"github.com/NyaaPantsu/nyaa/utils/log"
"github.com/NyaaPantsu/nyaa/utils/sanitize"
2017-05-26 01:48:14 +02:00
"github.com/bradfitz/slice"
2017-07-17 21:32:55 +02:00
"github.com/fatih/structs"
2017-05-05 14:20:51 +02:00
)
2017-05-20 01:10:16 +02:00
const (
2017-05-26 12:12:52 +02:00
// TorrentStatusNormal Int for Torrent status normal
TorrentStatusNormal = 1
// TorrentStatusRemake Int for Torrent status remake
TorrentStatusRemake = 2
// TorrentStatusTrusted Int for Torrent status trusted
2017-05-25 23:35:37 +02:00
TorrentStatusTrusted = 3
2017-05-26 12:12:52 +02:00
// TorrentStatusAPlus Int for Torrent status a+
TorrentStatusAPlus = 4
// TorrentStatusBlocked Int for Torrent status locked
2017-05-25 23:35:37 +02:00
TorrentStatusBlocked = 5
2017-05-20 01:10:16 +02:00
)
2017-05-26 12:12:52 +02:00
// Torrent model
Consistency, formatting, error checking, cleanup, and a couple bug fixes (#245)
* Checkpoint: it builds
The config, db, model, network, os, and public packages have had some
fixes to glaringly obvious flaws, dead code removed, and stylistic
changes.
* Style changes and old code removal in router
Router needs a lot of work done to its (lack of) error handling.
* Dead code removal and style changes
Now up to util/email/email.go. After I'm finished with the initial sweep
I'll go back and fix error handling and security issues. Then I'll fix
the broken API. Then I'll go through to add documentation and fix code
visibility.
* Finish dead code removal and style changes
Vendored libraries not touched. Everything still needs security fixes
and documentation. There's also one case of broken functionality.
* Fix accidental find-and-replace
* Style, error checking, saftey, bug fix changes
* Redo error checking erased during merge
* Re-add merge-erased fix. Make Safe safe.
2017-05-10 04:34:40 +02:00
type Torrent struct {
2017-05-11 00:06:21 +02:00
ID uint ` gorm:"column:torrent_id;primary_key" `
Name string ` gorm:"column:torrent_name" `
2017-06-14 08:42:15 +02:00
Hash string ` gorm:"column:torrent_hash;unique" `
2017-05-11 00:06:21 +02:00
Category int ` gorm:"column:category" `
SubCategory int ` gorm:"column:sub_category" `
Status int ` gorm:"column:status" `
2017-05-27 20:33:40 +02:00
Hidden bool ` gorm:"column:hidden" `
2017-05-11 00:06:21 +02:00
Date time . Time ` gorm:"column:date" `
UploaderID uint ` gorm:"column:uploader" `
Stardom int ` gorm:"column:stardom" `
Filesize int64 ` gorm:"column:filesize" `
Description string ` gorm:"column:description" `
WebsiteLink string ` gorm:"column:website_link" `
2017-05-27 00:45:18 +02:00
Trackers string ` gorm:"column:trackers" `
Tag Search + Tests + Search slight refactor [DONE] (#1342)
* Tag Search + Tests + Search slight refactor
First commit improving search. Different struct have their own file with their tests. This way of separating struct by files is inspired by the go packages I've seen so far.
Added new behaviour as discussed in #1334
* fix fallback to ES
* Added some comments to explain PG fallback + log err moved
* Refactored search
Nearly fully covered
WhereParams struct has disappeared for Query struct instead
In DB model, we use an interface implementing Query struct methods
* 1rst Refactor of Tags (WTF already?!)
Prepare Tags for the refactored system. Now there will be descriptive tags for a particular release (ecchi, BDSM, ....) and typed tags.
Typed tags are tags relevant to all torrents and can be limited to some input value. For example, video quality is a typed tag limited to some values (hd, full hd, sd, ...). In the same way, anidbid is also a typed tag but doesn't have default values.
Furthermore, the location storage of tags have changed, now accepted descriptive tags are stored in the torrents table in the column "tags" and they are separated by commas.
In the opposite, accepted typed tags can have have their own column in the torrents table. For example, anidbid, vndbid will populate the column DbID when accepted. On the other hand, videoquality will populate the same way as descriptive tags.
This behaviour depends on the callbackOnType function in tag/helpers.go
* fix for modtools :')
* Added anidb, vndb, dlsite & vmdb id fields in torrent model.
Tags don't have an accepted field anymore.
Accepted Tags are in torrent.AcceptedTags and non-accepted ones in torrrent.Tags.
New Helper + New Changelog for translation string.
* New upload/edit form for torrent tags.
Now the inputs are dynamically generated by the helper tag_form.
No more modal window in those form, only inputs.
Support of tags in API
New translation string for the link to the modal on torrent view.
More comments in the functions for tags
* Improving how config for tags work. Adding a test on them with understandable messages.
Config for tags have now a Field attribute which is linked to the Torrent model. For example anidbid tag type has now a AnidbID field in config which is the name of the field in torrent model (AnidbID). Every new tag type need to have a field attribute with its counterpart in torrent Model.
Fixing some errors
* Fix compile error + Tests Errors
* Improve performance by caching the list of tags with an index
Adding/removing tags works/tested
New translation strings
TODO: test/fix adding tag on upload/edit
* Mini fix to display video quality
+ tags works/tested on modo edit
* Fix editing tags on modpanel
* Edit tags works
* Add translation string
* Add search backend for tags.
?tags=xxx,eee,ddd
?anidb=21
?vndb=23
?vgmdb=24
?vq=full_hd
* Fix Ajax tag Removal&Add
* Added form for descriptive tags
* Forgot to add the link between database and form for descriptive tags.
* Adding the increase/decrease pantsu for descriptive tags
* Fix #1370
* When you actually forgot to commit files after having forgotten commits
2017-08-22 03:48:10 +02:00
// Torrent Details
AnidbID uint ` gorm:"column:anidbid" `
VndbID uint ` gorm:"column:vndbid" `
VgmdbID uint ` gorm:"column:vgmdbid" `
2017-09-17 21:27:20 +02:00
Dlsite string ` gorm:"column:dlsite" `
Tag Search + Tests + Search slight refactor [DONE] (#1342)
* Tag Search + Tests + Search slight refactor
First commit improving search. Different struct have their own file with their tests. This way of separating struct by files is inspired by the go packages I've seen so far.
Added new behaviour as discussed in #1334
* fix fallback to ES
* Added some comments to explain PG fallback + log err moved
* Refactored search
Nearly fully covered
WhereParams struct has disappeared for Query struct instead
In DB model, we use an interface implementing Query struct methods
* 1rst Refactor of Tags (WTF already?!)
Prepare Tags for the refactored system. Now there will be descriptive tags for a particular release (ecchi, BDSM, ....) and typed tags.
Typed tags are tags relevant to all torrents and can be limited to some input value. For example, video quality is a typed tag limited to some values (hd, full hd, sd, ...). In the same way, anidbid is also a typed tag but doesn't have default values.
Furthermore, the location storage of tags have changed, now accepted descriptive tags are stored in the torrents table in the column "tags" and they are separated by commas.
In the opposite, accepted typed tags can have have their own column in the torrents table. For example, anidbid, vndbid will populate the column DbID when accepted. On the other hand, videoquality will populate the same way as descriptive tags.
This behaviour depends on the callbackOnType function in tag/helpers.go
* fix for modtools :')
* Added anidb, vndb, dlsite & vmdb id fields in torrent model.
Tags don't have an accepted field anymore.
Accepted Tags are in torrent.AcceptedTags and non-accepted ones in torrrent.Tags.
New Helper + New Changelog for translation string.
* New upload/edit form for torrent tags.
Now the inputs are dynamically generated by the helper tag_form.
No more modal window in those form, only inputs.
Support of tags in API
New translation string for the link to the modal on torrent view.
More comments in the functions for tags
* Improving how config for tags work. Adding a test on them with understandable messages.
Config for tags have now a Field attribute which is linked to the Torrent model. For example anidbid tag type has now a AnidbID field in config which is the name of the field in torrent model (AnidbID). Every new tag type need to have a field attribute with its counterpart in torrent Model.
Fixing some errors
* Fix compile error + Tests Errors
* Improve performance by caching the list of tags with an index
Adding/removing tags works/tested
New translation strings
TODO: test/fix adding tag on upload/edit
* Mini fix to display video quality
+ tags works/tested on modo edit
* Fix editing tags on modpanel
* Edit tags works
* Add translation string
* Add search backend for tags.
?tags=xxx,eee,ddd
?anidb=21
?vndb=23
?vgmdb=24
?vq=full_hd
* Fix Ajax tag Removal&Add
* Added form for descriptive tags
* Forgot to add the link between database and form for descriptive tags.
* Adding the increase/decrease pantsu for descriptive tags
* Fix #1370
* When you actually forgot to commit files after having forgotten commits
2017-08-22 03:48:10 +02:00
VideoQuality string ` gorm:"column:videoquality" `
AcceptedTags string ` gorm:"column:tags" `
2017-06-12 01:14:26 +02:00
// Indicates the language of the torrent's content (eg. subs, dubs, raws, manga TLs)
2017-06-14 08:42:15 +02:00
Language string ` gorm:"column:language" `
DeletedAt * time . Time
2017-05-08 19:26:29 +02:00
2017-05-20 13:45:15 +02:00
Uploader * User ` gorm:"AssociationForeignKey:UploaderID;ForeignKey:user_id" `
2017-05-10 13:32:45 +02:00
OldUploader string ` gorm:"-" ` // ???????
2017-05-09 01:56:57 +02:00
OldComments [ ] OldComment ` gorm:"ForeignKey:torrent_id" `
Comments [ ] Comment ` gorm:"ForeignKey:torrent_id" `
2017-08-01 11:17:25 +02:00
Tags Tags ` gorm:"-" `
2017-06-16 01:13:09 +02:00
Scrape * Scrape ` gorm:"AssociationForeignKey:ID;ForeignKey:torrent_id" `
FileList [ ] File ` gorm:"ForeignKey:torrent_id" `
2017-07-02 23:53:23 +02:00
Languages [ ] string ` gorm:"-" ` // This is parsed when retrieved from db
2017-06-16 01:13:09 +02:00
}
2017-05-10 19:29:35 +02:00
This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
/ * We need a JSON object instead of a Gorm structure because magnet URLs are
not in the database and have to be generated dynamically * /
// TorrentJSON for torrent model in json for api
type TorrentJSON struct {
Tag Search + Tests + Search slight refactor [DONE] (#1342)
* Tag Search + Tests + Search slight refactor
First commit improving search. Different struct have their own file with their tests. This way of separating struct by files is inspired by the go packages I've seen so far.
Added new behaviour as discussed in #1334
* fix fallback to ES
* Added some comments to explain PG fallback + log err moved
* Refactored search
Nearly fully covered
WhereParams struct has disappeared for Query struct instead
In DB model, we use an interface implementing Query struct methods
* 1rst Refactor of Tags (WTF already?!)
Prepare Tags for the refactored system. Now there will be descriptive tags for a particular release (ecchi, BDSM, ....) and typed tags.
Typed tags are tags relevant to all torrents and can be limited to some input value. For example, video quality is a typed tag limited to some values (hd, full hd, sd, ...). In the same way, anidbid is also a typed tag but doesn't have default values.
Furthermore, the location storage of tags have changed, now accepted descriptive tags are stored in the torrents table in the column "tags" and they are separated by commas.
In the opposite, accepted typed tags can have have their own column in the torrents table. For example, anidbid, vndbid will populate the column DbID when accepted. On the other hand, videoquality will populate the same way as descriptive tags.
This behaviour depends on the callbackOnType function in tag/helpers.go
* fix for modtools :')
* Added anidb, vndb, dlsite & vmdb id fields in torrent model.
Tags don't have an accepted field anymore.
Accepted Tags are in torrent.AcceptedTags and non-accepted ones in torrrent.Tags.
New Helper + New Changelog for translation string.
* New upload/edit form for torrent tags.
Now the inputs are dynamically generated by the helper tag_form.
No more modal window in those form, only inputs.
Support of tags in API
New translation string for the link to the modal on torrent view.
More comments in the functions for tags
* Improving how config for tags work. Adding a test on them with understandable messages.
Config for tags have now a Field attribute which is linked to the Torrent model. For example anidbid tag type has now a AnidbID field in config which is the name of the field in torrent model (AnidbID). Every new tag type need to have a field attribute with its counterpart in torrent Model.
Fixing some errors
* Fix compile error + Tests Errors
* Improve performance by caching the list of tags with an index
Adding/removing tags works/tested
New translation strings
TODO: test/fix adding tag on upload/edit
* Mini fix to display video quality
+ tags works/tested on modo edit
* Fix editing tags on modpanel
* Edit tags works
* Add translation string
* Add search backend for tags.
?tags=xxx,eee,ddd
?anidb=21
?vndb=23
?vgmdb=24
?vq=full_hd
* Fix Ajax tag Removal&Add
* Added form for descriptive tags
* Forgot to add the link between database and form for descriptive tags.
* Adding the increase/decrease pantsu for descriptive tags
* Fix #1370
* When you actually forgot to commit files after having forgotten commits
2017-08-22 03:48:10 +02:00
ID uint ` json:"id" `
Name string ` json:"name" `
Status int ` json:"status" `
Hidden bool ` json:"-" `
Hash string ` json:"hash" `
Date string ` json:"date" `
2017-10-11 03:24:09 +02:00
FullDate time . Time ` json:"-" ` //Used to convert the date to full OR short format depending on the situation
Tag Search + Tests + Search slight refactor [DONE] (#1342)
* Tag Search + Tests + Search slight refactor
First commit improving search. Different struct have their own file with their tests. This way of separating struct by files is inspired by the go packages I've seen so far.
Added new behaviour as discussed in #1334
* fix fallback to ES
* Added some comments to explain PG fallback + log err moved
* Refactored search
Nearly fully covered
WhereParams struct has disappeared for Query struct instead
In DB model, we use an interface implementing Query struct methods
* 1rst Refactor of Tags (WTF already?!)
Prepare Tags for the refactored system. Now there will be descriptive tags for a particular release (ecchi, BDSM, ....) and typed tags.
Typed tags are tags relevant to all torrents and can be limited to some input value. For example, video quality is a typed tag limited to some values (hd, full hd, sd, ...). In the same way, anidbid is also a typed tag but doesn't have default values.
Furthermore, the location storage of tags have changed, now accepted descriptive tags are stored in the torrents table in the column "tags" and they are separated by commas.
In the opposite, accepted typed tags can have have their own column in the torrents table. For example, anidbid, vndbid will populate the column DbID when accepted. On the other hand, videoquality will populate the same way as descriptive tags.
This behaviour depends on the callbackOnType function in tag/helpers.go
* fix for modtools :')
* Added anidb, vndb, dlsite & vmdb id fields in torrent model.
Tags don't have an accepted field anymore.
Accepted Tags are in torrent.AcceptedTags and non-accepted ones in torrrent.Tags.
New Helper + New Changelog for translation string.
* New upload/edit form for torrent tags.
Now the inputs are dynamically generated by the helper tag_form.
No more modal window in those form, only inputs.
Support of tags in API
New translation string for the link to the modal on torrent view.
More comments in the functions for tags
* Improving how config for tags work. Adding a test on them with understandable messages.
Config for tags have now a Field attribute which is linked to the Torrent model. For example anidbid tag type has now a AnidbID field in config which is the name of the field in torrent model (AnidbID). Every new tag type need to have a field attribute with its counterpart in torrent Model.
Fixing some errors
* Fix compile error + Tests Errors
* Improve performance by caching the list of tags with an index
Adding/removing tags works/tested
New translation strings
TODO: test/fix adding tag on upload/edit
* Mini fix to display video quality
+ tags works/tested on modo edit
* Fix editing tags on modpanel
* Edit tags works
* Add translation string
* Add search backend for tags.
?tags=xxx,eee,ddd
?anidb=21
?vndb=23
?vgmdb=24
?vq=full_hd
* Fix Ajax tag Removal&Add
* Added form for descriptive tags
* Forgot to add the link between database and form for descriptive tags.
* Adding the increase/decrease pantsu for descriptive tags
* Fix #1370
* When you actually forgot to commit files after having forgotten commits
2017-08-22 03:48:10 +02:00
Filesize int64 ` json:"filesize" `
Description template . HTML ` json:"description" `
Comments [ ] CommentJSON ` json:"comments" `
SubCategory string ` json:"sub_category" `
Category string ` json:"category" `
// Torrent DBID
AnidbID uint ` json:"anidbid" `
VndbID uint ` json:"vndbid" `
VgmdbID uint ` json:"vgmdbid" `
2017-09-17 21:27:20 +02:00
Dlsite string ` json:"dlsite" `
Tag Search + Tests + Search slight refactor [DONE] (#1342)
* Tag Search + Tests + Search slight refactor
First commit improving search. Different struct have their own file with their tests. This way of separating struct by files is inspired by the go packages I've seen so far.
Added new behaviour as discussed in #1334
* fix fallback to ES
* Added some comments to explain PG fallback + log err moved
* Refactored search
Nearly fully covered
WhereParams struct has disappeared for Query struct instead
In DB model, we use an interface implementing Query struct methods
* 1rst Refactor of Tags (WTF already?!)
Prepare Tags for the refactored system. Now there will be descriptive tags for a particular release (ecchi, BDSM, ....) and typed tags.
Typed tags are tags relevant to all torrents and can be limited to some input value. For example, video quality is a typed tag limited to some values (hd, full hd, sd, ...). In the same way, anidbid is also a typed tag but doesn't have default values.
Furthermore, the location storage of tags have changed, now accepted descriptive tags are stored in the torrents table in the column "tags" and they are separated by commas.
In the opposite, accepted typed tags can have have their own column in the torrents table. For example, anidbid, vndbid will populate the column DbID when accepted. On the other hand, videoquality will populate the same way as descriptive tags.
This behaviour depends on the callbackOnType function in tag/helpers.go
* fix for modtools :')
* Added anidb, vndb, dlsite & vmdb id fields in torrent model.
Tags don't have an accepted field anymore.
Accepted Tags are in torrent.AcceptedTags and non-accepted ones in torrrent.Tags.
New Helper + New Changelog for translation string.
* New upload/edit form for torrent tags.
Now the inputs are dynamically generated by the helper tag_form.
No more modal window in those form, only inputs.
Support of tags in API
New translation string for the link to the modal on torrent view.
More comments in the functions for tags
* Improving how config for tags work. Adding a test on them with understandable messages.
Config for tags have now a Field attribute which is linked to the Torrent model. For example anidbid tag type has now a AnidbID field in config which is the name of the field in torrent model (AnidbID). Every new tag type need to have a field attribute with its counterpart in torrent Model.
Fixing some errors
* Fix compile error + Tests Errors
* Improve performance by caching the list of tags with an index
Adding/removing tags works/tested
New translation strings
TODO: test/fix adding tag on upload/edit
* Mini fix to display video quality
+ tags works/tested on modo edit
* Fix editing tags on modpanel
* Edit tags works
* Add translation string
* Add search backend for tags.
?tags=xxx,eee,ddd
?anidb=21
?vndb=23
?vgmdb=24
?vq=full_hd
* Fix Ajax tag Removal&Add
* Added form for descriptive tags
* Forgot to add the link between database and form for descriptive tags.
* Adding the increase/decrease pantsu for descriptive tags
* Fix #1370
* When you actually forgot to commit files after having forgotten commits
2017-08-22 03:48:10 +02:00
VideoQuality string ` json:"videoquality" `
AcceptedTags Tags ` json:"tags" `
2017-10-15 07:31:05 +02:00
UploaderID uint ` json:"uploader_id" `
UploaderName template . HTML ` json:"uploader_name" `
OldUploader template . HTML ` json:"uploader_old" `
WebsiteLink template . URL ` json:"website_link" `
Languages [ ] string ` json:"languages" `
Magnet template . URL ` json:"magnet" `
TorrentLink template . URL ` json:"torrent" `
Seeders uint32 ` json:"seeders" `
Leechers uint32 ` json:"leechers" `
Completed uint32 ` json:"completed" `
LastScrape time . Time ` json:"last_scrape" `
StatsObsolete [ ] bool ` json:"-" ` //First cell determines whether the stats are valid, second determines whether the stats need a refresh regardless of first cell (too old stats?)
FileList [ ] FileJSON ` json:"file_list" `
Tags Tags ` json:"-" ` // not needed in json to reduce db calls
2017-05-10 10:27:17 +02:00
}
2017-05-26 12:12:52 +02:00
// Size : Returns the total size of memory recursively allocated for this struct
2017-05-27 00:45:18 +02:00
// FIXME: Is it deprecated?
2017-05-10 10:47:06 +02:00
func ( t Torrent ) Size ( ) ( s int ) {
2017-05-27 00:45:18 +02:00
s = int ( reflect . TypeOf ( t ) . Size ( ) )
2017-05-10 10:27:17 +02:00
return
2017-05-10 11:08:02 +02:00
}
This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
// TableName : Return the table name of torrents table
2017-05-16 04:53:02 +02:00
func ( t Torrent ) TableName ( ) string {
2017-07-10 14:11:05 +02:00
return config . Get ( ) . Models . TorrentsTableName
2017-05-20 01:10:16 +02:00
}
2017-05-26 12:12:52 +02:00
// Identifier : Return the identifier of a torrent
2017-05-22 00:22:42 +02:00
func ( t * Torrent ) Identifier ( ) string {
2017-07-09 16:21:34 +02:00
return fmt . Sprintf ( "torrent_%d" , t . ID )
2017-05-20 20:53:05 +02:00
}
2017-05-26 12:12:52 +02:00
// IsNormal : Return if a torrent status is normal
This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
func ( t * Torrent ) IsNormal ( ) bool {
2017-05-20 01:10:16 +02:00
return t . Status == TorrentStatusNormal
}
2017-06-16 01:13:09 +02:00
// IsRemake : Return if a torrent status is remake
This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
func ( t * Torrent ) IsRemake ( ) bool {
2017-05-20 01:10:16 +02:00
return t . Status == TorrentStatusRemake
}
2017-05-26 12:12:52 +02:00
// IsTrusted : Return if a torrent status is trusted
This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
func ( t * Torrent ) IsTrusted ( ) bool {
2017-05-20 01:10:16 +02:00
return t . Status == TorrentStatusTrusted
}
2017-05-26 12:12:52 +02:00
// IsAPlus : Return if a torrent status is a+
This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
func ( t * Torrent ) IsAPlus ( ) bool {
2017-05-20 01:10:16 +02:00
return t . Status == TorrentStatusAPlus
2017-05-16 04:53:02 +02:00
}
2017-05-26 12:12:52 +02:00
// IsBlocked : Return if a torrent status is locked
2017-05-25 02:19:05 +02:00
func ( t * Torrent ) IsBlocked ( ) bool {
return t . Status == TorrentStatusBlocked
}
2017-05-26 12:12:52 +02:00
// IsDeleted : Return if a torrent status is deleted
2017-05-25 02:19:05 +02:00
func ( t * Torrent ) IsDeleted ( ) bool {
return t . DeletedAt != nil
}
Tag Search + Tests + Search slight refactor [DONE] (#1342)
* Tag Search + Tests + Search slight refactor
First commit improving search. Different struct have their own file with their tests. This way of separating struct by files is inspired by the go packages I've seen so far.
Added new behaviour as discussed in #1334
* fix fallback to ES
* Added some comments to explain PG fallback + log err moved
* Refactored search
Nearly fully covered
WhereParams struct has disappeared for Query struct instead
In DB model, we use an interface implementing Query struct methods
* 1rst Refactor of Tags (WTF already?!)
Prepare Tags for the refactored system. Now there will be descriptive tags for a particular release (ecchi, BDSM, ....) and typed tags.
Typed tags are tags relevant to all torrents and can be limited to some input value. For example, video quality is a typed tag limited to some values (hd, full hd, sd, ...). In the same way, anidbid is also a typed tag but doesn't have default values.
Furthermore, the location storage of tags have changed, now accepted descriptive tags are stored in the torrents table in the column "tags" and they are separated by commas.
In the opposite, accepted typed tags can have have their own column in the torrents table. For example, anidbid, vndbid will populate the column DbID when accepted. On the other hand, videoquality will populate the same way as descriptive tags.
This behaviour depends on the callbackOnType function in tag/helpers.go
* fix for modtools :')
* Added anidb, vndb, dlsite & vmdb id fields in torrent model.
Tags don't have an accepted field anymore.
Accepted Tags are in torrent.AcceptedTags and non-accepted ones in torrrent.Tags.
New Helper + New Changelog for translation string.
* New upload/edit form for torrent tags.
Now the inputs are dynamically generated by the helper tag_form.
No more modal window in those form, only inputs.
Support of tags in API
New translation string for the link to the modal on torrent view.
More comments in the functions for tags
* Improving how config for tags work. Adding a test on them with understandable messages.
Config for tags have now a Field attribute which is linked to the Torrent model. For example anidbid tag type has now a AnidbID field in config which is the name of the field in torrent model (AnidbID). Every new tag type need to have a field attribute with its counterpart in torrent Model.
Fixing some errors
* Fix compile error + Tests Errors
* Improve performance by caching the list of tags with an index
Adding/removing tags works/tested
New translation strings
TODO: test/fix adding tag on upload/edit
* Mini fix to display video quality
+ tags works/tested on modo edit
* Fix editing tags on modpanel
* Edit tags works
* Add translation string
* Add search backend for tags.
?tags=xxx,eee,ddd
?anidb=21
?vndb=23
?vgmdb=24
?vq=full_hd
* Fix Ajax tag Removal&Add
* Added form for descriptive tags
* Forgot to add the link between database and form for descriptive tags.
* Adding the increase/decrease pantsu for descriptive tags
* Fix #1370
* When you actually forgot to commit files after having forgotten commits
2017-08-22 03:48:10 +02:00
// GetDescriptiveTags : Return the descriptive tags
func ( t * Torrent ) GetDescriptiveTags ( ) string {
return t . AcceptedTags
}
2017-05-27 00:45:18 +02:00
// AddToESIndex : Adds a torrent to Elastic Search
2017-05-26 01:48:14 +02:00
func ( t Torrent ) AddToESIndex ( client * elastic . Client ) error {
ctx := context . Background ( )
2017-05-27 00:45:18 +02:00
torrentJSON := t . ToJSON ( )
2017-05-26 01:48:14 +02:00
_ , err := client . Index ( ) .
2017-07-10 14:11:05 +02:00
Index ( config . Get ( ) . Search . ElasticsearchIndex ) .
Type ( config . Get ( ) . Search . ElasticsearchType ) .
2017-05-27 03:54:41 +02:00
Id ( strconv . FormatUint ( uint64 ( torrentJSON . ID ) , 10 ) ) .
2017-05-27 00:45:18 +02:00
BodyJson ( torrentJSON ) .
2017-05-26 01:48:14 +02:00
Refresh ( "true" ) .
Do ( ctx )
return err
}
2017-05-27 00:45:18 +02:00
// DeleteFromESIndex : Removes a torrent from Elastic Search
2017-07-02 23:53:23 +02:00
func ( t * Torrent ) DeleteFromESIndex ( client * elastic . Client ) error {
2017-05-26 01:48:14 +02:00
ctx := context . Background ( )
_ , err := client . Delete ( ) .
2017-07-10 14:11:05 +02:00
Index ( config . Get ( ) . Search . ElasticsearchIndex ) .
Type ( config . Get ( ) . Search . ElasticsearchType ) .
2017-05-26 01:48:14 +02:00
Id ( strconv . FormatInt ( int64 ( t . ID ) , 10 ) ) .
Do ( ctx )
return err
}
2017-05-27 00:45:18 +02:00
// ParseTrackers : Takes an array of trackers, adds needed trackers and parse it to url string
func ( t * Torrent ) ParseTrackers ( trackers [ ] string ) {
v := url . Values { }
2017-07-10 14:11:05 +02:00
if len ( config . Get ( ) . Torrents . Trackers . NeededTrackers ) > 0 { // if we have some needed trackers configured
2017-05-27 00:45:18 +02:00
if len ( trackers ) == 0 {
2017-07-10 14:11:05 +02:00
trackers = config . Get ( ) . Torrents . Trackers . Default
2017-05-27 00:45:18 +02:00
} else {
2017-07-10 14:11:05 +02:00
for _ , id := range config . Get ( ) . Torrents . Trackers . NeededTrackers {
2017-05-27 00:45:18 +02:00
found := false
for _ , tracker := range trackers {
2017-07-10 14:11:05 +02:00
if tracker == config . Get ( ) . Torrents . Trackers . Default [ id ] {
2017-05-27 00:45:18 +02:00
found = true
break
}
}
if ! found {
2017-07-10 14:11:05 +02:00
trackers = append ( trackers , config . Get ( ) . Torrents . Trackers . Default [ id ] )
2017-05-27 00:45:18 +02:00
}
}
}
}
2017-10-29 04:38:33 +01:00
tempTrackers := [ ] string { }
for _ , line := range trackers {
if ! contains ( tempTrackers , line ) {
tempTrackers = append ( tempTrackers , line )
}
}
trackers = tempTrackers
2017-05-27 00:45:18 +02:00
v [ "tr" ] = trackers
t . Trackers = v . Encode ( )
}
2017-07-02 23:53:23 +02:00
func ( t * Torrent ) ParseLanguages ( ) {
t . Languages = strings . Split ( t . Language , "," )
}
func ( t * Torrent ) EncodeLanguages ( ) {
t . Language = strings . Join ( t . Languages , "," )
}
2017-05-27 00:45:18 +02:00
// GetTrackersArray : Convert trackers string to Array
func ( t * Torrent ) GetTrackersArray ( ) ( trackers [ ] string ) {
v , _ := url . ParseQuery ( t . Trackers )
trackers = v [ "tr" ]
return
}
This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
// ToTorrent :
2017-06-05 03:33:02 +02:00
// TODO: Need to get rid of TorrentJSON altogether and have only one true Torrent
// model
func ( t * TorrentJSON ) ToTorrent ( ) Torrent {
category , err := strconv . ParseInt ( t . Category , 10 , 64 )
if err != nil {
category = 0
}
subCategory , err := strconv . ParseInt ( t . SubCategory , 10 , 64 )
if err != nil {
subCategory = 0
}
// Need to add +00:00 at the end because ES doesn't store it by default
2017-06-26 17:10:53 +02:00
dateFixed := t . Date
2017-07-04 02:11:38 +02:00
if len ( dateFixed ) > 6 && dateFixed [ len ( dateFixed ) - 6 ] != '+' {
2017-06-26 17:10:53 +02:00
dateFixed += "Z"
}
date , err := time . Parse ( time . RFC3339 , dateFixed )
2017-06-05 03:33:02 +02:00
if err != nil {
2017-06-26 17:10:53 +02:00
log . Errorf ( "Problem parsing date '%s' from ES: %s" , dateFixed , err )
2017-06-05 03:33:02 +02:00
}
torrent := Torrent {
2017-06-06 00:06:04 +02:00
ID : t . ID ,
Name : t . Name ,
Hash : t . Hash ,
Category : int ( category ) ,
2017-06-05 03:33:02 +02:00
SubCategory : int ( subCategory ) ,
2017-06-06 00:06:04 +02:00
Status : t . Status ,
Date : date ,
UploaderID : t . UploaderID ,
Tag Search + Tests + Search slight refactor [DONE] (#1342)
* Tag Search + Tests + Search slight refactor
First commit improving search. Different struct have their own file with their tests. This way of separating struct by files is inspired by the go packages I've seen so far.
Added new behaviour as discussed in #1334
* fix fallback to ES
* Added some comments to explain PG fallback + log err moved
* Refactored search
Nearly fully covered
WhereParams struct has disappeared for Query struct instead
In DB model, we use an interface implementing Query struct methods
* 1rst Refactor of Tags (WTF already?!)
Prepare Tags for the refactored system. Now there will be descriptive tags for a particular release (ecchi, BDSM, ....) and typed tags.
Typed tags are tags relevant to all torrents and can be limited to some input value. For example, video quality is a typed tag limited to some values (hd, full hd, sd, ...). In the same way, anidbid is also a typed tag but doesn't have default values.
Furthermore, the location storage of tags have changed, now accepted descriptive tags are stored in the torrents table in the column "tags" and they are separated by commas.
In the opposite, accepted typed tags can have have their own column in the torrents table. For example, anidbid, vndbid will populate the column DbID when accepted. On the other hand, videoquality will populate the same way as descriptive tags.
This behaviour depends on the callbackOnType function in tag/helpers.go
* fix for modtools :')
* Added anidb, vndb, dlsite & vmdb id fields in torrent model.
Tags don't have an accepted field anymore.
Accepted Tags are in torrent.AcceptedTags and non-accepted ones in torrrent.Tags.
New Helper + New Changelog for translation string.
* New upload/edit form for torrent tags.
Now the inputs are dynamically generated by the helper tag_form.
No more modal window in those form, only inputs.
Support of tags in API
New translation string for the link to the modal on torrent view.
More comments in the functions for tags
* Improving how config for tags work. Adding a test on them with understandable messages.
Config for tags have now a Field attribute which is linked to the Torrent model. For example anidbid tag type has now a AnidbID field in config which is the name of the field in torrent model (AnidbID). Every new tag type need to have a field attribute with its counterpart in torrent Model.
Fixing some errors
* Fix compile error + Tests Errors
* Improve performance by caching the list of tags with an index
Adding/removing tags works/tested
New translation strings
TODO: test/fix adding tag on upload/edit
* Mini fix to display video quality
+ tags works/tested on modo edit
* Fix editing tags on modpanel
* Edit tags works
* Add translation string
* Add search backend for tags.
?tags=xxx,eee,ddd
?anidb=21
?vndb=23
?vgmdb=24
?vq=full_hd
* Fix Ajax tag Removal&Add
* Added form for descriptive tags
* Forgot to add the link between database and form for descriptive tags.
* Adding the increase/decrease pantsu for descriptive tags
* Fix #1370
* When you actually forgot to commit files after having forgotten commits
2017-08-22 03:48:10 +02:00
AnidbID : t . AnidbID ,
VndbID : t . VndbID ,
VgmdbID : t . VgmdbID ,
Dlsite : t . Dlsite ,
2017-06-05 03:33:02 +02:00
//Stardom: t.Stardom,
2017-06-23 00:54:31 +02:00
Filesize : t . Filesize ,
Description : string ( t . Description ) ,
Hidden : t . Hidden ,
2017-06-05 03:33:02 +02:00
//WebsiteLink: t.WebsiteLink,
//Trackers: t.Trackers,
//DeletedAt: t.DeletedAt,
// Uploader: TODO
//OldUploader: t.OldUploader,
//OldComments: TODO
// Comments: TODO
2017-06-16 01:13:09 +02:00
// LastScrape not stored in ES, counts won't show without a value however
2017-07-02 23:53:23 +02:00
Scrape : & Scrape { Seeders : t . Seeders , Leechers : t . Leechers , Completed : t . Completed , LastScrape : time . Now ( ) } ,
Languages : t . Languages ,
2017-06-05 03:33:02 +02:00
//FileList: TODO
}
2017-07-02 23:53:23 +02:00
torrent . EncodeLanguages ( )
2017-06-05 03:33:02 +02:00
return torrent
}
2017-07-01 23:09:35 +02:00
// ToJSON converts a models.Torrent to its equivalent JSON structure
Consistency, formatting, error checking, cleanup, and a couple bug fixes (#245)
* Checkpoint: it builds
The config, db, model, network, os, and public packages have had some
fixes to glaringly obvious flaws, dead code removed, and stylistic
changes.
* Style changes and old code removal in router
Router needs a lot of work done to its (lack of) error handling.
* Dead code removal and style changes
Now up to util/email/email.go. After I'm finished with the initial sweep
I'll go back and fix error handling and security issues. Then I'll fix
the broken API. Then I'll go through to add documentation and fix code
visibility.
* Finish dead code removal and style changes
Vendored libraries not touched. Everything still needs security fixes
and documentation. There's also one case of broken functionality.
* Fix accidental find-and-replace
* Style, error checking, saftey, bug fix changes
* Redo error checking erased during merge
* Re-add merge-erased fix. Make Safe safe.
2017-05-10 04:34:40 +02:00
func ( t * Torrent ) ToJSON ( ) TorrentJSON {
2017-05-27 00:45:18 +02:00
var trackers [ ] string
if t . Trackers == "" {
2017-07-10 14:11:05 +02:00
trackers = config . Get ( ) . Torrents . Trackers . Default
2017-05-27 00:45:18 +02:00
} else {
trackers = t . GetTrackersArray ( )
}
2017-07-02 16:54:55 +02:00
magnet := format . InfoHashToMagnet ( strings . TrimSpace ( t . Hash ) , t . Name , trackers ... )
Consistency, formatting, error checking, cleanup, and a couple bug fixes (#245)
* Checkpoint: it builds
The config, db, model, network, os, and public packages have had some
fixes to glaringly obvious flaws, dead code removed, and stylistic
changes.
* Style changes and old code removal in router
Router needs a lot of work done to its (lack of) error handling.
* Dead code removal and style changes
Now up to util/email/email.go. After I'm finished with the initial sweep
I'll go back and fix error handling and security issues. Then I'll fix
the broken API. Then I'll go through to add documentation and fix code
visibility.
* Finish dead code removal and style changes
Vendored libraries not touched. Everything still needs security fixes
and documentation. There's also one case of broken functionality.
* Fix accidental find-and-replace
* Style, error checking, saftey, bug fix changes
* Redo error checking erased during merge
* Re-add merge-erased fix. Make Safe safe.
2017-05-10 04:34:40 +02:00
commentsJSON := make ( [ ] CommentJSON , 0 , len ( t . OldComments ) + len ( t . Comments ) )
2017-05-08 20:07:25 +02:00
for _ , c := range t . OldComments {
2017-10-28 18:28:59 +02:00
commentsJSON = append ( commentsJSON , CommentJSON { Username : c . Username , UserID : - 1 , UserStatus : "" , Content : template . HTML ( c . Content ) , Date : c . Date . UTC ( ) } )
2017-05-08 20:07:25 +02:00
}
for _ , c := range t . Comments {
2017-05-17 07:58:40 +02:00
if c . User != nil {
2017-10-28 18:28:59 +02:00
userStatus := ""
if c . User . IsBanned ( ) {
userStatus = "userstatus_banned"
}
if c . User . HasAdmin ( ) {
userStatus = "userstatus_moderator"
}
if c . User . ID == t . ID {
userStatus = "userstatus_uploader"
}
commentsJSON = append ( commentsJSON , CommentJSON { Username : c . User . Username , UserID : int ( c . User . ID ) , UserStatus : userStatus , Content : sanitize . MarkdownToHTML ( c . Content ) , Date : c . CreatedAt . UTC ( ) , UserAvatar : c . User . MD5 } )
2017-05-17 00:05:16 +02:00
} else {
2017-05-17 07:58:40 +02:00
commentsJSON = append ( commentsJSON , CommentJSON { } )
2017-05-17 00:05:16 +02:00
}
2017-05-08 04:06:11 +02:00
}
2017-05-17 13:16:40 +02:00
// Sort comments by date
slice . Sort ( commentsJSON , func ( i , j int ) bool {
return commentsJSON [ i ] . Date . Before ( commentsJSON [ j ] . Date )
} )
2017-05-14 15:19:19 +02:00
fileListJSON := make ( [ ] FileJSON , 0 , len ( t . FileList ) )
for _ , f := range t . FileList {
2017-05-15 23:45:47 +02:00
fileListJSON = append ( fileListJSON , FileJSON {
2017-05-16 04:53:02 +02:00
Path : filepath . Join ( f . Path ( ) ... ) ,
2017-05-21 14:34:32 +02:00
Filesize : f . Filesize ,
2017-05-15 23:45:47 +02:00
} )
2017-05-14 15:19:19 +02:00
}
2017-05-17 13:16:40 +02:00
// Sort file list by lowercase filename
2017-05-20 01:10:16 +02:00
slice . Sort ( fileListJSON , func ( i , j int ) bool {
2017-05-17 13:16:40 +02:00
return strings . ToLower ( fileListJSON [ i ] . Path ) < strings . ToLower ( fileListJSON [ j ] . Path )
} )
2017-06-06 00:06:04 +02:00
uploader := "れんちょん" // by default
2017-05-27 20:33:40 +02:00
var uploaderID uint
2017-06-17 03:46:49 +02:00
if t . UploaderID > 0 && t . Uploader != nil {
2017-05-09 12:35:46 +02:00
uploader = t . Uploader . Username
2017-05-27 20:33:40 +02:00
uploaderID = t . UploaderID
2017-06-16 00:57:52 +02:00
} else if t . OldUploader != "" {
uploader = t . OldUploader
2017-05-09 12:35:46 +02:00
}
2017-05-09 16:42:12 +02:00
torrentlink := ""
Torrent Generation on not found error (#1600)
* [WIP] Torrent Generation on not found error
As asked in #1517, it allows on-the-fly torrent generation. Since it uses magnet links, it needs some time to connect to peers. So it can't be instant generation, we need the user to wait and try after a minute at least.
* Replace Fatal by simple error
* attempt at fixing travis
* del
* Add Anacrolyx dependency
* Add back difflib
* Remove .torrent suffix in the url example
* Add some explanations when file missing page shown
* Ignore downloads directory
* Either use cache (third-party site) or own download directory
* Wrong import
* If there is an error then it means we aren't generating a torrent file
May it be "torrent not found" or "We do not store torrent files" which are the two only existing errors for this page
* hash is never empty
* TorrentLink may be empty at times
So we add a /download/:hash link if it is
* Update README.md
* Made a mistake here, need to check if false
* Update en-us.all.json
* Update CHANGELOG.md
* Torrent file generation can be triggered by click on button if JS enabled
* Update download.go
* Update download.go
* Use c.JSON instead of text/template
* Return to default behavior if we don't generate the file
* Don't do the query if returned to default behavior
* Add "Could not generate torrent file" error
* Fix JS condition & lower delay until button updates
* Start download automatically once torrent file is generated
* Fix torrentFileExists() constantly returning false if external torrent download URL
* torrent-view-data is two tables instead of one
This allows the removal of useless things without any problem (e.g Website link), but also a better responsibe design since the previous one separated stats after a certain res looking very wonky
* CSS changes to go along
* Remove useless <b></b>
* Update main.css
* In torrentFileExists, check if filestorage path exists instead of looking at the domain in torrent link
When checking if the file is stored on another server i used to simply check if the domain name was inside the torrent link, but we can straight up check for filestorage length
* Fix JS of on-demand stat fetching
* ScrapeAge variable accessible through view.jet.html
Contains last scraped time in hours, is at -1 is torrent has never been scraped
Stats will get updated if it's either at -1 or above 1460 (2 months old)
* Refresh stats if older than two months OR unknown and older than 24h
Show last scraped date even if stats are unknown
* Add StatsObsolete variable to torrent
Indicating if:
- They can be shown
- They need to be updated
* Update scraped data even if Unknown, prevent users from trying to fetch stats every seconds
* Torrent file stored locally by default
* no need to do all of that if no filestorage
* fix filestorage path
* Fix torrent download button stuck on "Generating torrent file" at rare times
* fix some css rules that didn't work on IE
* Fix panic error
Seems like this error is a known bug from anacrolyx torrent https://github.com/anacrolix/torrent/issues/83
To prevent it, I'm creating a single client and modifying the socket.go to make it not raise a panic but a simple error log.
2017-10-21 09:40:43 +02:00
if len ( config . Get ( ) . Torrents . CacheLink ) > 0 { // Only use torrent cache if set, don't check id since better to have all .torrent
2017-05-20 12:45:27 +02:00
if config . IsSukebei ( ) {
torrentlink = "" // torrent cache doesn't have sukebei torrents
} else {
2017-07-10 14:11:05 +02:00
torrentlink = fmt . Sprintf ( config . Get ( ) . Torrents . CacheLink , t . Hash )
2017-05-20 12:45:27 +02:00
}
Torrent Generation on not found error (#1600)
* [WIP] Torrent Generation on not found error
As asked in #1517, it allows on-the-fly torrent generation. Since it uses magnet links, it needs some time to connect to peers. So it can't be instant generation, we need the user to wait and try after a minute at least.
* Replace Fatal by simple error
* attempt at fixing travis
* del
* Add Anacrolyx dependency
* Add back difflib
* Remove .torrent suffix in the url example
* Add some explanations when file missing page shown
* Ignore downloads directory
* Either use cache (third-party site) or own download directory
* Wrong import
* If there is an error then it means we aren't generating a torrent file
May it be "torrent not found" or "We do not store torrent files" which are the two only existing errors for this page
* hash is never empty
* TorrentLink may be empty at times
So we add a /download/:hash link if it is
* Update README.md
* Made a mistake here, need to check if false
* Update en-us.all.json
* Update CHANGELOG.md
* Torrent file generation can be triggered by click on button if JS enabled
* Update download.go
* Update download.go
* Use c.JSON instead of text/template
* Return to default behavior if we don't generate the file
* Don't do the query if returned to default behavior
* Add "Could not generate torrent file" error
* Fix JS condition & lower delay until button updates
* Start download automatically once torrent file is generated
* Fix torrentFileExists() constantly returning false if external torrent download URL
* torrent-view-data is two tables instead of one
This allows the removal of useless things without any problem (e.g Website link), but also a better responsibe design since the previous one separated stats after a certain res looking very wonky
* CSS changes to go along
* Remove useless <b></b>
* Update main.css
* In torrentFileExists, check if filestorage path exists instead of looking at the domain in torrent link
When checking if the file is stored on another server i used to simply check if the domain name was inside the torrent link, but we can straight up check for filestorage length
* Fix JS of on-demand stat fetching
* ScrapeAge variable accessible through view.jet.html
Contains last scraped time in hours, is at -1 is torrent has never been scraped
Stats will get updated if it's either at -1 or above 1460 (2 months old)
* Refresh stats if older than two months OR unknown and older than 24h
Show last scraped date even if stats are unknown
* Add StatsObsolete variable to torrent
Indicating if:
- They can be shown
- They need to be updated
* Update scraped data even if Unknown, prevent users from trying to fetch stats every seconds
* Torrent file stored locally by default
* no need to do all of that if no filestorage
* fix filestorage path
* Fix torrent download button stuck on "Generating torrent file" at rare times
* fix some css rules that didn't work on IE
* Fix panic error
Seems like this error is a known bug from anacrolyx torrent https://github.com/anacrolix/torrent/issues/83
To prevent it, I'm creating a single client and modifying the socket.go to make it not raise a panic but a simple error log.
2017-10-21 09:40:43 +02:00
} else if len ( config . Get ( ) . Torrents . StorageLink ) > 0 { // Only use own .torrent if storage set
2017-07-10 14:11:05 +02:00
torrentlink = fmt . Sprintf ( config . Get ( ) . Torrents . StorageLink , t . Hash )
2017-05-08 04:06:11 +02:00
}
2017-06-16 01:13:09 +02:00
scrape := Scrape { }
if t . Scrape != nil {
scrape = * t . Scrape
}
2017-10-15 07:31:05 +02:00
statsObsolete := [ ] bool { false , false }
if scrape . LastScrape . IsZero ( ) || ( scrape . Seeders == 0 && scrape . Leechers == 0 && scrape . Completed == 0 ) {
statsObsolete [ 0 ] = true
//The displayed stats are obsolete, S/D/L will show "Unknown"
}
2017-10-28 18:28:59 +02:00
if time . Since ( scrape . LastScrape ) . Hours ( ) > config . Get ( ) . Scrape . StatScrapingFrequency || ( scrape . Seeders == 0 && scrape . Leechers == 0 && scrape . Completed == 0 && time . Since ( scrape . LastScrape ) . Hours ( ) >= config . Get ( ) . Scrape . StatScrapingFrequencyUnknown ) {
2017-10-15 07:31:05 +02:00
statsObsolete [ 1 ] = true
2017-10-18 05:46:58 +02:00
//The stats need to be refreshed, either because they are valid and older than one month (not that reliable) OR if they are unknown but have been scraped 1h (or more) ago
2017-10-15 07:31:05 +02:00
}
2017-07-02 23:53:23 +02:00
t . ParseLanguages ( )
Consistency, formatting, error checking, cleanup, and a couple bug fixes (#245)
* Checkpoint: it builds
The config, db, model, network, os, and public packages have had some
fixes to glaringly obvious flaws, dead code removed, and stylistic
changes.
* Style changes and old code removal in router
Router needs a lot of work done to its (lack of) error handling.
* Dead code removal and style changes
Now up to util/email/email.go. After I'm finished with the initial sweep
I'll go back and fix error handling and security issues. Then I'll fix
the broken API. Then I'll go through to add documentation and fix code
visibility.
* Finish dead code removal and style changes
Vendored libraries not touched. Everything still needs security fixes
and documentation. There's also one case of broken functionality.
* Fix accidental find-and-replace
* Style, error checking, saftey, bug fix changes
* Redo error checking erased during merge
* Re-add merge-erased fix. Make Safe safe.
2017-05-10 04:34:40 +02:00
res := TorrentJSON {
2017-10-15 07:31:05 +02:00
ID : t . ID ,
Name : t . Name ,
Status : t . Status ,
Hidden : t . Hidden ,
Hash : t . Hash ,
2017-10-28 18:28:59 +02:00
Date : t . Date . UTC ( ) . Format ( time . RFC3339 ) ,
FullDate : t . Date . UTC ( ) ,
2017-10-15 07:31:05 +02:00
Filesize : t . Filesize ,
Description : sanitize . MarkdownToHTML ( t . Description ) ,
Comments : commentsJSON ,
SubCategory : strconv . Itoa ( t . SubCategory ) ,
Category : strconv . Itoa ( t . Category ) ,
UploaderID : uploaderID ,
UploaderName : sanitize . SafeText ( uploader ) ,
WebsiteLink : sanitize . Safe ( t . WebsiteLink ) ,
Languages : t . Languages ,
Magnet : template . URL ( magnet ) ,
TorrentLink : sanitize . Safe ( torrentlink ) ,
Leechers : scrape . Leechers ,
Seeders : scrape . Seeders ,
Completed : scrape . Completed ,
LastScrape : scrape . LastScrape ,
StatsObsolete : statsObsolete ,
FileList : fileListJSON ,
Tags : t . Tags ,
AnidbID : t . AnidbID ,
VndbID : t . VndbID ,
VgmdbID : t . VgmdbID ,
Dlsite : t . Dlsite ,
VideoQuality : t . VideoQuality ,
Tag Search + Tests + Search slight refactor [DONE] (#1342)
* Tag Search + Tests + Search slight refactor
First commit improving search. Different struct have their own file with their tests. This way of separating struct by files is inspired by the go packages I've seen so far.
Added new behaviour as discussed in #1334
* fix fallback to ES
* Added some comments to explain PG fallback + log err moved
* Refactored search
Nearly fully covered
WhereParams struct has disappeared for Query struct instead
In DB model, we use an interface implementing Query struct methods
* 1rst Refactor of Tags (WTF already?!)
Prepare Tags for the refactored system. Now there will be descriptive tags for a particular release (ecchi, BDSM, ....) and typed tags.
Typed tags are tags relevant to all torrents and can be limited to some input value. For example, video quality is a typed tag limited to some values (hd, full hd, sd, ...). In the same way, anidbid is also a typed tag but doesn't have default values.
Furthermore, the location storage of tags have changed, now accepted descriptive tags are stored in the torrents table in the column "tags" and they are separated by commas.
In the opposite, accepted typed tags can have have their own column in the torrents table. For example, anidbid, vndbid will populate the column DbID when accepted. On the other hand, videoquality will populate the same way as descriptive tags.
This behaviour depends on the callbackOnType function in tag/helpers.go
* fix for modtools :')
* Added anidb, vndb, dlsite & vmdb id fields in torrent model.
Tags don't have an accepted field anymore.
Accepted Tags are in torrent.AcceptedTags and non-accepted ones in torrrent.Tags.
New Helper + New Changelog for translation string.
* New upload/edit form for torrent tags.
Now the inputs are dynamically generated by the helper tag_form.
No more modal window in those form, only inputs.
Support of tags in API
New translation string for the link to the modal on torrent view.
More comments in the functions for tags
* Improving how config for tags work. Adding a test on them with understandable messages.
Config for tags have now a Field attribute which is linked to the Torrent model. For example anidbid tag type has now a AnidbID field in config which is the name of the field in torrent model (AnidbID). Every new tag type need to have a field attribute with its counterpart in torrent Model.
Fixing some errors
* Fix compile error + Tests Errors
* Improve performance by caching the list of tags with an index
Adding/removing tags works/tested
New translation strings
TODO: test/fix adding tag on upload/edit
* Mini fix to display video quality
+ tags works/tested on modo edit
* Fix editing tags on modpanel
* Edit tags works
* Add translation string
* Add search backend for tags.
?tags=xxx,eee,ddd
?anidb=21
?vndb=23
?vgmdb=24
?vq=full_hd
* Fix Ajax tag Removal&Add
* Added form for descriptive tags
* Forgot to add the link between database and form for descriptive tags.
* Adding the increase/decrease pantsu for descriptive tags
* Fix #1370
* When you actually forgot to commit files after having forgotten commits
2017-08-22 03:48:10 +02:00
}
// Split accepted tags
tags := strings . Split ( t . AcceptedTags , "," )
for _ , tag := range tags {
if tag != "" {
res . AcceptedTags = append ( res . AcceptedTags , Tag { Tag : tag , Type : config . Get ( ) . Torrents . Tags . Default , Total : config . Get ( ) . Torrents . Tags . MaxWeight , Accepted : true } )
}
2017-05-10 19:29:35 +02:00
}
2017-05-05 14:20:51 +02:00
return res
}
/* Complete the functions when necessary... */
2017-05-09 01:56:57 +02:00
2017-05-26 12:12:52 +02:00
// TorrentsToJSON : Map Torrents to TorrentsToJSON without reallocations
2017-05-20 12:45:27 +02:00
func TorrentsToJSON ( t [ ] Torrent ) [ ] TorrentJSON {
Consistency, formatting, error checking, cleanup, and a couple bug fixes (#245)
* Checkpoint: it builds
The config, db, model, network, os, and public packages have had some
fixes to glaringly obvious flaws, dead code removed, and stylistic
changes.
* Style changes and old code removal in router
Router needs a lot of work done to its (lack of) error handling.
* Dead code removal and style changes
Now up to util/email/email.go. After I'm finished with the initial sweep
I'll go back and fix error handling and security issues. Then I'll fix
the broken API. Then I'll go through to add documentation and fix code
visibility.
* Finish dead code removal and style changes
Vendored libraries not touched. Everything still needs security fixes
and documentation. There's also one case of broken functionality.
* Fix accidental find-and-replace
* Style, error checking, saftey, bug fix changes
* Redo error checking erased during merge
* Re-add merge-erased fix. Make Safe safe.
2017-05-10 04:34:40 +02:00
json := make ( [ ] TorrentJSON , len ( t ) )
2017-05-09 01:56:57 +02:00
for i := range t {
Consistency, formatting, error checking, cleanup, and a couple bug fixes (#245)
* Checkpoint: it builds
The config, db, model, network, os, and public packages have had some
fixes to glaringly obvious flaws, dead code removed, and stylistic
changes.
* Style changes and old code removal in router
Router needs a lot of work done to its (lack of) error handling.
* Dead code removal and style changes
Now up to util/email/email.go. After I'm finished with the initial sweep
I'll go back and fix error handling and security issues. Then I'll fix
the broken API. Then I'll go through to add documentation and fix code
visibility.
* Finish dead code removal and style changes
Vendored libraries not touched. Everything still needs security fixes
and documentation. There's also one case of broken functionality.
* Fix accidental find-and-replace
* Style, error checking, saftey, bug fix changes
* Redo error checking erased during merge
* Re-add merge-erased fix. Make Safe safe.
2017-05-10 04:34:40 +02:00
json [ i ] = t [ i ] . ToJSON ( )
2017-05-09 01:56:57 +02:00
}
return json
}
2017-06-24 10:53:22 +02:00
This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
// Update : Update a torrent based on model
func ( t * Torrent ) Update ( unscope bool ) ( int , error ) {
db := ORM
if unscope {
db = ORM . Unscoped ( )
}
2017-07-04 02:07:25 +02:00
t . EncodeLanguages ( ) // Need to transform array into single string
2017-07-17 21:32:55 +02:00
if db . Model ( t ) . UpdateColumn ( t . toMap ( ) ) . Error != nil {
This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
return http . StatusInternalServerError , errors . New ( "Torrent was not updated" )
}
// TODO Don't create a new client for each request
if ElasticSearchClient != nil {
err := t . AddToESIndex ( ElasticSearchClient )
if err == nil {
log . Infof ( "Successfully updated torrent to ES index." )
} else {
log . Errorf ( "Unable to update torrent to ES index: %s" , err )
}
}
2017-07-28 20:43:22 +02:00
// We only flush cache after update
cache . C . Delete ( t . Identifier ( ) )
This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
return http . StatusOK , nil
}
// UpdateUnscope : Update a torrent based on model
func ( t * Torrent ) UpdateUnscope ( ) ( int , error ) {
return t . Update ( true )
}
2017-07-28 20:43:22 +02:00
// Delete : delete a torrent based on id
2017-07-02 23:53:23 +02:00
func ( t * Torrent ) Delete ( definitely bool ) ( * Torrent , int , error ) {
2017-07-30 03:29:09 +02:00
if t . ID == 0 {
err := errors . New ( "ERROR: Tried to delete a torrent with ID 0" )
log . CheckErrorWithMessage ( err , "ERROR_IMPORTANT: " )
return t , http . StatusBadRequest , err
}
2017-07-02 23:53:23 +02:00
db := ORM
if definitely {
db = ORM . Unscoped ( )
This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
}
2017-07-02 23:53:23 +02:00
if db . Delete ( t ) . Error != nil {
return t , http . StatusInternalServerError , errors . New ( "torrent_not_deleted" )
}
if ElasticSearchClient != nil {
err := t . DeleteFromESIndex ( ElasticSearchClient )
if err == nil {
log . Infof ( "Successfully deleted torrent to ES index." )
} else {
log . Errorf ( "Unable to delete torrent to ES index: %s" , err )
}
This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
}
2017-07-28 20:43:22 +02:00
// We flush cache only after delete
cache . C . Flush ( )
2017-07-02 23:53:23 +02:00
return t , http . StatusOK , nil
}
// DefinitelyDelete : deletes definitely a torrent based on id
func ( t * Torrent ) DefinitelyDelete ( ) ( * Torrent , int , error ) {
return t . Delete ( true )
This is a prelimenary work
Showing how we can remove services, preventing cyclic imports and lessing the number of imports.
Now db is in models. Db and models are highly tightened, according to go standards, you should put them in the same package.
In models, there are folders separating the different methods used to modify the models. For example, if you want to create a user, you have to use /models (for the user struct) and /models/user (for creating a user.
However, if you want to delete a torrent, you just have to import /models and do torrent.Delete(definitely bool).
By the way packages in models are the plural name of a model. For example, you have torrent.go for a torrent model and its package torrents for db stuff related functions (Find, Create, Some helpers)
2017-06-29 00:44:07 +02:00
}
2017-07-17 21:32:55 +02:00
// toMap : convert the model to a map of interface
func ( t * Torrent ) toMap ( ) map [ string ] interface { } {
return structs . Map ( t )
}
2017-07-28 20:43:22 +02:00
// LoadTags : load all the unique tags with summed up weight from the database in torrent
func ( t * Torrent ) LoadTags ( ) {
// Only load if necessary
if len ( t . Tags ) == 0 {
// Should output a query like this: SELECT tag, type, accepted, SUM(weight) as total FROM tags WHERE torrent_id=923000 GROUP BY type, tag ORDER BY type, total DESC
Tag Search + Tests + Search slight refactor [DONE] (#1342)
* Tag Search + Tests + Search slight refactor
First commit improving search. Different struct have their own file with their tests. This way of separating struct by files is inspired by the go packages I've seen so far.
Added new behaviour as discussed in #1334
* fix fallback to ES
* Added some comments to explain PG fallback + log err moved
* Refactored search
Nearly fully covered
WhereParams struct has disappeared for Query struct instead
In DB model, we use an interface implementing Query struct methods
* 1rst Refactor of Tags (WTF already?!)
Prepare Tags for the refactored system. Now there will be descriptive tags for a particular release (ecchi, BDSM, ....) and typed tags.
Typed tags are tags relevant to all torrents and can be limited to some input value. For example, video quality is a typed tag limited to some values (hd, full hd, sd, ...). In the same way, anidbid is also a typed tag but doesn't have default values.
Furthermore, the location storage of tags have changed, now accepted descriptive tags are stored in the torrents table in the column "tags" and they are separated by commas.
In the opposite, accepted typed tags can have have their own column in the torrents table. For example, anidbid, vndbid will populate the column DbID when accepted. On the other hand, videoquality will populate the same way as descriptive tags.
This behaviour depends on the callbackOnType function in tag/helpers.go
* fix for modtools :')
* Added anidb, vndb, dlsite & vmdb id fields in torrent model.
Tags don't have an accepted field anymore.
Accepted Tags are in torrent.AcceptedTags and non-accepted ones in torrrent.Tags.
New Helper + New Changelog for translation string.
* New upload/edit form for torrent tags.
Now the inputs are dynamically generated by the helper tag_form.
No more modal window in those form, only inputs.
Support of tags in API
New translation string for the link to the modal on torrent view.
More comments in the functions for tags
* Improving how config for tags work. Adding a test on them with understandable messages.
Config for tags have now a Field attribute which is linked to the Torrent model. For example anidbid tag type has now a AnidbID field in config which is the name of the field in torrent model (AnidbID). Every new tag type need to have a field attribute with its counterpart in torrent Model.
Fixing some errors
* Fix compile error + Tests Errors
* Improve performance by caching the list of tags with an index
Adding/removing tags works/tested
New translation strings
TODO: test/fix adding tag on upload/edit
* Mini fix to display video quality
+ tags works/tested on modo edit
* Fix editing tags on modpanel
* Edit tags works
* Add translation string
* Add search backend for tags.
?tags=xxx,eee,ddd
?anidb=21
?vndb=23
?vgmdb=24
?vq=full_hd
* Fix Ajax tag Removal&Add
* Added form for descriptive tags
* Forgot to add the link between database and form for descriptive tags.
* Adding the increase/decrease pantsu for descriptive tags
* Fix #1370
* When you actually forgot to commit files after having forgotten commits
2017-08-22 03:48:10 +02:00
err := ORM . Select ( "tag, type, SUM(weight) as total" ) . Where ( "torrent_id = ?" , t . ID ) . Group ( "type, tag" ) . Order ( "type ASC, total DESC" ) . Find ( & t . Tags ) . Error
log . CheckErrorWithMessage ( err , "LOAD_TAGS_ERROR: Couldn't load tags from DB!" )
2017-07-28 20:43:22 +02:00
}
}
2017-08-02 15:06:34 +02:00
// DeleteTags cipes out all the tags from the torrent. Doesn't decrease pantsu on users!
func ( t * Torrent ) DeleteTags ( ) {
if t . ID > 0 {
// Should output a query like this: DELETE FROM tags WHERE torrent_id=923000
err := ORM . Where ( "torrent_id = ?" , t . ID ) . Delete ( & t . Tags ) . Error
log . CheckErrorWithMessage ( err , "LOAD_TAGS_ERROR: Couldn't delete tags!" )
}
}
2017-10-29 04:38:33 +01:00
func contains ( s [ ] string , e string ) bool {
for _ , a := range s {
if a == e {
return true
}
}
return false
}