* Implemented LFS client.
* Implemented scanning for pointer files.
* Implemented downloading of lfs files.
* Moved model-dependent code into services.
* Removed models dependency. Added TryReadPointerFromBuffer.
* Migrated code from service to module.
* Centralised storage creation.
* Removed dependency from models.
* Moved ContentStore into modules.
* Share structs between server and client.
* Moved method to services.
* Implemented lfs download on clone.
* Implemented LFS sync on clone and mirror update.
* Added form fields.
* Updated templates.
* Fixed condition.
* Use alternate endpoint.
* Added missing methods.
* Fixed typo and make linter happy.
* Detached pointer parser from gogit dependency.
* Fixed TestGetLFSRange test.
* Added context to support cancellation.
* Use ReadFull to probably read more data.
* Removed duplicated code from models.
* Moved scan implementation into pointer_scanner_nogogit.
* Changed method name.
* Added comments.
* Added more/specific log/error messages.
* Embedded lfs.Pointer into models.LFSMetaObject.
* Moved code from models to module.
* Moved code from models to module.
* Moved code from models to module.
* Reduced pointer usage.
* Embedded type.
* Use promoted fields.
* Fixed unexpected eof.
* Added unit tests.
* Implemented migration of local file paths.
* Show an error on invalid LFS endpoints.
* Hide settings if not used.
* Added LFS info to mirror struct.
* Fixed comment.
* Check LFS endpoint.
* Manage LFS settings from mirror page.
* Fixed selector.
* Adjusted selector.
* Added more tests.
* Added local filesystem migration test.
* Fixed typo.
* Reset settings.
* Added special windows path handling.
* Added unit test for HTTPClient.
* Added unit test for BasicTransferAdapter.
* Moved into util package.
* Test if LFS endpoint is allowed.
* Added support for git://
* Just use a static placeholder as the displayed url may be invalid.
* Reverted to original code.
* Added "Advanced Settings".
* Updated wording.
* Added discovery info link.
* Implemented suggestion.
* Fixed missing format parameter.
* Added Pointer.IsValid().
* Always remove model on error.
* Added suggestions.
* Use channel instead of array.
* Update routers/repo/migrate.go
* fmt
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: zeripath <art27@cantab.net>
* Close file on invalid range.
* Close on seek error
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Moved 'Seek' into server.
* io.ReadSeekCloser is only available in Go 1.16
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Continuing on from #14888
The previous implementation has race whereby an incomplete upload or
hash mismatch upload can end up in the ContentStore. This PR moves the
validation into the reader so that if there is a hash error or size
mismatch the reader will return with an error instead of an io.EOF
causing the storage to abort the storage.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Use [chi](https://github.com/go-chi/chi) instead of the forked [macaron](https://gitea.com/macaron/macaron). Since macaron and chi have conflicts with session share, this big PR becomes a have-to thing. According my previous idea, we can replace macaron step by step but I'm wrong. :( Below is a list of big changes on this PR.
- [x] Define `context.ResponseWriter` interface with an implementation `context.Response`.
- [x] Use chi instead of macaron, and also a customize `Route` to wrap chi so that the router usage is similar as before.
- [x] Create different routers for `web`, `api`, `internal` and `install` so that the codes will be more clear and no magic .
- [x] Use https://github.com/unrolled/render instead of macaron's internal render
- [x] Use https://github.com/NYTimes/gziphandler instead of https://gitea.com/macaron/gzip
- [x] Use https://gitea.com/go-chi/session which is a modified version of https://gitea.com/macaron/session and removed `nodb` support since it will not be maintained. **BREAK**
- [x] Use https://gitea.com/go-chi/captcha which is a modified version of https://gitea.com/macaron/captcha
- [x] Use https://gitea.com/go-chi/cache which is a modified version of https://gitea.com/macaron/cache
- [x] Use https://gitea.com/go-chi/binding which is a modified version of https://gitea.com/macaron/binding
- [x] Use https://github.com/go-chi/cors instead of https://gitea.com/macaron/cors
- [x] Dropped https://gitea.com/macaron/i18n and make a new one in `code.gitea.io/gitea/modules/translation`
- [x] Move validation form structs from `code.gitea.io/gitea/modules/auth` to `code.gitea.io/gitea/modules/forms` to avoid dependency cycle.
- [x] Removed macaron log service because it's not need any more. **BREAK**
- [x] All form structs have to be get by `web.GetForm(ctx)` in the route function but not as a function parameter on routes definition.
- [x] Move Git HTTP protocol implementation to use routers directly.
- [x] Fix the problem that chi routes don't support trailing slash but macaron did.
- [x] `/api/v1/swagger` now will be redirect to `/api/swagger` but not render directly so that `APIContext` will not create a html render.
Notices:
- Chi router don't support request with trailing slash
- Integration test `TestUserHeatmap` maybe mysql version related. It's failed on my macOS(mysql 5.7.29 installed via brew) but succeed on CI.
Co-authored-by: 6543 <6543@obermui.de>
* Fix minio bug
* Add tests for storage configuration
* Change the Seek flag to keep compitable minio?
* Fix test when first-byte-pos of all ranges is greater than the resource length
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* LFS support to be stored on minio
* Fix test
* Fix lint
* Fix lint
* Fix check
* Fix test
* Update documents and add migration for LFS
* Fix some bugs
Replace all calls to os.Remove/os.RemoveAll by retrying util.Remove/util.RemoveAll and remove circular dependencies from util.
Fix#12339
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
* Initial support of end Range header option in lfs
* Allow end range option to be unspecified
* Declare toByte for consistency
* Factor out content encoding tests from doLfs
This is so Range tests could still use doLfs but without repeating
not related tests
* Add Range header test
* implemented extraHeader
* parametrized expectedStatus
* Add more test cases of Range header
* Fix S1030: should use resp.Body.String()
* Add more tests for edge cases
* Fix tests
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: zeripath <art27@cantab.net>
* Add more logging in the LFS server
Adds more logging in the LFS server and stops sending internal server
error information to the client
* Add LFS Lock cursor implementation
* Simplify Claims in LFS and remove the float64 casts
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
This PR adds basic repository LFS management UI including the ability to find all possible pointers within the repository. Locks are not managed at present but would be addable through some simple additions.
* Add basic repository lfs management
* add auto-associate function
* Add functionality to find commits with this lfs file
* Add link to find commits on the lfs file view
* Adjust commit view to state the likely branch causing the commit
* Only read Oid from database
Fixes#6960
According to [spec][1], /verify requests must have `Accept: application/vnd.git-lfs+json`
Previous code works because `git-lfs` also [violates spec and doesn't send any Accept header at all][2]
For other clients that DO set `Accept: application/vnd.git-lfs+json`, addition of `Accept: application/vnd.git-lfs`
either forces them to violate the spec or is ignored, depending on order in what they create header list.
[1]: https://github.com/git-lfs/git-lfs/blob/master/docs/api/basic-transfers.md#verification
[2]: https://github.com/git-lfs/git-lfs/issues/3662
* Panic don't fatal on create new logger
Fixes#5854
Signed-off-by: Andrew Thornton <art27@cantab.net>
* partial broken
* Update the logging infrastrcture
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Reset the skip levels for Fatal and Error
Signed-off-by: Andrew Thornton <art27@cantab.net>
* broken ncsa
* More log.Error fixes
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Remove nal
* set log-levels to lowercase
* Make console_test test all levels
* switch to lowercased levels
* OK now working
* Fix vetting issues
* Fix lint
* Fix tests
* change default logging to match current gitea
* Improve log testing
Signed-off-by: Andrew Thornton <art27@cantab.net>
* reset error skip levels to 0
* Update documentation and access logger configuration
* Redirect the router log back to gitea if redirect macaron log but also allow setting the log level - i.e. TRACE
* Fix broken level caching
* Refactor the router log
* Add Router logger
* Add colorizing options
* Adjust router colors
* Only create logger if they will be used
* update app.ini.sample
* rename Attribute ColorAttribute
* Change from white to green for function
* Set fatal/error levels
* Restore initial trace logger
* Fix Trace arguments in modules/auth/auth.go
* Properly handle XORMLogger
* Improve admin/config page
* fix fmt
* Add auto-compression of old logs
* Update error log levels
* Remove the unnecessary skip argument from Error, Fatal and Critical
* Add stacktrace support
* Fix tests
* Remove x/sync from vendors?
* Add stderr option to console logger
* Use filepath.ToSlash to protect against Windows in tests
* Remove prefixed underscores from names in colors.go
* Remove not implemented database logger
This was removed from Gogs on 4 Mar 2016 but left in the configuration
since then.
* Ensure that log paths are relative to ROOT_PATH
* use path.Join
* rename jsonConfig to logConfig
* Rename "config" to "jsonConfig" to make it clearer
* Requested changes
* Requested changes: XormLogger
* Try to color the windows terminal
If successful default to colorizing the console logs
* fixup
* Colorize initially too
* update vendor
* Colorize logs on default and remove if this is not a colorizing logger
* Fix documentation
* fix test
* Use go-isatty to detect if on windows we are on msys or cygwin
* Fix spelling mistake
* Add missing vendors
* More changes
* Rationalise the ANSI writer protection
* Adjust colors on advice from @0x5c
* Make Flags a comma separated list
* Move to use the windows constant for ENABLE_VIRTUAL_TERMINAL_PROCESSING
* Ensure matching is done on the non-colored message - to simpify EXPRESSION
This PR makes it possible for the markdown renderer to render images and media straight from the LFS.
Fix#5746
Signed-off-by: Andrew Thornton [art27@cantab.net](mailto:art27@cantab.net)
* Add failing test
* Fix urls
* Improve url in tests
* improve testing
* Remove debug code
* Add deps
* LFS corner-case : Search on lower but store with case
* Temporary comment of blocking action
* fix hooks
* Use temporary repo for git client test
* Use userPassword in place of hard-coded password
* Implement routes
* move to api/sdk and create model
* Implement add + list
* List return 200 empty list no 404
* Add verify lfs lock api
* Add delete and start implementing auth control
* Revert to code.gitea.io/sdk/gitea vendor
* Apply needed check for all lfs locks route
* Add simple tests
* fix lint
* Improve tests
* Add delete test + fix
* Add lfs ascii header
* Various fixes from review + remove useless code + add more corner case testing
* Remove repo link since only id is needed.
Save a little of memory and cpu time.
* Improve tests
* Use TEXT column format for path + test
* fix mispell
* Use NewRequestWithJSON for POST tests
* Clean path
* Improve DB format
* Revert uniquess repoid+path
* (Re)-setup uniqueness + max path length
* Fixed TEXT in place of VARCHAR
* Settle back to maximum VARCHAR(3072)
* Let place for repoid in key
* Let place for repoid in key
* Let place for repoid in key
* Revert back
* Import github.com/git-lfs/lfs-test-server as lfs module base
Imported commit is 3968aac269a77b73924649b9412ae03f7ccd3198
Removed:
Dockerfile CONTRIBUTING.md mgmt* script/ vendor/ kvlogger.go
.dockerignore .gitignore README.md
* Remove config, add JWT support from github.com/mgit-at/lfs-test-server
Imported commit f0cdcc5a01599c5a955dc1bbf683bb4acecdba83
* Add LFS settings
* Add LFS meta object model
* Add LFS routes and initialization
* Import github.com/dgrijalva/jwt-go into vendor/
* Adapt LFS module: handlers, routing, meta store
* Move LFS routes to /user/repo/info/lfs/*
* Add request header checks to LFS BatchHandler / PostHandler
* Implement LFS basic authentication
* Rework JWT secret generation / load
* Implement LFS SSH token authentication with JWT
Specification: https://github.com/github/git-lfs/tree/master/docs/api
* Integrate LFS settings into install process
* Remove LFS objects when repository is deleted
Only removes objects from content store when deleted repo is the only
referencing repository
* Make LFS module stateless
Fixes bug where LFS would not work after installation without
restarting Gitea
* Change 500 'Internal Server Error' to 400 'Bad Request'
* Change sql query to xorm call
* Remove unneeded type from LFS module
* Change internal imports to code.gitea.io/gitea/
* Add Gitea authors copyright
* Change basic auth realm to "gitea-lfs"
* Add unique indexes to LFS model
* Use xorm count function in LFS check on repository delete
* Return io.ReadCloser from content store and close after usage
* Add LFS info to runWeb()
* Export LFS content store base path
* LFS file download from UI
* Work around git-lfs client issue with unauthenticated requests
Returning a dummy Authorization header for unauthenticated requests
lets git-lfs client skip asking for auth credentials
See: https://github.com/github/git-lfs/issues/1088
* Fix unauthenticated UI downloads from public repositories
* Authentication check order, Finish LFS file view logic
* Ignore LFS hooks if installed for current OS user
Fixes Gitea UI actions for repositories tracking LFS files.
Checks for minimum needed git version by parsing the semantic version
string.
* Hide LFS metafile diff from commit view, marking as binary
* Show LFS notice if file in commit view is tracked
* Add notbefore/nbf JWT claim
* Correct lint suggestions - comments for structs and functions
- Add comments to LFS model
- Function comment for GetRandomBytesAsBase64
- LFS server function comments and lint variable suggestion
* Move secret generation code out of conditional
Ensures no LFS code may run with an empty secret
* Do not hand out JWT tokens if LFS server support is disabled
