crazegift.blogg.se - Git annex download

#Git annex download install
#Git annex download registration

This git-annex repository contains 100k files, the entire collections "internetarchivebooks" and "usenethistorical". If we succeed, we will have backed up 1/1770th of the Internet Archive. This is our first part of the IA that we want to get backed up.

Client runtime environment (docker image maybe?) with warrior-like interface (all that needs to do is configure things and get git-annex running).

Write pre-receive git hook, to reject pushes of branches other then the git-annex branch (already done), and prevent bad/malicious pushes of the git-annex branch.

Create all 1770 shards, and see how that scales.

Help the user get the iabak-cronjob set up.

#Git annex download registration

Write client registration interface, which generates the client's ssh private key, git-annex UUID, and sends them to the client (done).

Get the clients to upload it to our server. Tell git-annex the content is no longer in the IA.

get fscking and dead client expiry working (done).

Get that first shard backed up enough to be able to say, "we have successfully backed up 1/1770th of the IA!" (done!).

Manually register a few clients to start, have them manually download some files, and `git annex sync` their state back to the server.

Put one shard repo on the server to start.

It needs to accept incoming ssh connections from registered clients, only letting them run git-annex-shell. Any linux system with a few hundred gb of disk and ssh and git-annex installed will do.

Set up a server to serve up the git repos.

Write a script to generate a git-annex repository with 100k files from the list.

Get a list of files, checksums, and urls.shuf (optional - will randomize the order you download files in).crontab OR systemd (NOTE: you may need to run loginctl enable-linger to make sure the job is not killed).sane UNIX environment (shell, df, perl, grep).To adjust this value later, use git config annex.diskreserve 200GB in all of the IA.BAK/shard* directories.Ĭonfiguration and maintenance information can be found in the README.md file. It should prompt you for how much disk space to not use. timer unit) to perform periodic maintenance.

#Git annex download install

It will walk you through setup and starting to download files, and install a cron job (or.