Posts tagged nginx

Raspberry Pi: Seafile installation from scratch plus WebDAV access
posted on 2015-04-21 21:34:33

the use case

After dropbox cut a program I took part in my free space went down from 25gb to 3gb, I had a reason to get an alternative up and running. iOS apps should work with it, too, as long as the can work via WebDAV.

There are a lot of comparisons of owncloud, pydio, seafile and all the alternatives, but I am somehow suspicious of pydio, owncloud has problems once you start having too many files it seems, so I ended up with a seafile test. The results were great, so a pi was bought and this guide is the result.

what will be covered

Here a rather detailed setup howto is given, to get a seafile install on a brand new raspberry pi. It will contain side info's on the networking stuff. These tend to be not covered in almost all the other guides usually since all this is in general considered 'trivial'. Which just means everybody was to lazy to give hints where that stuff just is NOT trivial, and actual work to explain properly.

Seafile will run with a mysql backend, and with an nginx webserver so I can get some education on it myself, having worked almost only with apaches until now.

However there are no guarantees, as once again, this is partly written just from memory.

prerequisites

You need these items:

  • a raspberry pi (get the latest model and be happy)
  • a case for the rasp, so it won't lie around in the open
  • a micro SD card with 4gb, with an SD card adapter
  • a cardreader (to write the pi's OS onto the micro SD with the SD adapter)
  • an AC adapter for the pi with a micro USB B connector
  • an ethernet cable (just a network cable with rj45 plugs)

And one of these:

  • a HDD/SSD with own power connection
  • or a usb-only HDD/SSD plus an USB hub with own power supply

If you try an external drive without an extra power supply, it won't run. The pi simply cannot provide enough power via its USB ports. You can see this through the red LED. If the voltage drops below 4.6V or something, it will flicker or just turn off.

DynDNS

Also some kind of DynDNS service would be helpful. Since there seemed to be a lot of trouble with the free ones, either spend some money or set up your own.

Since I was bored and have already a DNS server running and have a domain, I chose to roll my own setup. In that way you either run your whole domain via your DNS server (means your DNS is the primary domain server for your domain), or you can try using 'subzone delegation', so your DNS server runs only a subdomain whereas the domain hoster will run your 'main' domain.

On how to do this, get some other tutorials, I covered it here.

setup the system

get the OS

Install the hardware, which should not pose a problem. If it does, seriously get someone with more knowledge on computers to help you!

Get the Raspbian image, which is a debian-based OS for the pi. Download and unzip it.

get the OS onto the SD card

Open one console and enter watch -d -n1 lsblk, then insert the SD card. That way you will know what the device is called on your linux box.

Open another shell window, and then put the raspbian image onto the card:

dd if=/path/to/the/<raspbian-file.iso> of=/dev/sdX

Of course, fix path and device in the line above.

If you want to know how fast the copy process runs, try:

ps aux | grep dd

And search for the process id of the dd process from above. Then do in another shell window:

watch -n5 kill -usr1 <dd-process-id>

That way the copy process stats will be shown in the dd window every five seconds, which is nice since the 3GB image takes some time to copy. But pardon, I disgress.

fix ssh and networking

Once the copying is finished, mount the card and fix ssh. That way you will not need to hook up the pi to a keyboard and monitor (I have no HDMI capable screen here, so ...) but just connect the ethernet cable and be done.

So:

mkdir asdf
sudo mount /dev/sdX asdf
cd asdf
vim etc/network/interfaces

then either hand the eth0 interface a DHCP configuration (which is stupid), ar just give it a fixed ip.

If your home network is set to the 192.168.0.0/24 net, try configuring this:
If your network is 192.168.178.0/24 or 10.0.0.0/24, fix the ip in the following examples.

allow-hotplug eth0
iface eth0 inet static
    address 192.168.0.254
    netmask 255.255.255.0
    gateway 192.168.0.1
    network 192.168.0.0
    broadcast 192.168.0.255
    dns-nameservers 192.168.0.1

If you cannot use vim, try the damned nano or whatever editor you fancy.

Also these might be a good idea:

echo pi > etc/hostname

and

vim etc/resolv.conf

where you'd enter this line:

nameserver 192.168.0.1

Save and quit.

vim etc/ssh/sshd_config

and make sure there is this:

PermitRootLogin yes

If it were PermitRootLogin without-password, you'd not be able to connect. Save, close.

cd 
sudo umount /dev/sdX

And you can pull the SD card out and put it into the pi.

Hook up your pi to the network via network cable to your home router.

Try ping 192.168.0.254 and see if something answers after the pie has booted (which should take no longer than some minutes, I never measured the time.). If it doesn't work, you either have network issues, or misconfigured something above.

If it answers your ping, get a host entry so connecting to the pi is easier: (/etc/hosts entries are basically local DNS records, this will do you no harm.)

echo '192.168.0.254 pi' >> /etc/hosts

and copy your ssh key onto it, so passwordless login will work:

ssh-copy-id root@pi

You could also use ssh-copy-id 192.168.0.254 - it will work the same, but in the further text the pi ip will be referenced by the local dns name 'pi'. Period. Try it:

ssh root@pi

And you should be connected.

disk preparation

For the following it is assumed, that you have plugged in the usb hub/hdd already, partitioned it and created a file system. Choose your tools and filesystem to use, do it and do not forget to mount the disk afterwards.

To get the disk to be permanently included (across reboots), add it to /etc/fstab.

My entry looks like this:

/dev/sdb1       /var/seafile    btrfs   defaults          0       2

(Since the harddisk is /dev/sdb, with a single partition, filesystem btrfs or what you have to specify for mount -t when mounting by hand, default mount -o options, no dump, not root. In case you doubt what disk your harddisk is, try lsblk, the size should tell you which one to use. The last number is actually about the filesystem check: 0=off, 1=first, 2=afterwards. root is set to 1.)

The directory /var/seafile was created by me for later usage via mkdir, so I have a working mountpoint.

To reload /etc/fstab, a mount -a will do. All this was done as the root user.

Sidenote:
lsblk will not show the mountpoints for btrfs volumes, so you have to use mount to check if everthing looks as expected.

actual seafile install

Install will be done with the MySQL Backend, as the installer tells about Problems when using an USB disk (which we do) and SQLite.

get the install files

Head over to the official download section, so you will get the newest install files. Here choose the raspberry package. Intel Stuff, wether 32 or 64 bit, will not work, since the raspberry has an ARM processor. See the output of uname -m if in doubt.

prepare the system

Copy the link location, for wget'ing it later. Lets also create a dedicated user, too, as it is better to run the program without root rights, security-wise.

apt-get install python2.7 python-setuptools python-imaging mysql-server python-mysqldb -y

Remember the mysql root password, you will need it later on.

mkdir /opt/seafile
useradd seafile
chown seafile.seafile /opt/seafile

Also chown the seafile folder for the seafile user else the installer will have troubles:

chown seafile.seafile /var/seafile
chmod 775 /var/seafile

installing

su - seafile
wget https://github.com/haiwen/seafile-rpi/releases/download/v4.1.2/seafile-server_4.1.2_pi.tar.gz
tar xzvf seafile-server_4.1.2_pi.tar.gz
mkdir installed
mv seafile-server_4.1.2_pi.tar.gz installed/
cd seafile-server-4.1.2/

seafile setup

./setup-seafile-mysql.sh

Enter information:

NAME: is just a label
IP / DOMAIN: enter the pi's ip if you use seafile only on LAN / via VPN, or the dynamic dns
CCNET PORT: default on 10001, since there isn't anything running besides
PATH: /var/seafile/seafile-data, since /opt/seafile is on the SD card, and the data should go to the harddisk's mountpoint
SERVER PORTS: defaults, respectively 12001, and 8082
DATABASE INIT: 1, create new tables
MYSQL HOST: default, localhost
MYSQL PORT: default, 3306
MYSQL ROOT PASSWORT: the one you gave to mysql during install
MYSQL USER: seafile
MYSQL SEAFILE PASSWORT: use a new one
DATABASE NAMES: all default

After all this, the configuration is almost done. Basically the server can be started and run:

Since we want to use the services like any other ones usually, we will just link the scripts into /etc/init.d/. The following stuff is again done as root user:

ln /opt/seafile/seafile-server-latest/seafile.sh /etc/init.d/seafile
ln /opt/seafile/seafile-server-latest/seahub.sh /etc/init.d/seahub

I used hard links on purpose, but I cannot remember why I though this was a good idea.

Anyway, now you can do just:

service seafile start
service seahub start

and you will be asked to setup an seafile admin account, this time the one for the web interface, not the DB user.

Just use your email and yet another password, and remember them. You have to use this account for creating new user accounts, libraries, in short everything.

If your webinterface does not work, you might have to (re)start both services, in case you forgot one.

test seahub

Now, going into your browser, and entering the raspberry's local IP plus port 8000 should get you to the login screen. pi:8000 in the addressbar should do btw, when you set up the above mentioned /etc/hosts entry.

With the web login you just created, you should be able to login. :)

Looks promising so far.

open your firewall for external access

When using the service externally, without a VPN connection, don't forget to open port 8000 in your firewall/router. For testing the webgui, this is enough. To actually use seafile, these must be reachable: (copy-paste from the seafile install message)

port of ccnet server:         10001
port of seafile server:       12001
port of seafile fileserver:   8082
port of seahub:               8000

Later we will also open 8080 so WebDAV can be used. And 80 and 443 where the nginx will be listening to, too.

Of course, the services can be run on arbitrary ports. You might as well leave everything on default for the pi, but use different ports in the router forwarding to the actual ones on the pi, for security reasons. If you know what you are doing are bored when reading this, oh well. Just change it to your liking. Everybody else use default ports, makes the setup easier to debug.

security considerations

When using the service from the outside without a VPN or SSH tunnel, your traffic is plaintext. 'Thou shalt use thee TLS encryption!' in that case, but for that you will have to use a proper web server instead of the built-in one that comes with seafile.

Read: apache or nginx.

WebDAV

Since this is not everything which is needed, the WebDAV plugin has to be integrated.

configure webdav

Without a dedicated webserver this is rather easy.

In /opt/seafile/conf/seafdav.conf, set:

enabled = true

Save and close, plus afterwards:

service seafile restart

test webdav

Besides using webdav from the iOS app in question, you can as well us the linux command line to test, via the davfs2 package. Install it on your home computer (if you ran a linux there, else have a look at the official manual). As root do these:

In /etc/davfs2/davfs2.conf set:

use_locks 0

Save, close.

mkdir /mnt/davtest
mount -t davfs -o uid=<your linux system user> http://pi:8080 /mnt/davtest

Then you will be prompted for user credentials, you might just use the Web UI login from above. The one you entered, when you first started the seahub service.

The /mnt/davtest should now contain some more stuff, meaning WebDAV access works.

If in doubt, create a file with i.e. touch testfile in that folder, which you then can see in the web interface.

Now I have to repeat, using this externally means unencrypted data over the wire. Set up a proper webserver with TLS and configure WebDAV there, if you plan on using this setup from the outside of your home LAN without a VPN. That way you can also use proper fastcgi. ;)

a proper webserver - nginx with TLS

Since I need some nginx practice, this one will be used here.

First lets get the certificates up and running:

mkdir /etc/ssl/nginx
cd /etc/ssl/nginx
openssl genrsa 2048 > ca-key.pem
openssl req -new -x509 -nodes -days 3650 -key ca-key.pem -out ca-cert.pem
openssl req -newkey rsa:2048 -days 3650 -nodes -keyout server-key.pem -out server-req.pem
openssl rsa -in server-key.pem -out server-key.pem
openssl x509 -req -in server-req.pem -days 3650 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem

When prompted to enter something, do as you like. You could also hit just ENTER all the time until it's finished.

Then lets fix the domains the server is bound to:

In /opt/seafile/ccnet/ccnet.conf:

SERVICE_URL = https://www.yourdomain.com

In /opt/seafile/seahub_settings.py:

FILE_SERVER_ROOT = https://www.yourdomain.com/seafhttp

Now onto the nginx config, you just have to change your domain below. Open /etc/nginx/sites-available/yourdomain.com and change it accordingly, to have http directed to https and a working https:

server {
        listen       80;
        server_name  dyn.sjas.de;

        # force redirect http to https
        rewrite ^ https://$http_host$request_uri? permanent;
}

server {
        listen   443;
        server_name yourdomain.com;

        ssl on;
        ssl_certificate /etc/ssl/nginx/server-cert.pem;
        ssl_certificate_key /etc/ssl/nginx/server-key.pem;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_session_timeout 5m;
        ssl_prefer_server_ciphers on;

        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_connect_timeout       300;
        proxy_send_timeout          300;
        proxy_read_timeout          300;
        send_timeout                300;

        location / {
                fastcgi_pass   127.0.0.1:8000;
                fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
                fastcgi_read_timeout 300;

                fastcgi_param  HTTPS            on;
                fastcgi_param  HTTP_SCHEME      https;
                fastcgi_param  PATH_INFO        $fastcgi_script_name;
                fastcgi_param  SERVER_PROTOCOL  $server_protocol;
                fastcgi_param  QUERY_STRING     $query_string;
                fastcgi_param  REQUEST_METHOD   $request_method;
                fastcgi_param  CONTENT_TYPE     $content_type;
                fastcgi_param  CONTENT_LENGTH   $content_length;
                fastcgi_param  SERVER_ADDR      $server_addr;
                fastcgi_param  SERVER_PORT      $server_port;
                fastcgi_param  SERVER_NAME      $server_name;
                fastcgi_param  REMOTE_ADDR      $remote_addr;

                access_log      /var/log/nginx/seahub.access.log;
                error_log       /var/log/nginx/seahub.error.log;
        }

        location /seafhttp {
                rewrite ^/seafhttp(.*)$ $1 break;
                proxy_pass http://127.0.0.1:8082;
                client_max_body_size 0;
                proxy_connect_timeout  36000s;
                proxy_read_timeout  36000s;
        }

        location /media {
                root /opt/seafile/seafile-server-latest/seahub;
        }
}

Now just create a proper link for sites-enabled and restart nginx:

ln -s /etc/nginx/sites-available/yourdomain.com /etc/nginx/sites-enabled/yourdomain.com
service nginx restart

From the browser you should be able to test things now, https://pi.

automatically start everything

Let us make them known as services to be run on startup. This should be not too hard, but turned out to be a hairy problem.

Just using update-rc.d on the already present files won't work. seahub will start, but seafile will not.

Just putting service seafile start; service seahub start into /etc/rc.local will not work either. That way seafile will start, but seahub will not. Oh my.

Also, seahub tries to find seafile. (... ... ...)

Long story short: Open /etc/init.d/seafile and comment out in about line 150 the "warning_if_seafile_not_running" function:

function before_start() {
    check_python_executable;
    validate_ccnet_conf_dir;
    read_seafile_data_dir;

    #warning_if_seafile_not_running;

That way the check is turned off, and seahub will come up. I honestly have no idea where the problem lies, but its related to no proper startscripts being provided.

In the official manual there does exist a skeleton that you can adapt... Sadly that stuff over there is pretty outdated. Also I simply chose not to put up with it, as it will just wrap the scripts we are currently using.

By now this article is finished, and you should have a raspberry with a working seafile+webdav install.

offtopic

For fun and educational purposes, some words on initialization scripts, through a very pointy little anecdote:

Wrapping scripts with another script will cause endless headaches when things go haywire.

On a legacy system of rather complex web application with borked initscripts three really great people could not find the error over the course of like 1,5 years, and not for the lack of trying. File encodings, no proper initscripts, a subcontractor playing dumb (and not really having a clue, developers just ain't sysadmins, restarting via 'their' scripts did work, after all), all for a medium-sized clustered (but partly dysfunctional, of course) production system with harsh uptime requirements. To further worsen everything, several people on customer side had nagios notifications for EVERY SINGLE SERVICE, which got checked every 5 minutes, each. You could not count the SMS arriving, when a host went down. Even rebooting a service could cause a MESS, which did of course not help for locating the error. The initscripts (several application instances on each of the machines running) wrapping scripts which wrapped a script which wrapped scripts. Encoding was set in several applications, the system, also on boot time within grub. You name it, a puzzle in a puzzle in a puzzle in a puzzle.

I love bash, but debugging bash environments from scripts referencing each other is something you might have to do, when you happen to be in hell where you have to burn for your sins. At least that is how I imagine it.

Final result somewhere was a forgotten - after a su. Once I found it, my day was over. Will never forget this moment I found the cause, even if get a hundred years old. WRITE PROPER INITSCRIPTS, PEOPLE!

TODO

On the TODO list for this system could be:

  • logrotate and proper logging, since these are written in /opt/seafile/logs on the SD card, which is bad
  • a ramdisk for the /tmp folder
  • a custom fail2ban setup using the seafile configs

But for now, this post is finished.

To the brave soul reading this:
I hope you did like this little write up.

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apachebench, apple, arcconf, arch, architecture, areca, arping, asa, asdm, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, coleslaw, colorscheme, common lisp, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fbsd, fedora, file, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, github, gitolite, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pvecm, pvresize, python, qemu, qemu-img, qm, qmrestore, quicklisp, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh

View posts from 2017-02, 2017-01, 2016-12, 2016-11, 2016-10, 2016-09, 2016-08, 2016-07, 2016-06, 2016-05, 2016-04, 2016-03, 2016-02, 2016-01, 2015-12, 2015-11, 2015-10, 2015-09, 2015-08, 2015-07, 2015-06, 2015-05, 2015-04, 2015-03, 2015-02, 2015-01, 2014-12, 2014-11, 2014-10, 2014-09, 2014-08, 2014-07, 2014-06, 2014-05, 2014-04, 2014-03, 2014-01, 2013-12, 2013-11, 2013-10


Unless otherwise credited all material Creative Commons License by sjas