Posts tagged ftps

FTP vs. FTPS vs. SFTP vs. FTPs, and FXP
posted on 2014-07-26 18:59:26

For file transferring, there exists three protocols that are usually used. All these have in common that a FTP daemon (ProFTPd, vsftpd, ServU, ...), or in case of SFTP, an SSH daemon (i.e. OpenSSH), has to run on the server. (For FTPs you'd need even both of them.) To daemon/service the user can then connect with a FTP client (i.e. Filezilla) running from his machine.

First some comments on authentication, because this topic often causes confusion. Although this is mostly due to users' misunderstanding SSH, having no idea what their firewall is doing and them trying wildly all the options they can find in their client. ;)

Authentication methods

There exist several 'logon modes' that can be used:

For FTP/FTPs:
- anonymous FTP: The server provides an anonymous account, with which the connection can be established. No authentication is done.
- password: user credentials in terms of a username and password have to be supplied

For SFTP:
- password: here also user credentials in terms of username:password have to be supplied
- public-key: having exchanged public key halves out-of-band before, these can be used for encryption later

Technically, SFTP can be set up to provide an anonymous account, too. How much use this idea is, having no user access control present but securing the data transmission channel, is left as a thought exercise to the reader. But if id Software can use FTPS for its anonymous account, so why should you not use SFTP for your own server? :)

For anonymous SFTP make sure to use rssh (a restricted secure shell), to prevent shell access but provide sftp access to the user. Then just creating a user named 'anonymous' without a password will do the trick. rssh is usually available through the package manager of your linux distribution.

In Filezilla (the most-used FTP client AFAIK), these options are available:

anonymous:          no user credentials are submitted, to be used for anonymous logins
normal:             user:pw credential combination
ask for password:   just the user is provided, the pw will be asked for once
interactive:        just the user is provided, the pw will be asked for every time
account:            if the server uses the ACCounT command, it can be passed here

FTP

Standard protocol, but no encryption is used. All data is sent unencrypted over the network. This includes user credentials as well as the data being sent.

First a visual representation I stole from RFC 959:

                                        -------------
                                        |/---------\|
                                        ||   User  ||    --------
                                        ||Interface|<--->| User |
                                        |\----^----/|    --------
              ----------                |     |     |
              |/------\|  FTP Commands  |/----V----\|
              ||Server|<---------------->|   User  ||
              ||  PI  ||   FTP Replies  ||    PI   ||
              |\--^---/|                |\----^----/|
              |   |    |                |     |     |
  --------    |/--V---\|      Data      |/----V----\|    --------
  | File |<--->|Server|<---------------->|  User   |<--->| File |
  |System|    || DTP  ||   Connection   ||   DTP   ||    |System|
  --------    |\------/|                |\---------/|    --------
              ----------                -------------

              Server-FTP                   USER-FTP

Channels

Two kinds of connections are opened. One is the command, one the data channel. The data channel is solely used for payload transfers, and is opened when needed and closed when transfers are completed. Whereas the command channel stays open all the time, and handles things like authentication, starting transfers, closing the connection. FTP uses the TELNET protocol for exchanging control messages.

Ports

The command channel is an established TCP connection between a random port on the client (Port N, where N is an available port number.) and by default on port 21 of the server. "Random" usually means a port above 1023.

The data channel is established on port 20 on the server and channel N+1 on client side, as soon as one is needed for data exchange. Once the transfer is done, the channel is closed. It will be reopened, in case it is needed again.

ACTIVE vs. PASSIVE mode

This is just a matter of direction. Since usually ingres connections are filtered by the firewall, but egress traffic not so, passive mode can be used to work your way around restrictive firewalls on the client's side.

In short: (standard ports are used in this example.)

  • ACTIVE: client to server control connection onto port 21, server to client data connection from 20 to the port the client told the server to connect to. Connections change direction.
  • PASSIVE: client to server control connection onto port 21, client to server data connection to a port from the passive port range the server told the client. Connections always go from client to server.

The connections are build up in a different order, control channels (---) and data channels (===) shown below:
(ports above 1024 are arbitrary ones, depending on settings and current OS state.)

ACTIVE
======
client (1027) connects to server (21), establishing control channel
- client sends PORT command with its ip and opened port 1028 through control channel
- other commands may follow
SERVER (20) connects to CLIENT (1028), establishing data channel.


        CLIENT                                SERVER

port  1028  1027                            21     20
        |     |                              |      |
        |     |------ 1: PORT 1028     ----->|      |
        |     |                              |      |
        |     |<----- 2: ACK           ------|      |
        |     |                              |      |
        |     |                              |      |
        |     |                              |      |
        |<=========== 3: data transfer =============|
        |     |                              |      |
        |     |                              |      |
        |============ 4: ACK           ============>|
        |     |                              |      |
        |     |                              |      |


PASSIVE
=======
client (1027) to server (21), establishing control channel
- client sends PASV command through control channel
- server answers which port is has available for data channel through control channel
- other commands may follow
CLIENT (1028) connects to SERVER (2033), establishing data channel.


        CLIENT                                SERVER

port  1028  1027                            21     20     2033
        |     |                              |      |      |
        |     |------ 1: PASV          ----->|      |      |
        |     |                              |      |      |
        |     |<----- 2: PORT 2033     ------|      |      |
        |     |                              |      |      |
        |     |                              |      |      |
        |     |                              |      |      |
        |============ 3: data transfer ===================>|
        |     |                              |      |      |
        |     |                              |      |      |
        |<=========== 4: ACK           ====================|
        |     |                              |      |      |
        |     |                              |      |      |

Digits in brackets denoted ports, of course.

So while in passive mode the server admin has to open the high ports in his firewall, in active mode to client admin has to do so in his. Pretty simple stuff, one might guess. Real world tells: 'not so much'.

Further there can problems arise when FTPS/FTPES is used. This is also firewall-related, see the FTPS section at its end.

Commands, specification vs. actual implementation

Available commands are, according to RFC 959:
Note, not all these are implemented in all FTP clients/servers. Still all of them are listed, to get a better grasp on FTP and what happens beside what you see during its usage.

USER <username>             pass username to server
PASS <password>             pass password to server
ACCT <account-information>  pass account information to the server
CWD  <pathname>             change working directory
CDUP                        change working directory to parent
SMNT <pathname>             Structure MouNT, allow mounting of a different file system
QUIT                        quit connection
REIN                        REINitialize connection to state immediatly after opening control channel, prior to USER
PORT <host-port>            data port on host, can be several, comma-delimited
PASV                        enable passive mode
TYPE <type-code>            ascii or ebcdic plaintext, image (bytestream), local proprietary format for data during network transfer
STRU <structure-code>       set file, record, page for file structure
MODE <mode-code>            choose transfer mode (stream, block, compressed)
RETR <pathname>             download data
STOR <pathname>             STORe / upload data
STOU                        STORe Unique, save data under unique filename in CWD
APPE<pathname>              upload and APPEnd data
ALLO<decimal-integer> [<SP> R <SP> <decimal-integer>]
                            sometimes needed to ALLOcate space on server prior to upload
REST <marker>               restart, only works for block and compressed modes
RNFR <pathname>             old name to be renamed
RNTO <pathname>             new name, old name to be renamed to
ABOR                        abort currently running operation and associated data channels
DELE <pathname>             delete file on server
RMD  <pathname>             delete directory
MKD  <pathname>             create directory
PWD                         show current directory
LIST [<pathname>]           transfer filename list or file information from server
NLST [<pathname>]           Name LiST, transfer filename list
SITE <string>               send specific commands to remote server
SYST                        return system type of server
STAT [<pathname>]           show ftp status
HELP [<string>]             show help

Often you also find custom ones, ones that are not listed here. That is all dependant on the actual implementation of the client you use:

After all, RFC's are just specifications. These describe, how software/systems should look like.

Specifications are not Documentations, which describe how systems actually are shaped in reality. (In case the documentation is up-to-date...)

So you very likely might encounter these:

BYE                         same as QUIT
LS                          same as LIST
DIR                         same as LIST

Another command actually is the !. It runs local commands, see:

[sjas@beckett ~]% ftp
ftp> help !
!           escape to the shell
ftp>

There are also other commands specified (There exists more than one RFC for FTP stuff.), that take on encryption or other functionalities. In case you really are curious, see:

RFC 697 - CWD Command of FTP
RFC 959 - File Transfer Protocol (FTP)
RFC 1639 - FTP Operation Over Big Address Records (FOOBAR)
RFC 2026 - UTF-8 Option for FTP (Draft 2002) [1] - draft-ietf-ftpext-utf-8-option-00.txt
RFC 2228 - FTP Security Extensions
RFC 2389 - Feature negotiation mechanism for the File Transfer Protocol
RFC 2428 - FTP Extensions for IPv6 and NATs
RFC 2640 - Internationalization of the File Transfer Protocol
RFC 3659 - Extensions to FTP
RFC 5797 - FTP Command and Extension Registry

A list on all its return codes (200, ...) can be found here.

FTPS / FTPES (FTP over SSL, implicit or explicit)

Here the transfers are encrypted using certificates within a public key infrastructure (PKI), see X.509 certificates.

Implicit and explicit modes are possible:

Implicit: forces TLS, or quits the connection.
Explicit: uses TLS if possible, but works unencrypted in case encryption is not possible.

Implicit forces encryption for all commands/data being transferred.
Explicit lets the client choose, what he wants to have encrypted. (data channel? control channel? both? none?)

Usually implicit is not used anymore, if I am not mistaken.

Besides having SSL/TLS encryption, FTPS/FTPES is standard FTP.

FTPS firewall problems

There are occasions where FTPS will fail between two computers, whereas FTP works. This is usually due to 'intelligent' firewalls, watching out for FTP control messages. Upon finding these (PORT command!), they know which of their ports they have to open up for data exchange.

If however the connection is encrypted, this packet analysis will no longer work. So the needed ports for the data channel stay closed, causing SFTP to fail.

SFTP (Secure FTP, uses SSH)

Since SSHv2, SFTP is available. (SSHv1 implementations only provided 'secure copy'.)

Not much left to explain (except maybe SSH in depth, which is no use here.), so it will be cut short:

FTP/FTPs is an entirely different protocol from SFTP.
I.e. where FTP needs two connections (1x command, 1x data channel), SFTP only needs one. Which is from a random port on the client side, to port 22, the SSH port, on the server side. All data exchange is done through ssh specific commands. These may be similar to the ones specified above. But besides similarities they have nothing in common, they don't stem from the same specification.

On a sidenote, scp uses sftp internally.

FTPs (FTP over SSH)

This one is a rare occurance and usually seldom used. (Except by these notorious people that will do things, just because they can.)

Comparisons for easier understanding, spot the differences:

=======================================================
FTP
=======================================================

                     |  plain        |  encrypted
---------------------+---------------+-------------
   control channel   |  FTP          |
   data channel      |  FTP          |

=======================================================
FTPs, FTP-over-SSH
=======================================================

                     |  plain        |  encrypted
---------------------+---------------+--------------
   control channel   |               |  FTP-over-SSH
   data channel      |  FTP-over-SSH |

=======================================================
FTPS / FTPES, FTP-over-SSL
=======================================================

                     |  plain        |  encrypted
---------------------+---------------+--------------
   control channel   |               |  FTP-over-SSL
   data channel      |               |  FTP-over-SSL

=======================================================
SFTP, Secure FTP (only one channel!)
=======================================================

                     |  plain        |  encrypted
---------------------+---------------+--------------
control+data channel |               |  SFTP

Instead of using the regular FTP (the 'plaintext' version, if I may say so), where both control and data channel are unencrypted, a SSH tunnel is established prior to FTP usage and its encryption is used for the control channel. The data is transmitted without encryption.

So USER and PASS commands cannot be read in plain text anymore, which makes FTPs better than regular FTP.
Since the initial overhead of creating the SSH tunnel is to complicated, people stick to FTP/FTPS/SFTP instead of FTPs. Also as noted at the beginning, you'd need both a ftp server and a ssh server running and and a tunnel previously and properly set up.

But why bother and not just either go the easy way with FTP or the secure with SFTP or FTPS?

FXP (File Exchange Protocol)

This one has nothing at all to do with encryption, at least per se. It's also only possible with the FTP variants, not the SFTP one.

Scenario: Say you have a client, and two servers. (Instead of regular FTP where you just have a client and a single server.)

With regular FTP used, to get files from server1 to server2, the transfer would have to work over the client (and it's possible bad connection). No direct connection is possible from server1 to server2 over the 'good'/fast network.

With an FXP client, direct transfers are possible, if the FTP servers support this functionality. It is achieved by exchanging the ip and ports of server1 to server2 via the client machine.

Basically, the client connects to both servers at once. He negotiates connection between himself and server1 in passive mode, to know which port server1 has open for data exchange. He tells server2, and calls for data transmission. server1 will be contacted by server2, data channel is established, and files will flow directly.

The first server here was contacted via passive mode. The second was passed the first servers ip and port.

Of course this should also work with active/passive switched, but the client has to have to know the second server's open ports prior, without being told by the server, which complicates things. Without this information, passing the first server in the first step the ip of the second server and a wrong port makes just no sense.

Lastly two sidenotes:
Usually when customers are unable to use SFTP/FTPS it's due to them forgetting to adjust the protocol in their FTP client (i.e. Filezilla). Or they miss the proper port. Or they changed encodings/transfer modes/passive while trying to get things to run...

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apachebench, apple, arcconf, arch, architecture, areca, arping, asa, asdm, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, coleslaw, colorscheme, common lisp, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fbsd, fedora, file, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, github, gitolite, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, innodb, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pvecm, pvresize, python, qemu, qemu-img, qm, qmrestore, quicklisp, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh

View posts from 2017-03, 2017-02, 2017-01, 2016-12, 2016-11, 2016-10, 2016-09, 2016-08, 2016-07, 2016-06, 2016-05, 2016-04, 2016-03, 2016-02, 2016-01, 2015-12, 2015-11, 2015-10, 2015-09, 2015-08, 2015-07, 2015-06, 2015-05, 2015-04, 2015-03, 2015-02, 2015-01, 2014-12, 2014-11, 2014-10, 2014-09, 2014-08, 2014-07, 2014-06, 2014-05, 2014-04, 2014-03, 2014-01, 2013-12, 2013-11, 2013-10


Unless otherwise credited all material Creative Commons License by sjas