Install S3FS on Ubuntu 14.04 LTS

Door Joachim op wo, 09/03/2014 - 23:03

S3FS is a virtual filesystem that enables you to mount an Amazon S3 bucket as a folder on your computer or server.

In this article, we'll provide you with a step-by-step guide to install S3FS on Ubuntu 14.04.

Install the required packages

Execute following command as root (or via sudo):

apt-get install build-essential libfuse-dev libcurl4-openssl-dev libxml2-dev mime-support automake libtool

Install S3FS

Download and unpack S3FS in a temporary folder. Change the version number if there is a new release. Make sure to execute the last command (make install) as root!

cd /tmp
wget https://github.com/s3fs-fuse/s3fs-fuse/archive/v1.77.tar.gz
mv v1.77.tar.gz s3fs-fuse-1.77.tar.gz
tar zxvf s3fs-fuse-1.77.tar.gz
cd s3fs-fuse-1.77/
./autogen.sh
./configure --prefix=/usr
make
make install

Try it out!

First, we need to enter our AWS credentials in a place where s3fs can find them. I prefer to place this file globally in /etc/passwd-s3fs. Just enter your Credentials (AWS Access Key and Secret Access Key) in following format:

AWS_ACCESS_KEY:SECRET_ACCESS_KEY

and secure the file for unauthorized access:

chmod 640 /etc/passwd-s3fs

Then we can try to mount our bucket in the filesystem.

s3fs bucketname /my/mountpoint

Ensure the bucket is mounted at boot

To make sure the bucket is mounted again in case of a reboot, we'll need to edit our /etc/fstab file. Add a line like below

s3fs#bucketname /my/mountpoint fuse allow_other 0 0

Optimizing our s3fs

To avoid read/write errors and speed up access to frequent files, or reduce costs there are some interesting options that can be included when mounting our bucket.

retries

The number of times s3fs will retry a failed s3 transaction. By default this is set to 2, but we could increase this if we face errors.

To use this commandline: 

s3fs -o retries=5 bucketname /my/mountpoint
To use this in /etc/fstab:
s3fs#bucketname /my/mountpoint fuse retries=5,allow_other 0 0

use_cache

By adding this parameter, we can use a local folder to cache the files that are written to and retrieved from s3. This can really speed up things (and reduce costs) if you frequently request the same files.

To use this commandline:

s3fs -o use_cache=/tmp_bucket bucketname /my/mountpoint

To use this in /etc/fstab:

s3fs#bucketname /my/mountpoint fuse use_cache=/tmp_bucket,allow_other 0 0

use_rrs

This way, you can use the Reduced Redundancy Storage, which - if your use case permits it - will give you a considerable cost benefit.

To use this commandline:

s3fs -o use_rrs=1 bucketname /my/mountpoint
To use this in /etc/fstab:
s3fs#bucketname /my/mountpoint fuse use_rrs=1,allow_other 0 0

Our setup

We use s3 buckets continuously, and really like to have them available on boot, so we added following line for each bucket in our /etc/fstab

s3fs#bucketname /my/mountpoint fuse retries=5,use_cache=/tmp_bucket,use_rrs,allow_other 0 0

Pitfall

Using a local cache is nice, it will speed up things and reduce cost, but s3fs doesn't manage the size of this local cache folder. To limit the size of our local cache folder, we could create a cron job to periodically clean up all files that were not accessed in the last 7 days, or not modified during the last month by adding following line to root's crontab:

@daily find /tmp_bucket -atime +7 -mtime +30 -exec rm {} \;

Do you have other cleanup strategies? Do let us know in the comments!

Reacties

Door Dave op ma, 01/26/2015 - 19:39

"I prefer to place this file globally in /etc/passwd_s3fs. " Should be "I prefer to place this file globally in /etc/passwd-s3fs. "

Door robustadmin7 op do, 04/09/2015 - 22:19

Dave,

Thanks for the good eyes. It is now corrected.

Door gurayc op ma, 05/25/2015 - 21:15

Very useful. Thanks.

Door Jeff Bristow op vr, 08/14/2015 - 14:10

Thank you so much for this. Got this running in just a few minutes. I do have one question though, how would I mount to a specific folder in my S3 storage?

Door Khan op zo, 06/19/2016 - 00:07

Hello, i followed your guide, i am getting following error at "make"
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_sasl_bind@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_get_dn_ber@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ber_sockbuf_add_io@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_unbind_ext@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_get_attribute_ber@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_parse_result@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_set_option@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_abandon_ext@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_msgfree@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_result@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_search_ext@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_get_option@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ber_memfree@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_memfree@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_pvt_url_scheme2proto@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_next_message@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ber_free@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_err2string@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_init_fd@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_msgtype@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_free_urldesc@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_url_parse@OPENLDAP_2.4_2'
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `ldap_first_message@OPENLDAP_2.4_2'
collect2: error: ld returned 1 exit status
make[2]: *** [s3fs] Error 1
make[2]: Leaving directory `/tmp/s3fs-fuse-1.80/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tmp/s3fs-fuse-1.80'
make: *** [all] Error 2

Door Conor op za, 12/17/2016 - 04:18

Hi Joachim,

Quick question as I'm a little confused with how you wrote it in your post. Where's the best place to put the cache folder? Should I create a folder specifically for it or are you just throwing it in the /tmp folder? I assume by putting it in /tmp, the cache will be flushed on reboot

Many thanks for your excellent post by the way. I followed your guidance and finally managed to get s3fs up and running on Ubuntu 16.04 without bricking the server :)

Door robustadmin7 op za, 01/07/2017 - 12:01
Hi Conor, You can choose where to put the cache folder. I would put it in the /tmp folder, that seems the best option to make sure it's cleaned up on reboot. But if you want to choose yourself when to clean up, you could also put it in another folder.

Door SANJEEV RAO BASINENI op ma, 01/23/2017 - 14:55

Hi,
I am having permissions issue for new added files.
---------- 1 root root 2376351 Jan 23 21:53 3432434.pdf

Iam getting error if i change the permission with sudo chmod -R 777 or 775 <folder>, chmod: changing permissions of ‘insurance/’: Input/output error
Please help me.

Reactie toevoegen