Deploy and use JuiceFS to store data on Amazon AWS

Requirements

  1. a JuiceFS client installed on the server
  2. the object storage used to store data
  3. a database for storing metadata

1. Servers

2. Object Storage

  • Amazon S3 STANDARD: general-purpose storage for frequently accessed data
  • Amazon S3 STANDARD_IA: for data that is needed for a long time but accessed less frequently
  • S3 Glacier: for data that is archived over time

3. Database

4. Cautions

  • JuiceFS is not business invasive and will not affect the normal operation of existing systems.
  • When selecting cloud services, it is recommended to select all cloud services in the same region, which is equivalent to all services being on the same intranet, with the lowest latency and fastest inter-access. And, according to AWS billing rules, it is free to transfer data between basic cloud services in the same region. In other words, when you select cloud services in different regions, for example, EC2 is selected in ap-east-1, ElastiCache is selected in ap-southeast-1, and S3 is selected in us-east-2, the inter-access between each cloud service in this case will incur traffic charges.
  • JuiceFS does not require the use of object storage and databases from the same cloud platform; you can flexibly mix and match cloud services from different platforms as needed. For example, you can use EC2 to run JuiceFS client with AliCloud’s Redis database and Backbalze B2 object storage. Of course, JuiceFS storage composed of cloud services on the same platform and in the same region will perform even better.

Deployment and Usage

1. Install the client

$ JFS_LATEST_TAG=$(curl -s https://api.github.com/repos/juicedata/juicefs/releases/latest | grep 'tag_name' | cut -d '"' -f 4 | tr -d 'v')$ wget "https://github.com/juicedata/juicefs/releases/download/v${JFS_LATEST_TAG}/juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz"
$ mkdir juice && tar -zxvf "juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" -C juice
$ sudo install juice/juicefs /usr/local/bin
$ juicefs
NAME:
juicefs - A POSIX file system built on Redis and object storage.

USAGE:
juicefs [global options] command [command options] [arguments...]

VERSION:
0.17.0 (2021-09-24T04:17:26Z e115dc4)

COMMANDS:
format format a volume
mount mount a volume
umount unmount a volume
gateway S3-compatible gateway
sync sync between two storage
rmr remove directories recursively
info show internal information for paths or inodes
bench run benchmark to read/write/stat big/small files
gc collect any leaked objects
fsck Check consistency of file system
profile analyze access log
stats show runtime statistics
status show status of JuiceFS
warmup build cache for target directories/files
dump dump metadata into a JSON file
load load metadata from a previously dumped JSON file
help, h Shows a list of commands or help for one command

GLOBAL OPTIONS:
--verbose, --debug, -v enable debug log (default: false)
--quiet, -q only warning and errors (default: false)
--trace enable trace log (default: false)
--no-agent Disable pprof (:6060) and gops (:6070) agent (default: false)
--help, -h show help (default: false)
--version, -V print only the version (default: false)

COPYRIGHT:
AGPLv3

3. Create File System

$ juicefs format \
--storage s3 \
--bucket https://<bucket>.s3.<region>.amazonaws.com \
--access-key <access-key-id> \
--secret-key <access-key-secret> \
redis://[<redis-username>]:<redis-password>@<redis-url>:6379/1 \
mystor
  • --storage: Specify the type of object storage, here we use S3. For other object storage, please refer to the JuiceFS Supported Object Storage and Setup Guide.
  • --bucket: Bucket domain for object storage.
  • --access-key and --secret-key: The secret key pair to access the S3 API.
$ juicefs format \
--storage s3 \
--bucket https://herald-demo.s3.<region>.amazonaws.com \
redis://herald-demo.abcdefg.0001.apse1.cache.amazonaws.com:6379/1 \
mystor
2021/10/14 08:38:32.211044 juicefs[10391] <INFO>: Meta address: redis://herald-demo.abcdefg.0001.apse1.cache.amazonaws.com:6379/1
2021/10/14 08:38:32.216566 juicefs[10391] <INFO>: Ping redis: 383.789µs
2021/10/14 08:38:32.216915 juicefs[10391] <INFO>: Data use s3://herald-demo/mystor/
2021/10/14 08:38:32.412112 juicefs[10391] <INFO>: Volume is formatted as {Name:mystor UUID:21a2cafd-f5d8-4a76-ae4d-482c8e2d408d Storage:s3 Bucket:https://herald-demo.s3.ap-southeast-1.amazonaws.com AccessKey: SecretKey: BlockSize:4096 Compression:none Shards:0 Partitions:0 Capacity:0 Inodes:0 EncryptKey:}

4. Mount the file system

$ sudo juicefs mount -d redis://[<redis-username>]:<redis-password>@<redis-url>:6379/1  /mnt/jfs
$ sudo juicefs mount --cache-size 20480 -d redis://herald-demo.abcdefg.0001.apse1.cache.amazonaws.com:6379/1  /mnt/jfs
2021/10/14 08:47:49.623814 juicefs[10601] <INFO>: Meta address: redis://herald-demo.abcdefg.0001.apse1.cache.amazonaws.com:6379/1
2021/10/14 08:47:49.628157 juicefs[10601] <INFO>: Ping redis: 426.127µs
2021/10/14 08:47:49.628941 juicefs[10601] <INFO>: Data use s3://herald-demo/mystor/
2021/10/14 08:47:49.629198 juicefs[10601] <INFO>: Disk cache (/var/jfsCache/21a2cafd-f5d8-4a76-ae4d-482c8e2d408d/): capacity (20480 MB), free ratio (10%), max pending pages (15)
2021/10/14 08:47:50.132003 juicefs[10601] <INFO>: OK, mystor is ready at /mnt/jfs
$ df -Th
File system type capacity used usable used% mount point
JuiceFS:mystor fuse.juicefs 1.0P 64K 1.0P 1% /mnt/jfs

5. Uninstall JuiceFS Storage

$ sudo juicefs umount /mnt/jfs

6. Auto-mount on boot

$ sudo cp juice/juicefs /sbin/mount.juicefs
redis://[<redis-username>]:<redis-password>@<redis-url>:6379/1    /mnt/jfs       juicefs     _netdev,cache-size=20480     0  0

Summary

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
JuiceFS

JuiceFS

JuiceFS is a distributed POSIX file system built on top of Redis and S3.