• Redis: supports standalone, sentinel and cluster modes, which is suitable for high performance usage with up to 100 million files. Asynchronous copying files based on AOF would cause data loss in a tiny chance, while Amazon MemoryDB for Redis provides high reliability since it works synchronously.
  • Relational Database: includes MySQL, MariaDB and PostgreSQL, which is suitable for scenarios requiring high reliability more than high performance.
  • TiKV: is designed for massive (more than 100 million) files scenarios which require both high reliability and high performance, while better maintenance capabilities are necessary.
  • etcd: is designed for scenarios requiring high availability and high reliability with no more than 2 million files.
  • Embedded Database: includes BadgerDB and SQLite. Could be used without the need of multi-machine concurrent accessing.

Metadata engine support for etcd

# Create file system
$ juicefs format etcd://localhost:2379/myjfs jfs-etcd
# Mount file system
$ juicefs mount -d etcd://localhost:2379/myjfs /mnt/jfs

Redis Cluster and Amazon MemoryDB for Redis support

Delayed slice cleanup

Enhanced sync command

Upgrade of S3 gateway

Other new features

  1. Supports TLS when using TiKV metadata engine.
  2. When creating a file system, you can set a hash prefix to the data written to the object storage by specifying the option --hash-prefix. Since many object storages have prefix-based QPS limitations or performance bottlenecks, setting hash prefix is a workaround to bypass these limitations to obtain a better performance. Please notice that the existing file systems that have already had data written into cannot change this option.
  3. When mounting a file system, you can set the option --heartbeat to specify the client heartbeat interval. This is useful in some scenarios that care about failover time. Please notice that the default heartbeat interval has been changed from 60s to 12s.
  4. Supports Oracle Object Storage for storing data.
  5. Supports Java SDK metrics reporting to Graphite or other compatible systems.
  6. Supports non UTF-8 encoded file name for SQL metadata engines. For existing file systems, you need to upgrade your client first, and then modify the table schemas in your database.

Other changes

  1. When creating a new file system, JuiceFS will automatically write a placeholder object with UUID into the data storage, to keep it away from being used again by other file systems.
  2. The command juicefs dump will automatically hide the secret key of the object storage to prevent confidentials from leaking.
  3. Encrypt the saved credentials of the object storage to lower the security risk. You can change the encryption mode for existing file systems by using the command juicefs config META-URL --encrypt-secret. Please notice that old version clients will not be able to mount the file system when enable credential encryption.
  4. Improve the default metadata backup mechanism. You need to set the backup interval explicitly when you work with more than 1 million files.
  5. For Linux, when mounted as non-root users, the cache and log directories are set to the home directory of the current user to avoid insufficient permission failures.
  6. Improve the ability to import huge directory (contains more than 1 million files) into Redis and SQL databases.
  7. Add primary keys for all relational database tables to improve the log duplication performance. See here.

Upgrade tips

Ⅰ. Session format changes

Ⅱ. SQL table schema changes, support non UTF-8 encoded file names

Fixed bugs

  • Fix not releasing memory problem when failed to backup metadata.
  • Fix wrong return values of scan function when using SQL as the metadata engine.
  • Fix wrong values of some counters when load metadata by command juicefs load.
  • Fix wrong scanning result of object list when enabling multiple buckets for object storage.
  • Fix hang scanning problem when using Ceph RADOS as the object storage with too many objects.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
JuiceFS

JuiceFS

JuiceFS is a distributed POSIX file system built on top of Redis and S3.