org.apache.hadoop.fs

Class FileSystem

• org.apache.hadoop.fs.FileSystem

• All Implemented Interfaces:

  ,  , 

  Direct Known Subclasses:

  ,  ,  ,  ,  ,  , 

  ---

  @InterfaceAudience.Public@InterfaceStability.Stablepublic abstract class extends implements

  An abstract base class for a fairly generic filesystem. It may be implemented as a distributed filesystem, or as a "local" one that reflects the locally-connected disk. The local version exists for small Hadoop instances and for testing.

  All user code that may potentially use the Hadoop Distributed File System should be written to use a FileSystem object. The Hadoop DFS is a multi-machine system that appears as a single disk. It's useful because of its fault tolerance and potentially very large capacity.

  The local implementation is  and distributed implementation is DistributedFileSystem.

• • Field Summary

    Fields

     

    Modifier and Type	Field and Description
    static
    static
    static org.apache.commons.logging.Log
    static int	Priority of the FileSystem shutdown hook.
    protected org.apache.hadoop.fs.FileSystem.Statistics	The statistics for this file system.
  • Constructor Summary

    Constructors

     

    Modifier	Constructor and Description
    protected	()
  • Method Summary

    Methods

     

    Modifier and Type	Method and Description
    	( f) Append to an existing file (optional operation).
    	( f, int bufferSize) Append to an existing file (optional operation).
    abstract	( f, int bufferSize,  progress) Append to an existing file (optional operation).
    boolean	( f) Cancel the deletion of the path when the FileSystem is closed
    protected void	( path) Check that a Path belongs to this FileSystem.
    static void	() Reset all statistics for all file systems
    void	() No more filesystem operations are needed.
    static void	() Close all cached filesystems.
    static void	(org.apache.hadoop.security.UserGroupInformation ugi) Close all cached filesystems for a given UGI.
    void	( fsOutputFile,  tmpLocalFile) Called when we're all done writing to the target.
    void	( trg, [] psrcs) Concat existing files together.
    void	(boolean delSrc, boolean overwrite, [] srcs,  dst) The src files are on the local disk.
    void	(boolean delSrc, boolean overwrite,  src,  dst) The src file is on the local disk.
    void	(boolean delSrc,  src,  dst) The src file is on the local disk.
    void	( src,  dst) The src file is on the local disk.
    void	(boolean delSrc,  src,  dst) The src file is under FS, and the dst is on the local disk.
    void	(boolean delSrc,  src,  dst, boolean useRawLocalFileSystem) The src file is under FS, and the dst is on the local disk.
    void	( src,  dst) The src file is under FS, and the dst is on the local disk.
    static	( fs,  file,  permission) create a file with the provided permission The permission of the file is set to be the provided permission as in setPermission, not permission&~umask It is implemented using two RPCs.
    	( f) Create an FSDataOutputStream at the indicated Path.
    	( f, boolean overwrite) Create an FSDataOutputStream at the indicated Path.
    	( f, boolean overwrite, int bufferSize) Create an FSDataOutputStream at the indicated Path.
    	( f, boolean overwrite, int bufferSize,  progress) Create an FSDataOutputStream at the indicated Path with write-progress reporting.
    	( f, boolean overwrite, int bufferSize, short replication, long blockSize) Create an FSDataOutputStream at the indicated Path.
    	( f, boolean overwrite, int bufferSize, short replication, long blockSize,  progress) Create an FSDataOutputStream at the indicated Path with write-progress reporting.
    abstract	( f,  permission, boolean overwrite, int bufferSize, short replication, long blockSize,  progress) Create an FSDataOutputStream at the indicated Path with write-progress reporting.
    	( f,  permission, <> flags, int bufferSize, short replication, long blockSize,  progress) Create an FSDataOutputStream at the indicated Path with write-progress reporting.
    	( f,  permission, <> flags, int bufferSize, short replication, long blockSize,  progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt) Create an FSDataOutputStream at the indicated Path with a custom checksum option
    	( f,  progress) Create an FSDataOutputStream at the indicated Path with write-progress reporting.
    	( f, short replication) Create an FSDataOutputStream at the indicated Path.
    	( f, short replication,  progress) Create an FSDataOutputStream at the indicated Path with write-progress reporting.
    boolean	( f) Creates the given Path as a brand-new zero-length file.
    	( f, boolean overwrite, int bufferSize, short replication, long blockSize,  progress) Deprecated.  API only for 0.20-append
    	( f,  permission, boolean overwrite, int bufferSize, short replication, long blockSize,  progress) Deprecated.  API only for 0.20-append
    	( f,  permission, <> flags, int bufferSize, short replication, long blockSize, progress) Deprecated.  API only for 0.20-append
    boolean	( f) Deprecated.  Use  instead.
    abstract boolean	( f, boolean recursive) Delete a file.
    boolean	( f) Mark a path to be deleted when FileSystem is closed.
    boolean	( f) Check if exists.
    static	( conf) Returns the configured filesystem implementation.
    static	( uri,  conf) Returns the FileSystem for this URI's scheme and authority.
    static	( uri,  conf,  user) Get a filesystem instance based on the uri, the passed configuration and the user
    static <org.apache.hadoop.fs.FileSystem.Statistics>	() Return the FileSystem classes that have Statistics
    long	( f) Deprecated.  Use getFileStatus() instead
    protected	() Resolve the uri's hostname and add the default port if not in the uri
    	( f) Return the   of a given  .
    long	() Deprecated.  use  instead
    long	( f) Return the number of bytes that large input files should be optimally be split into to minimize i/o time.
    protected int	() Get the default port for this file system.
    short	() Deprecated.  use  instead
    short	( path) Get the default replication for a path.
    static	( conf) Get the default filesystem URI from a configuration.
    []	( file, long start, long len) Return an array containing hostnames, offset and size of portions of the given file.
    []	( p, long start, long len) Return an array containing hostnames, offset and size of portions of the given file.
    	( f) Get the checksum of a file.
    abstract	( f) Return a file status object that represents the path.
    static <? extends >	( scheme,  conf)
    	() Return the current user's home directory in this filesystem.
    protected	() Note: with the new FilesContext class, getWorkingDirectory() will be removed.
    long	( f) Deprecated.  Use getFileStatus() instead
    static	( conf) Get the local file system.
    	() Deprecated.  call #getUri() instead.
    static	( name,  conf) Deprecated.  call #get(URI,Configuration) instead.
    short	( src) Deprecated.  Use getFileStatus() instead
    	() Return the protocol scheme for the FileSystem.
    	() Deprecated.  use  instead
    	( p) Return a set of server default configuration values
    static <,org.apache.hadoop.fs.FileSystem.Statistics>	() Deprecated.  use  instead
    static org.apache.hadoop.fs.FileSystem.Statistics	( scheme, <? extends > cls) Get the statistics for a particular file system
    	() Returns a status object describing the use and capacity of the file system.
    	( p) Returns a status object describing the use and capacity of the file system.
    abstract	() Returns a URI whose scheme and authority identify this FileSystem.
    long	() Return the total size of all files in the filesystem.
    abstract	() Get the current working directory for the given file system
    []	( pathPattern) Return all the files that match filePattern and are not checksum files.
    []	( pathPattern,  filter) Return an array of FileStatus objects whose path names match pathPattern and is accepted by the user-supplied path filter.
    void	( name,  conf) Called after a new FileSystem instance is constructed.
    boolean	( f) True iff the named path is a directory.
    boolean	( f) True iff the named path is a regular file.
    org.apache.hadoop.fs.RemoteIterator<>	( path)
    org.apache.hadoop.fs.RemoteIterator<>	( f, boolean recursive) List the statuses and block locations of the files in the given path.
    org.apache.hadoop.fs.RemoteIterator<>	( f) List the statuses of the files/directories in the given path if the path is a directory.
    protected org.apache.hadoop.fs.RemoteIterator<>	( f,  filter) Listing a directory The returned results include its block location if it is a file The results are filtered by the given path filter
    abstract []	( f) List the statuses of the files/directories in the given path if the path is a directory.
    []	([] files) Filter files/directories in the given list of paths using default path filter.
    []	([] files,  filter) Filter files/directories in the given list of paths using user-supplied path filter.
    []	( f,  filter) Filter files/directories in the given path using the user-supplied path filter.
    	( path) Make sure that a path specifies a FileSystem.
    static boolean	( fs,  dir,  permission) create a directory with the provided permission The permission of the directory is set to be the provided permission as in setPermission, not permission&~umask
    boolean	( f) Call   with default permission.
    abstract boolean	( f,  permission) Make the given file and all non-existent parents into directories.
    void	([] srcs,  dst) The src files is on the local disk.
    void	( src,  dst) The src file is on the local disk.
    void	( src,  dst) The src file is under FS, and the dst is on the local disk.
    static	( conf) Returns a unique configured filesystem implementation.
    static	( uri,  conf) Returns the FileSystem for this URI's scheme and authority.
    static	( uri,  conf,  user) Returns the FileSystem for this URI's scheme and authority and the passed user.
    static	( conf) Get a unique local file system object
    	( f) Opens an FSDataInputStream at the indicated Path.
    abstract	( f, int bufferSize) Opens an FSDataInputStream at the indicated Path.
    protected	( f,  absolutePermission, <> flag, int bufferSize, short replication, long blockSize, progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt) Deprecated.
    protected boolean	( f,  absolutePermission) Deprecated.
    protected void	( f,  absolutePermission, boolean createParent) Deprecated.
    static void	() Print all statistics for all file systems
    protected void	() Delete all files that were marked as delete-on-exit.
    abstract boolean	( src,  dst) Renames Path src to Path dst.
    protected void	( src,  dst, org.apache.hadoop.fs.Options.Rename... options) Deprecated.
    	( p) Return the fully-qualified path of path f resolving the path through any symlinks or mount point
    static void	( conf,  uri) Set the default filesystem URI in a configuration.
    static void	( conf,  uri) Set the default filesystem URI in a configuration.
    void	( p,  username,  groupname) Set owner of a path (i.e.
    void	( p,  permission) Set permission of a path.
    boolean	( src, short replication) Set replication for an existing file.
    void	( p, long mtime, long atime) Set access time of a file
    void	(boolean verifyChecksum) Set the verify checksum flag.
    abstract void	( new_dir) Set the current working directory for the given file system.
    void	(boolean writeChecksum) Set the write checksum flag.
    	( fsOutputFile,  tmpLocalFile) Returns a local File that the user can write output to.

    • Methods inherited from class org.apache.hadoop.conf.

      , 
    • Methods inherited from class java.lang.

      , , , , , , , , , , 

• • Field Detail

    • FS_DEFAULT_NAME_KEY

      public static final 

      See Also:
    • DEFAULT_FS

      public static final 

      See Also:
    • LOG

      public static final org.apache.commons.logging.Log
    • SHUTDOWN_HOOK_PRIORITY

      public static final int

      Priority of the FileSystem shutdown hook.

      See Also:
    • statistics

      protected org.apache.hadoop.fs.FileSystem.Statistics

      The statistics for this file system.
  • Constructor Detail

    • FileSystem

      protected ()
  • Method Detail

    • get

      public static  ( uri,              conf,              user)                      throws ,

      Get a filesystem instance based on the uri, the passed configuration and the user

      Parameters:

      uri - of the filesystem

      conf - the configuration to use

      user - to perform the get as

      Returns:

      the filesystem instance

      Throws:
    • get

      public static  ( conf)                      throws

      Returns the configured filesystem implementation.

      Parameters:

      conf - the configuration to use

      Throws:
    • getDefaultUri

      public static  ( conf)

      Get the default filesystem URI from a configuration.

      Parameters:

      conf - the configuration to use

      Returns:

      the uri of the default filesystem
    • setDefaultUri

      public static void ( conf,                  uri)

      Set the default filesystem URI in a configuration.

      Parameters:

      conf - the configuration to alter

      uri - the new default filesystem uri
    • setDefaultUri

      public static void ( conf,                  uri)

      Set the default filesystem URI in a configuration.

      Parameters:

      conf - the configuration to alter

      uri - the new default filesystem uri
    • initialize

      public void ( name,               conf)                throws

      Called after a new FileSystem instance is constructed.

      Parameters:

      name - a uri whose authority section names the host, port, etc. for this FileSystem

      conf - the configuration

      Throws:
    • getScheme

      public  ()

      Return the protocol scheme for the FileSystem.

      This implementation throws an UnsupportedOperationException.

      Returns:

      the protocol scheme for the FileSystem.
    • getUri

      public abstract  ()

      Returns a URI whose scheme and authority identify this FileSystem.
    • getCanonicalUri

      protected  ()

      Resolve the uri's hostname and add the default port if not in the uri

      Returns:

      URI

      See Also:

      NetUtils.getCanonicalUri(URI, int)
    • getDefaultPort

      protected int ()

      Get the default port for this file system.

      Returns:

      the default port or 0 if there isn't one
    • getName

      public  ()

      Deprecated. 

      call #getUri() instead.
    • getNamed

      public static  ( name,                              conf)                           throws

      Deprecated. 

      call #get(URI,Configuration) instead.

      Throws:
    • getLocal

      public static  ( conf)                                throws

      Get the local file system.

      Parameters:

      conf - the configuration to configure the file system with

      Returns:

      a LocalFileSystem

      Throws:
    • get

      public static  ( uri,              conf)                      throws

      Returns the FileSystem for this URI's scheme and authority. The scheme of the URI determines a configuration property name, 

      fs.scheme.class whose value names the FileSystem class. The entire URI is passed to the FileSystem instance's initialize method.

      Throws:
    • newInstance

      public static  ( uri,                      conf,                      user)                              throws ,

      Returns the FileSystem for this URI's scheme and authority and the passed user. Internally invokes 

      Parameters:

      uri - of the filesystem

      conf - the configuration to use

      user - to perform the get as

      Returns:

      filesystem instance

      Throws:
    • newInstance

      public static  ( uri,                      conf)                              throws

      Returns the FileSystem for this URI's scheme and authority. The scheme of the URI determines a configuration property name, 

      fs.scheme.class whose value names the FileSystem class. The entire URI is passed to the FileSystem instance's initialize method. This always returns a new FileSystem object.

      Throws:
    • newInstance

      public static  ( conf)                              throws

      Returns a unique configured filesystem implementation. This always returns a new FileSystem object.

      Parameters:

      conf - the configuration to use

      Throws:
    • newInstanceLocal

      public static  ( conf)                                        throws

      Get a unique local file system object

      Parameters:

      conf - the configuration to configure the file system with

      Returns:

      a LocalFileSystem This always returns a new FileSystem object.

      Throws:
    • closeAll

      public static void ()                     throws

      Close all cached filesystems. Be sure those filesystems are not used anymore.

      Throws:
    • closeAllForUGI

      public static void (org.apache.hadoop.security.UserGroupInformation ugi)                           throws

      Close all cached filesystems for a given UGI. Be sure those filesystems are not used anymore.

      Parameters:

      ugi - user group info to close

      Throws:
    • makeQualified

      public  ( path)

      Make sure that a path specifies a FileSystem.

      Parameters:

      path - to use
    • create

      public static  ( fs,                         file,                         permission)                                 throws

      create a file with the provided permission The permission of the file is set to be the provided permission as in setPermission, not permission&~umask It is implemented using two RPCs. It is understood that it is inefficient, but the implementation is thread-safe. The other option is to change the value of umask in configuration to be 0, but it is not thread-safe.

      Parameters:

      fs - file system handle

      file - the name of the file to be created

      permission - the permission of the file

      Returns:

      an output stream

      Throws:
    • mkdirs

      public static boolean ( fs,              dir,              permission)                      throws

      create a directory with the provided permission The permission of the directory is set to be the provided permission as in setPermission, not permission&~umask

      Parameters:

      fs - file system handle

      dir - the name of the directory to be created

      permission - the permission of the directory

      Returns:

      true if the directory creation succeeds; false otherwise

      Throws:

      See Also:
    • checkPath

      protected void ( path)

      Check that a Path belongs to this FileSystem.

      Parameters:

      path - to check
    • getFileBlockLocations

      public [] ( file,                                    long start,                                    long len)                                      throws

      Return an array containing hostnames, offset and size of portions of the given file. For a nonexistent file or regions, null will be returned. This call is most helpful with DFS, where it returns hostnames of machines that contain the given file. The FileSystem will simply return an elt containing 'localhost'.

      Parameters:

      file - FilesStatus to get data from

      start - offset into the given file

      len - length for which to get locations for

      Throws:
    • getFileBlockLocations

      public [] ( p,                                    long start,                                    long len)                                      throws

      Return an array containing hostnames, offset and size of portions of the given file. For a nonexistent file or regions, null will be returned. This call is most helpful with DFS, where it returns hostnames of machines that contain the given file. The FileSystem will simply return an elt containing 'localhost'.

      Parameters:

      p - path is used to identify an FS since an FS could have another FS that it could be delegating the call to

      start - offset into the given file

      len - length for which to get locations for

      Throws:
    • getServerDefaults

      public  ()                                   throws

      Deprecated. 

      use  instead
      Return a set of server default configuration values

      Returns:

      server default configuration values

      Throws:
    • getServerDefaults

      public  ( p)                                   throws

      Return a set of server default configuration values

      Parameters:

      p - path is used to identify an FS since an FS could have another FS that it could be delegating the call to

      Returns:

      server default configuration values

      Throws:
    • resolvePath

      public  ( p)                 throws

      Return the fully-qualified path of path f resolving the path through any symlinks or mount point

      Parameters:

      p - path to be resolved

      Returns:

      fully qualified path

      Throws:
    • open

      public abstract  ( f,                     int bufferSize)                                throws

      Opens an FSDataInputStream at the indicated Path.

      Parameters:

      f - the file name to open

      bufferSize - the size of the buffer to be used.

      Throws:
    • open

      public  ( f)                       throws

      Opens an FSDataInputStream at the indicated Path.

      Parameters:

      f - the file to open

      Throws:
    • create

      public  ( f)                          throws

      Create an FSDataOutputStream at the indicated Path. Files are overwritten by default.

      Parameters:

      f - the file to create

      Throws:
    • create

      public  ( f,                        boolean overwrite)                          throws

      Create an FSDataOutputStream at the indicated Path.

      Parameters:

      f - the file to create

      overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an exception will be thrown.

      Throws:
    • create

      public  ( f,                         progress)                          throws

      Create an FSDataOutputStream at the indicated Path with write-progress reporting. Files are overwritten by default.

      Parameters:

      f - the file to create

      progress - to report progress

      Throws:
    • create

      public  ( f,                        short replication)                          throws

      Create an FSDataOutputStream at the indicated Path. Files are overwritten by default.

      Parameters:

      f - the file to create

      replication - the replication factor

      Throws:
    • create

      public  ( f,                        short replication,                         progress)                          throws

      Create an FSDataOutputStream at the indicated Path with write-progress reporting. Files are overwritten by default.

      Parameters:

      f - the file to create

      replication - the replication factor

      progress - to report progress

      Throws:
    • create

      public  ( f,                        boolean overwrite,                        int bufferSize)                          throws

      Create an FSDataOutputStream at the indicated Path.

      Parameters:

      f - the file name to create

      overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

      bufferSize - the size of the buffer to be used.

      Throws:
    • create

      public  ( f,                        boolean overwrite,                        int bufferSize,                         progress)                          throws

      Create an FSDataOutputStream at the indicated Path with write-progress reporting.

      Parameters:

      f - the path of the file to open

      overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

      bufferSize - the size of the buffer to be used.

      Throws:
    • create

      public  ( f,                        boolean overwrite,                        int bufferSize,                        short replication,                        long blockSize)                          throws

      Create an FSDataOutputStream at the indicated Path.

      Parameters:

      f - the file name to open

      overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

      bufferSize - the size of the buffer to be used.

      replication - required block replication for the file.

      Throws:
    • create

      public  ( f,                        boolean overwrite,                        int bufferSize,                        short replication,                        long blockSize,                         progress)                          throws

      Create an FSDataOutputStream at the indicated Path with write-progress reporting.

      Parameters:

      f - the file name to open

      overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

      bufferSize - the size of the buffer to be used.

      replication - required block replication for the file.

      Throws:
    • create

      public abstract  ( f,                         permission,                        boolean overwrite,                        int bufferSize,                        short replication,                        long blockSize,                         progress)                                   throws

      Create an FSDataOutputStream at the indicated Path with write-progress reporting.

      Parameters:

      f - the file name to open

      permission -

      overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

      bufferSize - the size of the buffer to be used.

      replication - required block replication for the file.

      blockSize -

      progress -

      Throws:

      See Also:
    • create

      public  ( f,                         permission,                        <> flags,                        int bufferSize,                        short replication,                        long blockSize,                         progress)                          throws

      Create an FSDataOutputStream at the indicated Path with write-progress reporting.

      Parameters:

      f - the file name to open

      permission -

      flags -  s to use for this stream.

      bufferSize - the size of the buffer to be used.

      replication - required block replication for the file.

      blockSize -

      progress -

      Throws:

      See Also:
    • create

      public  ( f,                         permission,                        <> flags,                        int bufferSize,                        short replication,                        long blockSize,                         progress,                        org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt)                          throws

      Create an FSDataOutputStream at the indicated Path with a custom checksum option

      Parameters:

      f - the file name to open

      permission -

      flags -  s to use for this stream.

      bufferSize - the size of the buffer to be used.

      replication - required block replication for the file.

      blockSize -

      progress -

      checksumOpt - checksum parameter. If null, the values found in conf will be used.

      Throws:

      See Also:
    • primitiveCreate

      protected  ( f,                                             absolutePermission,                                            <> flag,                                            int bufferSize,                                            short replication,                                            long blockSize,                                             progress,                                            org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt)                                      throws

      Deprecated. 

      Throws:
    • primitiveMkdir

      protected boolean ( f,                                 absolutePermission)                          throws

      Deprecated. 
      This version of the mkdirs method assumes that the permission is absolute. It has been added to support the FileContext that processes the permission with umask before calling this method. This a temporary method added to support the transition from FileSystem to FileContext for user applications.

      Throws:
    • primitiveMkdir

      protected void ( f,                              absolutePermission,                             boolean createParent)                       throws

      Deprecated. 
      This version of the mkdirs method assumes that the permission is absolute. It has been added to support the FileContext that processes the permission with umask before calling this method. This a temporary method added to support the transition from FileSystem to FileContext for user applications.

      Throws:
    • createNonRecursive

      public  ( f,                                               boolean overwrite,                                               int bufferSize,                                               short replication,                                               long blockSize,                                                progress)                                      throws

      Deprecated. 

      API only for 0.20-append
      Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.

      Parameters:

      f - the file name to open

      overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

      bufferSize - the size of the buffer to be used.

      replication - required block replication for the file.

      blockSize -

      progress -

      Throws:

      See Also:
    • createNonRecursive

      public  ( f,                                                permission,                                               boolean overwrite,                                               int bufferSize,                                               short replication,                                               long blockSize,                                                progress)                                      throws

      Deprecated. 

      API only for 0.20-append
      Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.

      Parameters:

      f - the file name to open

      permission -

      overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

      bufferSize - the size of the buffer to be used.

      replication - required block replication for the file.

      blockSize -

      progress -

      Throws:

      See Also:
    • createNonRecursive

      public  ( f,                                                permission,                                               <> flags,                                               int bufferSize,                                               short replication,                                               long blockSize,                                                progress)                                      throws

      Deprecated. 

      API only for 0.20-append
      Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.

      Parameters:

      f - the file name to open

      permission -

      flags -  s to use for this stream.

      bufferSize - the size of the buffer to be used.

      replication - required block replication for the file.

      blockSize -

      progress -

      Throws:

      See Also:
    • createNewFile

      public boolean ( f)                      throws

      Creates the given Path as a brand-new zero-length file. If create fails, or if it already existed, return false.

      Parameters:

      f - path to use for create

      Throws:
    • append

      public  ( f)                          throws

      Append to an existing file (optional operation). Same as append(f, getConf().getInt("io.file.buffer.size", 4096), null)

      Parameters:

      f - the existing file to be appended.

      Throws:
    • append

      public  ( f,                        int bufferSize)                          throws

      Append to an existing file (optional operation). Same as append(f, bufferSize, null).

      Parameters:

      f - the existing file to be appended.

      bufferSize - the size of the buffer to be used.

      Throws:
    • append

      public abstract  ( f,                        int bufferSize,                         progress)                                   throws

      Append to an existing file (optional operation).

      Parameters:

      f - the existing file to be appended.

      bufferSize - the size of the buffer to be used.

      progress - for reporting progress if it is not null.

      Throws:
    • concat

      public void ( trg,          [] psrcs)            throws

      Concat existing files together.

      Parameters:

      trg - the path to the target destination.

      psrcs - the paths to the sources to use for the concatenation.

      Throws:
    • getReplication

      public short ( src)                     throws

      Deprecated. 

      Use getFileStatus() instead
      Get replication.

      Parameters:

      src - file name

      Returns:

      file replication

      Throws:
    • setReplication

      public boolean ( src,                     short replication)                       throws

      Set replication for an existing file.

      Parameters:

      src - file name

      replication - new replication

      Returns:

      true if successful; false if file does not exist or is a directory

      Throws:
    • rename

      public abstract boolean ( src,              dst)                        throws

      Renames Path src to Path dst. Can take place on local fs or remote DFS.

      Parameters:

      src - path to be renamed

      dst - new path after rename

      Returns:

      true if rename is successful

      Throws:

       - on failure
    • rename

      protected void ( src,                      dst,                     org.apache.hadoop.fs.Options.Rename... options)               throws

      Deprecated. 
      Renames Path src to Path dst

      • Fails if src is a file and dst is a directory.
      • Fails if src is a directory and dst is a file.
      • Fails if the parent of dst does not exist or is a file.

      If OVERWRITE option is not passed as an argument, rename fails if the dst already exists.

      If OVERWRITE option is passed as an argument, rename overwrites the dst if it is a file or an empty directory. Rename fails if dst is a non-empty directory.

      Note that atomicity of rename is dependent on the file system implementation. Please refer to the file system documentation for details. This default implementation is non atomic.

      This method is deprecated since it is a temporary method added to support the transition from FileSystem to FileContext for user applications.

      Parameters:

      src - path to be renamed

      dst - new path after rename

      Throws:

       - on failure
    • delete

      public boolean ( f)               throws

      Deprecated. 

      Use  instead.
      Delete a file

      Throws:
    • delete

      public abstract boolean ( f,             boolean recursive)                        throws

      Delete a file.

      Parameters:

      f - the path to delete.

      recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.

      Returns:

      true if delete is successful else false.

      Throws:
    • deleteOnExit

      public boolean ( f)                     throws

      Mark a path to be deleted when FileSystem is closed. When the JVM shuts down, all FileSystem objects will be closed automatically. Then, the marked path will be deleted as a result of closing the FileSystem. The path has to exist in the file system.

      Parameters:

      f - the path to delete.

      Returns:

      true if deleteOnExit is successful, otherwise false.

      Throws:
    • cancelDeleteOnExit

      public boolean ( f)

      Cancel the deletion of the path when the FileSystem is closed

      Parameters:

      f - the path to cancel deletion
    • processDeleteOnExit

      protected void ()

      Delete all files that were marked as delete-on-exit. This recursively deletes all files in the specified paths.
    • exists

      public boolean ( f)               throws

      Check if exists.

      Parameters:

      f - source file

      Throws:
    • isDirectory

      public boolean ( f)                    throws

      True iff the named path is a directory. Note: Avoid using this method. Instead reuse the FileStatus returned by getFileStatus() or listStatus() methods.

      Parameters:

      f - path to check

      Throws:
    • isFile

      public boolean ( f)               throws

      True iff the named path is a regular file. Note: Avoid using this method. Instead reuse the FileStatus returned by getFileStatus() or listStatus() methods.

      Parameters:

      f - path to check

      Throws:
    • getLength

      public long ( f)               throws

      Deprecated. 

      Use getFileStatus() instead

      Throws:
    • getContentSummary

      public  ( f)                                 throws

      Return the   of a given  .

      Parameters:

      f - path to use

      Throws:
    • listStatus

      public abstract [] ( f)                                 throws ,

      List the statuses of the files/directories in the given path if the path is a directory.

      Parameters:

      f - given path

      Returns:

      the statuses of the files/directories in the given patch

      Throws:

       - when the path does not exist; IOException see specific implementation
    • listCorruptFileBlocks

      public org.apache.hadoop.fs.RemoteIterator<> ( path)                                                                throws

      Returns:

      an iterator over the corrupt files under the given path (may contain duplicates if a file has more than one corrupt block)

      Throws:
    • listStatus

      public [] ( f,                       filter)                        throws ,

      Filter files/directories in the given path using the user-supplied path filter.

      Parameters:

      f - a path name

      filter - the user-supplied path filter

      Returns:

      an array of FileStatus objects for the files under the given path after applying the filter

      Throws:

       - when the path does not exist; IOException see specific implementation
    • listStatus

      public [] ([] files)                        throws ,

      Filter files/directories in the given list of paths using default path filter.

      Parameters:

      files - a list of paths

      Returns:

      a list of statuses for the files under the given paths after applying the filter default Path filter

      Throws:

       - when the path does not exist; IOException see specific implementation
    • listStatus

      public [] ([] files,                       filter)                        throws ,

      Filter files/directories in the given list of paths using user-supplied path filter.

      Parameters:

      files - a list of paths

      filter - the user-supplied path filter

      Returns:

      a list of statuses for the files under the given paths after applying the filter

      Throws:

       - when the path does not exist; IOException see specific implementation
    • globStatus

      public [] ( pathPattern)                        throws

      Return all the files that match filePattern and are not checksum files. Results are sorted by their names.

      A filename pattern is composed of regular characters and special pattern matching characters, which are:

       

      ?

      Matches any single character.

       

      *

      Matches zero or more characters.

       

      [abc]

      Matches a single character from character set 

      {

      a,b,c}.

       

      [a-b]

      Matches a single character from the character range 

      {

      a...b}. Note that character 

      a must be lexicographically less than or equal to character 

      b.

       

      [^a]

      Matches a single character that is not from character set or range 

      {

      a}. Note that the 

      ^ character must occur immediately to the right of the opening bracket.

       

      \c

      Removes (escapes) any special meaning of character 

      c.

       

      {ab,cd}

      Matches a string from the string set 

      {

      ab, cd}

       

      {ab,c{de,fh}}

      Matches a string from the string set 

      {

      ab, cde, cfh}

      Parameters:

      pathPattern - a regular expression specifying a pth pattern

      Returns:

      an array of paths that match the path pattern

      Throws:
    • globStatus

      public [] ( pathPattern,                       filter)                        throws

      Return an array of FileStatus objects whose path names match pathPattern and is accepted by the user-supplied path filter. Results are sorted by their path names. Return null if pathPattern has no glob and the path does not exist. Return an empty array if pathPattern has a glob and no path matches it.

      Parameters:

      pathPattern - a regular expression specifying the path pattern

      filter - a user-supplied path filter

      Returns:

      an array of FileStatus objects

      Throws:

       - if any I/O error occurs when fetching file status
    • listLocatedStatus

      public org.apache.hadoop.fs.RemoteIterator<> ( f)                                                                         throws ,

      List the statuses of the files/directories in the given path if the path is a directory. Return the file's status and block locations If the path is a file. If a returned status is a file, it contains the file's block locations.

      Parameters:

      f - is the path

      Returns:

      an iterator that traverses statuses of the files/directories in the given path

      Throws:

       - If 

      f does not exist

       - If an I/O error occurred
    • listLocatedStatus

      protected org.apache.hadoop.fs.RemoteIterator<> ( f,                                                                        filter)                                                                            throws ,

      Listing a directory The returned results include its block location if it is a file The results are filtered by the given path filter

      Parameters:

      f - a path

      filter - a path filter

      Returns:

      an iterator that traverses statuses of the files/directories in the given path

      Throws:

       - if 

      f does not exist

       - if any I/O error occurred
    • listFiles

      public org.apache.hadoop.fs.RemoteIterator<> ( f,                                                               boolean recursive)                                                                 throws ,

      List the statuses and block locations of the files in the given path. If the path is a directory, if recursive is false, returns files in the directory; if recursive is true, return files in the subtree rooted at the path. If the path is a file, return the file's status and block locations.

      Parameters:

      f - is the path

      recursive - if the subdirectories need to be traversed recursively

      Returns:

      an iterator that traverses statuses of the files

      Throws:

       - when the path does not exist; IOException see specific implementation
    • getHomeDirectory

      public  ()

      Return the current user's home directory in this filesystem. The default implementation returns "/user/$USER/".
    • setWorkingDirectory

      public abstract void ( new_dir)

      Set the current working directory for the given file system. All relative paths will be resolved relative to it.

      Parameters:

      new_dir -
    • getWorkingDirectory

      public abstract  ()

      Get the current working directory for the given file system

      Returns:

      the directory pathname
    • getInitialWorkingDirectory

      protected  ()

      Note: with the new FilesContext class, getWorkingDirectory() will be removed. The working directory is implemented in FilesContext. Some file systems like LocalFileSystem have an initial workingDir that we use as the starting workingDir. For other file systems like HDFS there is no built in notion of an initial workingDir.

      Returns:

      if there is built in notion of workingDir then it is returned; else a null is returned.
    • mkdirs

      public boolean ( f)               throws

      Call   with default permission.

      Throws:
    • mkdirs

      public abstract boolean ( f,              permission)                        throws

      Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

      Parameters:

      f - path to create

      permission - to apply to f

      Throws:
    • copyFromLocalFile

      public void ( src,                      dst)                       throws

      The src file is on the local disk. Add it to FS at the given dst name and the source is kept intact afterwards

      Parameters:

      src - path

      dst - path

      Throws:
    • moveFromLocalFile

      public void ([] srcs,                      dst)                       throws

      The src files is on the local disk. Add it to FS at the given dst name, removing the source afterwards.

      Parameters:

      srcs - path

      dst - path

      Throws:
    • moveFromLocalFile

      public void ( src,                      dst)                       throws

      The src file is on the local disk. Add it to FS at the given dst name, removing the source afterwards.

      Parameters:

      src - path

      dst - path

      Throws:
    • copyFromLocalFile

      public void (boolean delSrc,                      src,                      dst)                       throws

      The src file is on the local disk. Add it to FS at the given dst name. delSrc indicates if the source should be removed

      Parameters:

      delSrc - whether to delete the src

      src - path

      dst - path

      Throws:
    • copyFromLocalFile

      public void (boolean delSrc,                     boolean overwrite,                     [] srcs,                      dst)                       throws

      The src files are on the local disk. Add it to FS at the given dst name. delSrc indicates if the source should be removed

      Parameters:

      delSrc - whether to delete the src

      overwrite - whether to overwrite an existing file

      srcs - array of paths which are source

      dst - path

      Throws:
    • copyFromLocalFile

      public void (boolean delSrc,                     boolean overwrite,                      src,                      dst)                       throws

      The src file is on the local disk. Add it to FS at the given dst name. delSrc indicates if the source should be removed

      Parameters:

      delSrc - whether to delete the src

      overwrite - whether to overwrite an existing file

      src - path

      dst - path

      Throws:
    • copyToLocalFile

      public void ( src,                    dst)                     throws

      The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name.

      Parameters:

      src - path

      dst - path

      Throws:
    • moveToLocalFile

      public void ( src,                    dst)                     throws

      The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name. Remove the source afterwards

      Parameters:

      src - path

      dst - path

      Throws:
    • copyToLocalFile

      public void (boolean delSrc,                    src,                    dst)                     throws

      The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name. delSrc indicates if the src will be removed or not.

      Parameters:

      delSrc - whether to delete the src

      src - path

      dst - path

      Throws:
    • copyToLocalFile

      public void (boolean delSrc,                    src,                    dst,                   boolean useRawLocalFileSystem)                     throws

      The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name. delSrc indicates if the src will be removed or not. useRawLocalFileSystem indicates whether to use RawLocalFileSystem as local file system or not. RawLocalFileSystem is non crc file system.So, It will not create any crc files at local.

      Parameters:

      delSrc - whether to delete the src

      src - path

      dst - path

      useRawLocalFileSystem - whether to use RawLocalFileSystem as local file system or not.

      Throws:

       - - if any IO error
    • startLocalOutput

      public  ( fsOutputFile,                     tmpLocalFile)                      throws

      Returns a local File that the user can write output to. The caller provides both the eventual FS target name and the local working file. If the FS is local, we write directly into the target. If the FS is remote, we write into the tmp local area.

      Parameters:

      fsOutputFile - path of output file

      tmpLocalFile - path of local tmp file

      Throws:
    • completeLocalOutput

      public void ( fsOutputFile,                        tmpLocalFile)                         throws

      Called when we're all done writing to the target. A local FS will do nothing, because we've written to exactly the right place. A remote FS will copy the contents of tmpLocalFile to the correct target at fsOutputFile.

      Parameters:

      fsOutputFile - path of output file

      tmpLocalFile - path to local tmp file

      Throws:
    • close

      public void ()           throws

      No more filesystem operations are needed. Will release any held locks.

      Specified by:

       in interface 

      Specified by:

       in interface 

      Throws:
    • getUsed

      public long ()             throws

      Return the total size of all files in the filesystem.

      Throws:
    • getBlockSize

      public long ( f)                  throws

      Deprecated. 

      Use getFileStatus() instead

      Throws:
    • getDefaultBlockSize

      public long ()

      Deprecated. 

      use  instead
      Return the number of bytes that large input files should be optimally be split into to minimize i/o time.
    • getDefaultBlockSize

      public long ( f)

      Return the number of bytes that large input files should be optimally be split into to minimize i/o time. The given path will be used to locate the actual filesystem. The full path does not have to exist.

      Parameters:

      f - path of file

      Returns:

      the default block size for the path's filesystem
    • getDefaultReplication

      public short ()

      Deprecated. 

      use  instead
      Get the default replication.
    • getDefaultReplication

      public short ( path)

      Get the default replication for a path. The given path will be used to locate the actual filesystem. The full path does not have to exist.

      Parameters:

      path - of the file

      Returns:

      default replication for the path's filesystem
    • getFileStatus

      public abstract  ( f)                                  throws

      Return a file status object that represents the path.

      Parameters:

      f - The path we want information from

      Returns:

      a FileStatus object

      Throws:

       - when the path does not exist; IOException see specific implementation
    • getFileChecksum

      public  ( f)                             throws

      Get the checksum of a file.

      Parameters:

      f - The file path

      Returns:

      The file checksum. The default return value is null, which indicates that no checksum algorithm is implemented in the corresponding FileSystem.

      Throws:
    • setVerifyChecksum

      public void (boolean verifyChecksum)

      Set the verify checksum flag. This is only applicable if the corresponding FileSystem supports checksum. By default doesn't do anything.

      Parameters:

      verifyChecksum -
    • setWriteChecksum

      public void (boolean writeChecksum)

      Set the write checksum flag. This is only applicable if the corresponding FileSystem supports checksum. By default doesn't do anything.

      Parameters:

      writeChecksum -
    • getStatus

      public  ()                   throws

      Returns a status object describing the use and capacity of the file system. If the file system has multiple partitions, the use and capacity of the root partition is reflected.

      Returns:

      a FsStatus object

      Throws:

       - see specific implementation
    • getStatus

      public  ( p)                   throws

      Returns a status object describing the use and capacity of the file system. If the file system has multiple partitions, the use and capacity of the partition pointed to by the specified path is reflected.

      Parameters:

      p - Path for which status should be obtained. null means the default partition.

      Returns:

      a FsStatus object

      Throws:

       - see specific implementation
    • setPermission

      public void ( p,                  permission)                   throws

      Set permission of a path.

      Parameters:

      p -

      permission -

      Throws:
    • setOwner

      public void ( p,             username,             groupname)              throws

      Set owner of a path (i.e. a file or a directory). The parameters username and groupname cannot both be null.

      Parameters:

      p - The path

      username - If it is null, the original username remains unchanged.

      groupname - If it is null, the original groupname remains unchanged.

      Throws:
    • setTimes

      public void ( p,            long mtime,            long atime)              throws

      Set access time of a file

      Parameters:

      p - The path

      mtime - Set the modification time of this file. The number of milliseconds since Jan 1, 1970. A value of -1 means that this call should not set modification time.

      atime - Set the access time of this file. The number of milliseconds since Jan 1, 1970. A value of -1 means that this call should not set access time.

      Throws:
    • getFileSystemClass

      public static 
                   ( scheme,                                              conf)                                                      throws

      Throws:
    • getStatistics

      public static <,org.apache.hadoop.fs.FileSystem.Statistics> ()

      Deprecated. 

      use  instead
      Get the Map of Statistics object indexed by URI Scheme.

      Returns:

      a Map having a key as URI scheme and value as Statistics object
    • getAllStatistics

      public static 
                  
                    ()
                  

      Return the FileSystem classes that have Statistics
    • getStatistics

      public static org.apache.hadoop.fs.FileSystem.Statistics ( scheme,                                                       
                   cls)

      Get the statistics for a particular file system

      Parameters:

      cls - the class to lookup

      Returns:

      a statistics object
    • clearStatistics

      public static void ()

      Reset all statistics for all file systems
    • printStatistics

      public static void ()                            throws

      Print all statistics for all file systems

      Throws: