Syncs arbitrary files between local and fabric environments.
@resources decorator can be applied to the functions that are wrapped by the
- path (str) -- File or directory path, could be relative or absolute. If path is a directory, all directories and files are uploaded recursively.
- paths (str) -- Additional paths.
Function object being decorated.
import pandas as pd
from layer.decorators import dataset, resources
When you execute
layer.run([titanic]), the following happens:
titanicis submitted to Layer fabric,
data/titanic.csvis uploaded to internal resource storage.
- Before the execution, fabric runtime downloads the resource
data/titanic.csvand makes it available to use within the function scope.
If you have multiple resources, separate them with commas.
@resources("path", "path1", "path2")
Note: while absolute or relative paths are allowed as parameters to
@resources, they should always be relative when used within function body. For example:
@resources("/data/titanic.csv") # it's ok to use absolute paths as they're resolve on local machine
return pd.read_csv("data/titanic.csv") # while run inside function fabric, always use the relative paths!
- You can upload a maximum of 1000 files
- Total upload size is limited to 5GB