Skip to main content

resources

resources()

Syncs arbitrary files between local and fabric environments. @resources decorator can be applied to the functions that are wrapped by the @dataset or @model decorators.

  • Parameters

    • path (str) -- File or directory path, could be relative or absolute. If path is a directory, all directories and files are uploaded recursively.
    • paths (str) -- Additional paths.

  • Returns

    Function object being decorated.

  • Return type

    Callable[[...], Any]

import pandas as pd
from layer
from layer.decorators import dataset, resources

@dataset("titanic")
@resources("data/titanic.csv")
def titanic():
    return pd.read_csv("data/titanic.csv")

When you execute layer.run([titanic]), the following happens:

  • Function titanic is submitted to Layer fabric, data/titanic.csv is uploaded to internal resource storage.
  • Before the execution, fabric runtime downloads the resource data/titanic.csv and makes it available to use within the function scope.

If you have multiple resources, separate them with commas.

@resources("path", "path1", "path2")

Note: while absolute or relative paths are allowed as parameters to @resources, they should always be relative when used within function body. For example:

@dataset("titanic")
@resources("/data/titanic.csv")  # it's ok to use absolute paths as they're resolve on local machine
def titanic():
    return pd.read_csv("data/titanic.csv")  # while run inside function fabric, always use the relative paths!

Limitations:

  • You can upload a maximum of 1000 files
  • Total upload size is limited to 5GB