GitHub this note shows basic concepts and application of SageMaker Experiment.

  • Purpose of SageMaker Experiment
  • Using SageMaker Experiment
sagemaker experiment

Experiment Purpose#

Developing ML models is an interative process, we need to track input, output, metrics, log, etc. So SageMaker experiment's purpose is to 1) log, and 2) analyse those objects.

  • What SageMaker Experiment can do?

Log a single parameter

run.log_parameter("param1", "value1")

Log multiple parameters

"param2": "value2",
"param3": "value3"

Log metric

run.log_metric(name="test:loss", value=loss, step=epoch)

Log files to S3

run.log_file("training_data.csv", name="training_data", is_output=False)

Log artifacts

run.log_artifact(name="training_data", value="data.csv" is_output=False)

Experiment SDK#

sagemaker experiment result

Let look at a example below, Run is an object which profile the run method to log things mentioned above. We aslo can pass the run method in to a callback to log metrics such las loss_train, loss_test.

with Run(experiment_name=experiment_name, sagemaker_session=sagemaker_session) as run:
run.log_parameter("batch_size", batch_size)
run.log_parameter("epochs", epochs)
run.log_parameter("dropout", dropout)
run.log_file("datasets/input_train.npy", is_output=False)
run.log_file("datasets/input_test.npy", is_output=False)
run.log_file("datasets/input_train_labels.npy", is_output=False)
run.log_file("datasets/input_test_labels.npy", is_output=False)
callbacks=[ExperimentCallback(run, model, x_test, y_test)]
score = model.evaluate(x_test, y_test, verbose=0)
print("test loss: ", score[0])
print("test accuracy: ", score[1])
run.log_metric(name="Final test loss: ", value=score[0])
run.log_metric(name="Final test accuracy: ", value=score[1])

The callback

class ExperimentCallback(keras.callbacks.Callback):
""" """
def __init__(self, run, model, x_test, y_test):
"""Save params in constructor""" = run
self.model = model
self.x_test = x_test
self.y_test = y_test
def on_epoch_end(self, epoch, logs=None):
""" """
keys = list(logs.keys())
for key in keys:, value=logs[key], step=epoch)
print("{} -> {}".format(key, logs[key]))

There are some examples: