Welcome to our guide on the Metaflow ServiceMetadata service! This service offers an efficient way to manage metadata associated with various Metaflow entities such as Flows, Runs, Steps, Tasks, and Artifacts. Today, we will walk you through the process of setting it up, running it with Docker, and troubleshooting common issues.

Getting Started

Before diving into the setup, ensure you have the following environment variables configured:

• MF_METADATA_DB_HOST (defaults to localhost)
• MF_METADATA_DB_PORT (defaults to 5432)
• MF_METADATA_DB_USER (defaults to postgres)
• MF_METADATA_DB_PSWD (defaults to postgres)
• MF_METADATA_DB_NAME (defaults to postgres)
• MF_METADATA_PORT (defaults to 8080, optional)
• MF_MIGRATION_PORT (defaults to 8082, optional)
• MF_METADATA_HOST (defaults to 0.0.0.0, optional)

Creating Database Triggers

To ensure the service communicates effectively, create triggers to broadcast any database changes:

DB_TRIGGER_CREATE - [metadata_service defaults to 0] - [ui_backend_service defaults to 1]

Running the Service Using Docker Compose

The simplest way to start the Metaflow ServiceMetadata is using Docker Compose. There are two options:

• docker-compose.yml – Assumes that Dockerfiles are pre-built.
• docker-compose.development.yml – Recommended for development, includes automatic Dockerfile builds.

Run the following commands to start the services:

sh docker-compose up -d

For the development version:

sh docker-compose -f docker-compose.development.yml up

Accessing the Services

Once you have the services running:

• Metadata service is available at port 8080.
• Migration service is available at port 8082.
• UI service is available at port 8083.

To access the container directly, use:

sh docker exec -it metadata_service binbash

Using the Published Image on DockerHub

If you prefer to use the latest image published on DockerHub, run the following command:

sh docker pull netflixoss/metaflow_metadata_service

Make sure to set the proper environment variables similar to the previous configuration before running the image:

sh docker run -e MF_METADATA_DB_HOST=instance_name.us-east-1.rds.amazonaws.com -e MF_METADATA_DB_PORT=5432 -e MF_METADATA_DB_USER=postgres -e MF_METADATA_DB_PSWD=postgres -e MF_METADATA_DB_NAME=metaflow -it -p 8082:8082 -p 8080:8080 metaflow_metadata_service

Running Tests

You can run tests using Tox and pytest. Use the following commands to execute tests:

sh docker-compose -f docker-compose.test.yml up -V --abort-on-container-exit

Executing Flows Against the Local Metadata Service

Once the metadata service is up and running on http://localhost:8080, you can execute Flows with the following command:

sh METAFLOW_SERVICE_URL=http://localhost:8080 METAFLOW_DEFAULT_METADATA=service python3 basicflow.py run

Managing Database Migrations

The Migration service allows users to manage underlying DB migrations effectively. It’s essential to maintain the metadata service and migration service independently while recognizing that they don’t communicate directly during runtime.

Migrating to the Latest DB Schema

For migration, follow these steps:

• Check the current status of the database schema.
• Back up your database before starting a migration.
• Run the upgrade command to apply any necessary migrations.
• Check back to ensure your schema is up-to-date.
• Perform a rolling restart of your metadata service cluster for the migration to take effect.

Troubleshooting

If you encounter issues while setting up the Metaflow ServiceMetadata:

• Ensure all necessary environment variables are correctly set.
• Verify that Docker is properly installed and running.
• Check service logs for error messages and follow the guidance provided.

For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Conclusion

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.