Welcome to the world of MinSQL, a log search engine designed for simplicity! In this article, we will explore how to set up and run MinSQL, making log management easier than ever without the need for complex SDKs. Instead, you’ll only need basic HTTP request capabilities. Let’s dive in!

Building and Running MinSQL

To start using MinSQL, you’ll first need to build it. You can either use Docker or a direct build using Make. Here’s how to do both:

• Using Docker:

  docker build . -t miniominsql

  docker run --rm miniominsql --help

• Using Make:

  make minsqld --help

Preparing MinSQL Environment

MinSQL requires an instance of MinIO as the storage engine. Below is a sample docker-compose.yml configuration:

version: '2'
services:
  minio-engine:
    image: minio/minio
    volumes:
      - data:data
    environment:
      MINIO_ACCESS_KEY: minio
      MINIO_SECRET_KEY: minio123
    command: server data

  mc:
    image: minio/mc
    depends_on:
      - minio
    entrypoint:
      /bin/sh -c "echo usr/bin/mc config host add http:minio-engine:9000 minio minio123; /usr/bin/mc mb minsql-meta;"

  minsqld:
    image: miniominsql
    depends_on:
      - minio
      - mc
    ports:
      - "9999:9999"
    environment:
      MINSQL_METABUCKET_NAME: minsql-meta
      MINSQL_METABUCKET_ENDPOINT: http:minio-engine:9000
      MINSQL_ACCESS_KEY: minio
      MINSQL_SECRET_KEY: minio123
      MINSQL_ROOT_ACCESS_KEY: minsqlaaccesskeyx
      MINSQL_ROOT_SECRET_KEY: minsqlexamplekeypleasechangexxxx
volumes:
  data:

To run the project, execute:

docker-compose up

Configuring Environment Variables

Before you can start storing logs, you need to set up some environment variables:

• MINSQL_METABUCKET_NAME: Name of the meta bucket
• MINSQL_METABUCKET_ENDPOINT: Example: http://localhost:9000
• MINSQL_METABUCKET_ACCESS_KEY: Meta Bucket access key
• MINSQL_METABUCKET_SECRET_KEY: Meta Bucket secret key
• MINSQL_ROOT_ACCESS_KEY: Root access key for MinSQL
• MINSQL_ROOT_SECRET_KEY: Root secret key for MinSQL

Adding Data to MinSQL

‘Think of MinSQL as a library,’ where each book represents a log entry. To add a book (or log), you need to categorize it first. This is achieved by creating a datastore, a log, generating a token, and authorizing that token.

Step-by-Step Instructions:

• Add a Sample Datastore:

  curl -X POST http://127.0.0.1:9999/api/datastores -H "Content-Type: application/json" -d '{"bucket": "play-minsql", "endpoint": "https://play.minio.io:9000", "prefix": "", "name": "minioplay", "access_key": "Q3AM3UQ867SPQQA43P2F", "secret_key": "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"}'

• Add a Sample Log:

  curl -X POST http://127.0.0.1:9999/api/logs -H "Content-Type: application/json" -d '{"name": "mylog", "datastores": ["minioplay"], "commit_window": "5s"}'

• Create a Sample Token:

  curl -X POST http://127.0.0.1:9999/api/tokens -H "Content-Type: application/json" -d '{"access_key": "abcdefghijklmnop", "secret_key": "abcdefghijklmnopabcdefghijklmnop", "description": "test", "is_admin": true, "enabled": false}'

• Authorize Token to Log:

  curl -X POST http://127.0.0.1:9999/api/auth/abcdefghijklmnop -H "Content-Type: application/json" -d '{"log_name": "mylog", "api": ["search", "store"]}'

Storing and Querying Logs

To store logs in MinSQL, perform a PUT operation:

curl -X PUT http://127.0.0.1:9999/mylog/store -H "MINSQL-TOKEN: TOKEN1" -d "10.8.0.1 - - [16May2019:23:02:56 +0000] GET HTTP/1.1 400 256 - Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"

To query logs, perform a POST with a SQL statement:

curl -X POST http://127.0.0.1:9999/search -H "MINSQL-TOKEN: TOKEN1" -d "SELECT * FROM mylog"

Troubleshooting Tips

If you encounter issues while setting up or using MinSQL, here are some troubleshooting ideas:

• Ensure that MinIO is running and accessible at the specified endpoint.
• Check that your environment variables are correctly set before starting MinSQL.
• Make sure that all URLs are properly formatted and accessible.
• If using Docker, ensure that the Docker service is running and that you’ve executed all commands in the correct order.

For assistance or guidance, feel free to reach us at fxis.ai for more insights, updates, or to collaborate on AI development projects.

At fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.