Vector Analysis Performance Testing with Ann-Benchmarks ​

1. Preparation ​

Test Environment ​

• Install Python >= 3.8.6.
• Download the Ann-Benchmarks performance testing tool adapted for the openGauss database. Download: ann-benchmarks-openGauss
• Install the dependencies required by the testing tool: pip3 install -r requirements.txt.
• Deploy an openGauss-DataVec container instance. For details, see Installing the openGauss-DataVec Container Image.

Test Data ​

Note:

• Dataset location: Place the datasets in /ann-benchmarks-openGauss/data. Create the directory first by running mkdir data.
• Dataset download: You can download the datasets directly using wget. For example: wget http://ann-benchmarks.com/glove-50-angular.hdf5 --no-check-certificate.

2. Test Procedure ​

Database Configuration ​

The configuration file is located at the following path inside the container. Note that you must restart the container for changes to database parameters to take effect.

/var/lib/opengauss/data/postgresql.conf

Recommended configuration parameters:

shared_buffers=50GB # Recommended to be greater than the total size of the database and indexes.
maintenance_work_mem=4GB
password_encryption_type=1
max_connections=1000 # Maximum number of connections.

For details about modifying these parameters, see GUC Parameter Usage.

Ann-Benchmarks Configuration ​

Modify the database connection settings in go_opgs.sh.

export ANN_BENCHMARKS_OG_USER='YourUserName'
export ANN_BENCHMARKS_OG_PASSWORD='YourPassword'
export ANN_BENCHMARKS_OG_DBNAME='YourDBName'
export ANN_BENCHMARKS_OG_HOST='YourHost'
export ANN_BENCHMARKS_OG_PORT=YourPort

Modify the index construction and index query parameters in ann-benchmarks-openGauss/ann_benchmarks/algorithms/openGauss/config.yml as required.

  - base_args: ['@metric']
    constructor: openGaussHNSW
    disabled: false
    docker_tag: ann-benchmarks-openGauss
    module: ann_benchmarks.algorithms.openGauss
    name: openGauss-hnsw
    run_groups:
       M-16:
         arg_groups: [{M: 16, efConstruction: 200, concurrents: 80}]
         args: {}
         query_args: [[10, 20, 40, 80, 120, 200, 400, 800]]
       M-24:
         arg_groups: [{M: 24, efConstruction: 200, concurrents: 80}]
         args: {}
         query_args: [[10, 20, 40, 80, 120, 200, 400, 800]]

• name: name of the approximate search algorithm.
• run_groups: index construction and index query parameter settings. arg_groups specifies the HNSW index construction parameters M and efConstruction. concurrents specifies the number of concurrent threads. The recommended value is the number of CPU cores. query_args specifies the HNSW index query parameter ef_search.

Running the Test ​

Modify the startup command in go_opgs.sh.

python3 run.py --algorithm openGauss-hnsw --dataset fashion-mnist-784-euclidean --local --runs 1 -k 10 --batch

• --algorithm: algorithm name. The algorithm name is specified by the name field in ann-benchmarks-openGauss/ann_benchmarks/algorithms/openGauss/config.yml. The currently supported algorithms are openGauss-hnsw, openGauss-hnswpq, and openGauss-ivfflat.
• --dataset: dataset name. For supported datasets, see the Test Data section in Preparation.
• --runs: number of times to run the test set.
• --k: number of top-K results.
• --batch: enables concurrent vector queries when this parameter is specified.

Start the test.

sh go_opgs.sh

Note:
If the same test group has been run previously, rename or delete the corresponding files under ann-benchmarks-openGauss/results/<dataset>/<k>/<algorithm>. Otherwise, the test group will be skipped directly.

3. Test Results ​

Generating an Interactive HTML Web Page ​

python3 create_website.py --outputdir <YOUR_RESULT_PATH> --scatter --recompute

Exporting Test Results ​

python3 data_export.py --out <result_file_name>.csv

Test result metrics:

• Recall: recall
• qps: throughput
• p99: P99 latency
• build: index construction time