Skip to content

Commit 2448a01

Browse files
kocsismatekocsismate
authored
Add real-time benchmark to CI as a nightly job (#15696)
This PR integrates https://github.com/kocsismate/php-version-benchmarks/ into the CI as a nightly job running every day at 12:30 AM UTC. Roughly, the following happens: the benchmark suite spins up an AWS EC2 instance via Terraform, runs the tests according to the configuration, and then the results are committed to the https://github.com/kocsismate/php-version-benchmark-results repository. In order to have as stable results as possible, the CPU, kernel and other settings of the AWS instance are fine-tuned: - Hyper-threading is disabled - Turbo boost is disabled - C states of the CPU are limited: https://docs.aws.amazon.com/linux/al2/ug/processor_state_control.html#baseline-perf - The workload is dedicated to a single core by using taskset according to Intel's recommendations (https://web.archive.org/web/20210614053522/https://01.org/node/3774) - An io2 SSD volume is attached to the instance which has a provisioned IOPS (https://docs.aws.amazon.com/ebs/latest/userguide/provisioned-iops.html#io2-block-express) so that IO performance is nearly constant - The instance is dedicated so that the noisy neighbor effect is eliminated: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/dedicated-instance.html - ASLR is disabled (Disable ASLR for benchmark #13769) Customizing the CPU is only supported by metal instances among recent instance types according to https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/processor_state_control.html, so at last, a c7i.metal-24xl instance is used in the eu-west-1 region. The benchmark suite compares the performance of the latest commit of the master branch in the time when the benchmark runs with the last commit of master from the day before yesterday. I.e. if the benchmark runs tomorrow morning at 2 AM, then the performance of the latest commit will be benchmarked against the last commit pushed yesterday. This makes it possible to spot outstanding regressions (or progressions) in time. Actually, the end goal is to send notifications in case of any significant changes for further analyzation. The reason why the benchmark is run for previous commits as well (while they may have already been measured the day before) is to make the results less sensitive for changes in the environment or the benchmark suite itself. I.e.: if AWS upgrades the OS, or if the code under test is modified, then the numbers will likely be affected, and the previous results will be invalidated).
1 parent 3c8c0df commit 2448a01

File tree

1 file changed

+132
-0
lines changed

1 file changed

+132
-0
lines changed

‎.github/workflows/real-time-benchmark.yml

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
name: Real-time Benchmark
2+
on:
3+
schedule:
4+
- cron: "30 0 * * *"
5+
permissions:
6+
contents: read
7+
jobs:
8+
REAL_TIME_BENCHMARK:
9+
name: REAL_TIME_BENCHMARK
10+
if: github.repository == 'php/php-src'
11+
runs-on: ubuntu-22.04
12+
steps:
13+
- name: Install dependencies
14+
run: |
15+
set -ex
16+
sudo apt-get update
17+
sudo apt-get install gpg
18+
19+
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
20+
gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint
21+
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
22+
export DEBIAN_FRONTEND=noninteractive
23+
sudo apt-get update -y
24+
sudo apt-get install -y terraform=1.5.7-*
25+
- name: Checkout benchmark suite
26+
uses: actions/checkout@v4
27+
with:
28+
repository: 'kocsismate/php-version-benchmarks'
29+
ref: 'main'
30+
fetch-depth: 1
31+
path: 'php-version-benchmarks'
32+
- name: Checkout php-src
33+
uses: actions/checkout@v4
34+
with:
35+
repository: 'php/php-src'
36+
ref: '${{ github.sha }}'
37+
fetch-depth: 100
38+
path: 'php-version-benchmarks/tmp/php_master'
39+
- name: Setup benchmark results
40+
run: |
41+
git config --global user.name "Benchmark"
42+
git config --global user.email "[email protected]"
43+
44+
rm -rf ./php-version-benchmarks/docs/results
45+
- name: Checkout benchmark data
46+
uses: actions/checkout@v4
47+
with:
48+
repository: php/real-time-benchmark-data
49+
ssh-key: ${{ secrets.PHP_VERSION_BENCHMARK_RESULTS_DEPLOY_KEY }}
50+
path: 'php-version-benchmarks/docs/results'
51+
- name: Set benchmark config
52+
run: |
53+
set -e
54+
55+
# Set infrastructure config
56+
cp ./php-version-benchmarks/config/infra/aws/x86_64-metal.ini.dist ./php-version-benchmarks/config/infra/aws/x86_64-metal.ini
57+
ESCAPED_DOCKER_REGISTRY=$(printf '%s\n' "${{ secrets.PHP_VERSION_BENCHMARK_DOCKER_REGISTRY }}" | sed -e 's/[\/&]/\\&/g')
58+
sed -i "s/INFRA_DOCKER_REGISTRY=public.ecr.aws\/abcdefgh/INFRA_DOCKER_REGISTRY=$ESCAPED_DOCKER_REGISTRY/g" ./php-version-benchmarks/config/infra/aws/x86_64-metal.ini
59+
cp ./php-version-benchmarks/build/infrastructure/config/aws.tfvars.dist ./php-version-benchmarks/build/infrastructure/config/aws.tfvars
60+
sed -i 's/access_key = ""/access_key = "${{ secrets.PHP_VERSION_BENCHMARK_AWS_ACCESS_KEY }}"/g' ./php-version-benchmarks/build/infrastructure/config/aws.tfvars
61+
sed -i 's/secret_key = ""/secret_key = "${{ secrets.PHP_VERSION_BENCHMARK_AWS_SECRET_KEY }}"/g' ./php-version-benchmarks/build/infrastructure/config/aws.tfvars
62+
63+
YEAR="$(date '+%Y')"
64+
DATABASE="./php-version-benchmarks/docs/results/$YEAR/database.tsv"
65+
if [ -f "$DATABASE" ]; then
66+
LAST_RESULT_SHA="$(tail -n 2 "$DATABASE" | head -n 1 | cut -f 6)"
67+
else
68+
YESTERDAY="$(date -d "-2 day 23:59:59" '+%Y-%m-%d %H:%M:%S')"
69+
LAST_RESULT_SHA="$(cd ./php-version-benchmarks/tmp/php_master/ && git --no-pager log --until="$YESTERDAY" -n 1 --pretty='%H')"
70+
fi
71+
72+
BASELINE_SHA="d5f6e56610c729710073350af318c4ea1b292cfe"
73+
BASELINE_SHORT_SHA="$(echo "$BASELINE_SHA" | cut -c1-4)"
74+
75+
# Set config for the baseline PHP version
76+
cp ./php-version-benchmarks/config/php/master.ini.dist ./php-version-benchmarks/config/php/master_baseline.ini
77+
sed -i 's/PHP_NAME="PHP - master"/PHP_NAME="PHP - baseline@'"$BASELINE_SHORT_SHA"'"/g' ./php-version-benchmarks/config/php/master_baseline.ini
78+
sed -i "s/PHP_ID=php_master/PHP_ID=php_master_baseline/g" ./php-version-benchmarks/config/php/master_baseline.ini
79+
sed -i "s/PHP_COMMIT=/PHP_COMMIT=$BASELINE_SHA/g" ./php-version-benchmarks/config/php/master_baseline.ini
80+
81+
# Set config for the previous PHP version
82+
cp ./php-version-benchmarks/config/php/master.ini.dist ./php-version-benchmarks/config/php/master_last.ini
83+
sed -i 's/PHP_NAME="PHP - master"/PHP_NAME="PHP - previous master"/g' ./php-version-benchmarks/config/php/master_last.ini
84+
sed -i "s/PHP_ID=php_master/PHP_ID=php_master_previous/g" ./php-version-benchmarks/config/php/master_last.ini
85+
sed -i "s/PHP_COMMIT=/PHP_COMMIT=$LAST_RESULT_SHA/g" ./php-version-benchmarks/config/php/master_last.ini
86+
87+
# Set config for the current PHP version
88+
cp ./php-version-benchmarks/config/php/master.ini.dist ./php-version-benchmarks/config/php/master_now.ini
89+
sed -i "s/PHP_COMMIT=/PHP_COMMIT=${{ github.sha }}/g" ./php-version-benchmarks/config/php/master_now.ini
90+
91+
# Set config for current PHP version with JIT
92+
git clone ./php-version-benchmarks/tmp/php_master/ ./php-version-benchmarks/tmp/php_master_jit
93+
cp ./php-version-benchmarks/config/php/master_jit.ini.dist ./php-version-benchmarks/config/php/master_now_jit.ini
94+
sed -i "s/PHP_COMMIT=/PHP_COMMIT=${{ github.sha }}/g" ./php-version-benchmarks/config/php/master_now_jit.ini
95+
96+
# Set test configs
97+
cp ./php-version-benchmarks/config/test/1_laravel.ini.dist ./php-version-benchmarks/config/test/1_laravel.ini
98+
cp ./php-version-benchmarks/config/test/2_symfony_main.ini.dist ./php-version-benchmarks/config/test/2_symfony_main.ini
99+
cp ./php-version-benchmarks/config/test/4_wordpress.ini.dist ./php-version-benchmarks/config/test/4_wordpress.ini
100+
cp ./php-version-benchmarks/config/test/5_bench.php.ini.dist ./php-version-benchmarks/config/test/5_bench.php.ini
101+
cp ./php-version-benchmarks/config/test/6_micro_bench.php.ini.dist ./php-version-benchmarks/config/test/6_micro_bench.php.ini
102+
- name: Run benchmark
103+
run: ./php-version-benchmarks/benchmark.sh run aws
104+
- name: Store results
105+
run: |
106+
set -ex
107+
108+
cd ./php-version-benchmarks/docs/results
109+
git pull --autostash
110+
if [ -e ".git/MERGE_HEAD" ]; then
111+
echo "Merging, can't proceed"
112+
exit 1
113+
fi
114+
git add .
115+
if git diff --cached --quiet; then
116+
exit 1
117+
fi
118+
git commit -m "Add result for ${{ github.repository }}@${{ github.sha }}"
119+
git push
120+
- name: Cleanup
121+
if: always()
122+
run: |
123+
set -ex
124+
125+
rm -rf ./php-version-benchmarks/tmp/
126+
rm -f ./php-version-benchmarks/build/infrastructure/config/*.tfvars
127+
rm -rf ./php-version-benchmarks/build/infrastructure/aws/.terraform/
128+
rm -f ./php-version-benchmarks/build/infrastructure/aws/.terraform.lock.hcl
129+
rm -f ./php-version-benchmarks/build/infrastructure/aws/aws.tfplan
130+
rm -f ./php-version-benchmarks/build/infrastructure/aws/terraform.tfstate
131+
rm -f ./php-version-benchmarks/build/infrastructure/aws/terraform.tfstate.backup
132+
rm -f ./php-version-benchmarks/config/infra/aws/*.ini

0 commit comments

Comments
 (0)