Automate benchmarking - disabling the docs workflow (#752)

## Summary
<!--- This is a required section; please describe the main purpose of
this proposed code change. --->
The docs workflow currently replaces the gh-pages branch with new files
due to which the benchmark data is lost. Currently I am disabling this
so that we can get the data appended in the gh-pages branch.
Also, modified the script to add commit hashes to a json file, so that
we do not call github api to fetch it due to API limitations
<!---
## Details
This is an optional section; is there anything specific that reviewers
should be aware of?
--->

## Testing Done
<!--- This is a required section; please describe how this change was
tested. --->

<!-- 
Replace BLANK with your device type. For example, A100-80G-PCIe

Complete the following tasks before sending your PR, and replace `[ ]`
with
`[x]` to indicate you have done them. 
-->

- Hardware Type: <BLANK>
- [ ] run `make test` to ensure correctness
- [ ] run `make checkstyle` to ensure code style
- [ ] run `make test-convergence` to ensure convergence

.github/workflows/benchmark.yml

@@ -29,7 +29,6 @@ jobs:
       OUTPUT_FILENAME: benchmark.csv
       GENERATED_CSV: benchmark/data/all_benchmark_data.csv
 
-
     steps:
       - name: Checkout code
         uses: actions/checkout@v3
@@ -52,9 +51,15 @@ jobs:
           pip install modal
           pip install pandas
 
-      # - name: Run benchmarks on GPU
-      #   run: |
-      #     modal run dev.modal.benchmarks
+    # Delete previous benchmark results.
+     - name: Remove previous benchmark data
+        run: |
+          rm -f benchmark/data/all_benchmark_data.csv
+
+
+      - name: Run benchmarks on GPU
+        run: |
+          modal run dev.modal.benchmarks
         
        # Step 5: Checkout gh-pages branch in a subfolderAdd commentMore actions
       - name: Checkout gh-pages
@@ -69,6 +74,21 @@ jobs:
           mkdir -p gh-pages/${OUTPUT_DIR}/${{ steps.get_hash.outputs.hash }}
           cp ${GENERATED_CSV} gh-pages/${OUTPUT_DIR}/${{ steps.get_hash.outputs.hash }}/${OUTPUT_FILENAME}
 
+      # Step 7: Append commit hash to commits.txt if not already present
+      - name: Update commits.txt
+        run: |
+          cd gh-pages
+          echo "commits.txt file path: ${OUTPUT_DIR}/commits.txt"
+
+          # Create file if it doesn't exist
+          mkdir -p ${OUTPUT_DIR}
+          touch ${OUTPUT_DIR}/commits.txt
+
+          # Append only if not already present
+          if ! grep -q "${{ steps.get_hash.outputs.hash }}" ${OUTPUT_DIR}/commits.txt; then
+            echo "${{ steps.get_hash.outputs.hash }}" >> ${OUTPUT_DIR}/commits.txt
+          fi
+
       # Step 7: Commit and push
       - name: Commit and push to gh-pages
         run: |

.github/workflows/docs.yml

@@ -3,10 +3,15 @@ on:
   push:
     branches:
       - main
+
+env:
+  ENABLE_DEPLOY: false
+
 permissions:
   contents: write
 jobs:
   deploy:
+    if: env.ENABLE_DEPLOY == 'true'
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4

dev/modal/benchmarks.py

@@ -14,7 +14,7 @@ app = modal.App("liger_benchmarks", image=image)
 repo = image.add_local_dir(ROOT_PATH, remote_path=REMOTE_ROOT_PATH)
 
 
-@app.function(gpu="A10G", image=repo, timeout=60 * 45)
+@app.function(gpu="H100", image=repo, timeout=60 * 45)
 def liger_benchmarks():
     import subprocess
 
@@ -25,3 +25,4 @@ def liger_benchmarks():
         cwd=REMOTE_ROOT_PATH,
     )
     subprocess.run(["python benchmark/scripts/benchmark_kto_loss.py"], check=True, shell=True, cwd=REMOTE_ROOT_PATH)
+    subprocess.run(["python benchmark/scripts/benchmark_cpo_loss.py"], check=True, shell=True, cwd=REMOTE_ROOT_PATH)