langchain-ai/langsmith-self-hosted-workshops

mirror of https://github.com/langchain-ai/langsmith-self-hosted-workshops.git synced 2026-06-30 20:48:02 -04:00

Files

T

History

Cory Waddingham b0b424eabd Add tests for Module 3 and Module 4 notebooks

This commit adds comprehensive test coverage for Module 3 (Production
Operations & Scaling) and Module 4 (Troubleshooting & Incident Response)
notebooks, following the same pattern established for Modules 1 and 2.

Test Infrastructure:
- tests/test_notebook_execution.py:
  - Added TestModule3Notebooks class with syntax and execution tests
    for 01_ops_sanity_checks.ipynb
  - Added TestModule4Notebooks class with:
    - Syntax tests for all 6 Module 4 notebooks (00, 01, 10, 20, 30, 40)
    - Execution tests for setup/baseline notebooks (00, 01) - read-only
    - Execution tests for failure labs (10, 20, 30, 40) - with warnings
      about failure injection and secret modification

CI/CD Integration:
- .github/workflows/test-notebooks.yml:
  - Added Module 3 and Module 4 syntax tests to CI pipeline
  - All syntax tests now run automatically on PRs and pushes

Test Features:
- Respects CI_SKIP_EXECUTION environment variable (same as Module 1)
- Uses environment variables for configuration (cloud provider, region, etc.)
- Appropriate timeouts: 600s for ops checks, 900s for failure labs
- Safety warnings for failure lab execution tests
- Syntax tests always run (including in CI)
- Execution tests skip in CI when CI_SKIP_EXECUTION=true

Module 4 Test Structure:
- Syntax tests: All 6 notebooks validated for structure
- Setup/Baseline execution: Read-only validation notebooks (00, 01)
- Failure lab execution: Separate test method with warnings about
  secret modification and failure injection (10, 20, 30, 40)

This ensures all workshop notebooks are validated for syntax correctness
and can be execution-tested when infrastructure is available, maintaining
consistency with existing test patterns.

2026-01-02 10:55:02 -08:00

README.md

feat: Add CI/CD testing infrastructure for notebooks

2026-01-02 09:19:36 -08:00

test-notebooks.yml

Add tests for Module 3 and Module 4 notebooks

2026-01-02 10:55:02 -08:00

README.md

GitHub Actions Workflows

This directory contains CI/CD workflows for the LangSmith Self-Hosted Workshops repository.

Workflows

`test-notebooks.yml`

Main workflow for testing notebook syntax and execution.

Triggers:

Pull requests to main/master
Pushes to main/master
Manual workflow dispatch

Jobs:

test-notebook-syntax: Validates notebook JSON structure
test-module-1-preflight: Tests Module 1 preflight notebook
test-module-2-syntax: Tests Module 2 auth validation notebooks
lint-python: Lints Python code in shared modules

Environment Variables

Required for Syntax Tests (Always Available)

These are set in the workflow and don't require secrets:

NAMESPACE: "langsmith-test"
CLUSTER_NAME: "test-cluster"
HELM_RELEASE: "langsmith"
CLOUD_PROVIDER: "aws"
AWS_REGION: "us-west-2"
LANGSMITH_DOMAIN: "test.langsmith.example.com"

Required for Full Execution Tests (Optional)

For full notebook execution, set these in GitHub Secrets:

AWS:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_REGION
AWS_ACCOUNT_ID (optional, for validation)

Azure:

AZURE_CLIENT_ID
AZURE_CLIENT_SECRET
AZURE_TENANT_ID
AZURE_SUBSCRIPTION_ID
AZURE_LOCATION

Infrastructure:

CLUSTER_NAME
NAMESPACE
TERRAFORM_REPO_DIR
HELM_REPO_DIR

OIDC/SAML (Module 2):

OIDC_ISSUER
OIDC_CLIENT_ID
OIDC_CLIENT_SECRET
OIDC_REDIRECT_URI
SAML_METADATA_URL (if using SAML)

Customizing Workflows

Adding New Test Jobs

Add job to test-notebooks.yml
Set appropriate needs: dependencies
Configure environment variables
Add artifact uploads if needed

Enabling Full Execution Tests

To enable full notebook execution in CI:

Set required secrets in GitHub repository settings
Remove or modify CI_SKIP_EXECUTION environment variable
Update test conditions in test_notebook_execution.py

Adding New Modules

When adding Module 3, 4, etc.:

Create new test class in test_notebook_execution.py
Add parametrized tests for new notebooks
Add new job in GitHub Actions workflow
Update this README

Workflow Status

Workflow status badges can be added to README:

![Test Notebooks](https://github.com/your-org/langsmith-self-hosted-workshops/workflows/Test%20Notebooks/badge.svg)

Troubleshooting

Workflow Fails on Syntax Tests

Check notebook JSON is valid
Verify all imports are available
Check Python version compatibility

Workflow Times Out

Increase timeout-minutes in job definition
Check for long-running operations
Consider splitting into smaller jobs

Environment Variable Issues

Verify secrets are set in repository settings
Check variable names match exactly
Ensure secrets are accessible to workflow