markitect-main/TESTING.md

# Testing Guide

This document provides comprehensive guidelines for testing the MarkiTect project.

## Overview

MarkiTect uses a multi-layered testing approach with pytest as the primary testing framework. Our testing strategy ensures code quality, reliability, and maintainability across all components.

## Testing Framework

- **Primary Framework**: pytest
- **Configuration**: `pytest.ini`
- **Test Directory**: `tests/`
- **Python Versions**: 3.8+

## Test Structure

```
tests/
├── conftest.py          # Shared test configuration and fixtures
├── e2e/                 # End-to-end tests
├── fixtures/            # Test data and fixtures
├── integration/         # Integration tests
├── unit/                # Unit tests (by component)
├── test_*.py           # Individual test modules
└── __pycache__/        # Python cache (auto-generated)
```

## Running Tests

### Quick Start

```bash
# Run all tests
pytest

# Run tests with verbose output
pytest -v

# Run specific test file
pytest tests/test_cli.py

# Run tests matching pattern
pytest -k "test_database"

# Run with coverage
pytest --cov=markitect --cov-report=html
```

### Test Categories

#### Unit Tests
```bash
# Run unit tests only
pytest tests/unit/

# Example: Test specific component
pytest tests/test_database.py
pytest tests/test_template_engine.py
```

#### Integration Tests
```bash
# Run integration tests
pytest tests/integration/

# Example: Test CLI integration
pytest tests/test_cli_integration.py
```

#### End-to-End Tests
```bash
# Run E2E tests
pytest tests/e2e/
```

## Test Configuration

### pytest.ini Configuration
- **Strict markers**: Enforces defined test markers
- **Verbose output**: Detailed test results
- **Duration tracking**: Shows slowest 10 tests
- **Fail fast**: Stops after 3 failures

### Custom Markers
```bash
# Performance tests
pytest -m performance

# Slow tests (run separately)
pytest -m slow

# Database tests
pytest -m database
```

## Writing Tests

### Test Naming Conventions
- Test files: `test_*.py`
- Test functions: `test_*`
- Test classes: `Test*`

### Example Test Structure
```python
import pytest
from markitect.core import MarkiTect

class TestMarkiTect:
    """Test suite for core MarkiTect functionality."""

    def test_basic_functionality(self):
        """Test basic operation."""
        # Arrange
        markitect = MarkiTect()

        # Act
        result = markitect.process("# Test")

        # Assert
        assert result is not None

    @pytest.mark.slow
    def test_performance_intensive(self):
        """Test that requires significant time."""
        pass
```

### Fixtures and Test Data
```python
# conftest.py
@pytest.fixture
def sample_markdown():
    """Provide sample markdown for testing."""
    return "# Sample\n\nTest content"

@pytest.fixture
def temp_database():
    """Provide temporary test database."""
    # Setup
    db = create_test_db()
    yield db
    # Cleanup
    db.close()
```

## Test Types and Guidelines

### Unit Tests
- **Scope**: Single function/method
- **Dependencies**: Mocked/isolated
- **Speed**: Fast (<100ms)
- **Location**: `tests/unit/`

### Integration Tests
- **Scope**: Component interaction
- **Dependencies**: Real dependencies within system
- **Speed**: Medium (100ms-2s)
- **Location**: `tests/integration/`

### End-to-End Tests
- **Scope**: Full system workflows
- **Dependencies**: Complete system
- **Speed**: Slow (>2s)
- **Location**: `tests/e2e/`

## Performance Testing

### Benchmarking
```bash
# Run performance benchmarks
markitect perf-benchmark --test-type all

# Validate performance thresholds
markitect perf-validate --threshold-ops 100
```

### Performance Tests in pytest
```python
@pytest.mark.performance
def test_large_document_processing():
    """Ensure large documents process within time limits."""
    start_time = time.time()
    # ... test logic ...
    duration = time.time() - start_time
    assert duration < 5.0  # Max 5 seconds
```

## Database Testing

### Test Database Setup
- Uses temporary SQLite databases
- Automatic cleanup after tests
- Isolated transactions per test

```python
@pytest.fixture
def test_db():
    """Provide isolated test database."""
    from markitect.database import DatabaseManager
    db = DatabaseManager(":memory:")  # In-memory database
    yield db
    db.close()
```

## CLI Testing

### Testing CLI Commands
```python
from click.testing import CliRunner
from markitect.cli import cli

def test_cli_help():
    """Test CLI help command."""
    runner = CliRunner()
    result = runner.invoke(cli, ['--help'])
    assert result.exit_code == 0
    assert 'MarkiTect' in result.output
```

## Continuous Integration

### GitHub Actions
- Automatic test execution on push/PR
- Multiple Python versions tested
- Coverage reports generated
- Configuration: `.github/workflows/test.yml`

### Quality Gates
- All tests must pass
- Coverage minimum: 80%
- No failing static analysis checks

## Test Data Management

### Fixtures Directory
```
tests/fixtures/
├── sample_documents/    # Test markdown files
├── expected_outputs/    # Expected test results
├── schemas/            # Test schemas
└── data/               # Test data files
```

### Test Data Guidelines
- Keep test data minimal but representative
- Use meaningful names
- Include edge cases
- Document complex test scenarios

## Debugging Tests

### Common Debugging Commands
```bash
# Run single test with detailed output
pytest tests/test_module.py::test_function -vvv

# Drop into debugger on failure
pytest --pdb

# Stop on first failure
pytest -x

# Show local variables in tracebacks
pytest --tb=long -l
```

### Logging in Tests
```python
import logging
import pytest

def test_with_logging(caplog):
    """Test that captures log output."""
    with caplog.at_level(logging.INFO):
        # ... test code that logs ...
        assert "Expected message" in caplog.text
```

## Best Practices

### Test Organization
1. **One concept per test**: Each test should verify one specific behavior
2. **Clear naming**: Test names should describe what is being tested
3. **Arrange-Act-Assert**: Structure tests clearly
4. **Independent tests**: Tests should not depend on each other

### Test Maintenance
1. **Keep tests simple**: Complex tests are hard to maintain
2. **Regular cleanup**: Remove obsolete tests
3. **Update documentation**: Keep this guide current
4. **Review coverage**: Aim for high but meaningful coverage

### Performance Considerations
1. **Fast feedback**: Unit tests should be very fast
2. **Parallel execution**: Tests should support parallel running
3. **Resource cleanup**: Always clean up resources
4. **Mocking**: Mock external dependencies appropriately

## Troubleshooting

### Common Issues

#### Import Errors
```bash
# Ensure PYTHONPATH is set correctly
export PYTHONPATH=.
pytest
```

#### Database Conflicts
```bash
# Clean test database
rm -f test_markitect.db
pytest
```

#### Slow Tests
```bash
# Profile test execution
pytest --durations=0
```

## Contributing

When contributing tests:

1. **Follow naming conventions**
2. **Add appropriate markers**
3. **Include docstrings**
4. **Test edge cases**
5. **Update this documentation if needed**

For more information about contributing, see the project's contribution guidelines.

## Resources

- [pytest Documentation](https://docs.pytest.org/)
- [Python Testing Best Practices](https://realpython.com/python-testing/)
- [Project Architecture Documentation](docs/architecture/)
- [Development Guidelines](docs/development/)