markitect-main/TESTING.md

Testing Guide

This document provides comprehensive guidelines for testing the MarkiTect project.

Overview

MarkiTect uses a multi-layered testing approach with pytest as the primary testing framework. Our testing strategy ensures code quality, reliability, and maintainability across all components.

Testing Framework

• Primary Framework: pytest
• Configuration: pytest.ini
• Test Directory: tests/
• Python Versions: 3.8+

Test Structure

tests/
├── conftest.py          # Shared test configuration and fixtures
├── e2e/                 # End-to-end tests
├── fixtures/            # Test data and fixtures
├── integration/         # Integration tests
├── unit/                # Unit tests (by component)
├── test_*.py           # Individual test modules
└── __pycache__/        # Python cache (auto-generated)

Running Tests

Quick Start

# Run all tests
pytest

# Run tests with verbose output
pytest -v

# Run specific test file
pytest tests/test_cli.py

# Run tests matching pattern
pytest -k "test_database"

# Run with coverage
pytest --cov=markitect --cov-report=html

Test Categories

Unit Tests

# Run unit tests only
pytest tests/unit/

# Example: Test specific component
pytest tests/test_database.py
pytest tests/test_template_engine.py

Integration Tests

# Run integration tests
pytest tests/integration/

# Example: Test CLI integration
pytest tests/test_cli_integration.py

End-to-End Tests

# Run E2E tests
pytest tests/e2e/

Test Configuration

pytest.ini Configuration

• Strict markers: Enforces defined test markers
• Verbose output: Detailed test results
• Duration tracking: Shows slowest 10 tests
• Fail fast: Stops after 3 failures

Custom Markers

# Performance tests
pytest -m performance

# Slow tests (run separately)
pytest -m slow

# Database tests
pytest -m database

Writing Tests

Test Naming Conventions

• Test files: test_*.py
• Test functions: test_*
• Test classes: Test*

Example Test Structure

import pytest
from markitect.core import MarkiTect

class TestMarkiTect:
    """Test suite for core MarkiTect functionality."""

    def test_basic_functionality(self):
        """Test basic operation."""
        # Arrange
        markitect = MarkiTect()

        # Act
        result = markitect.process("# Test")

        # Assert
        assert result is not None

    @pytest.mark.slow
    def test_performance_intensive(self):
        """Test that requires significant time."""
        pass

Fixtures and Test Data

# conftest.py
@pytest.fixture
def sample_markdown():
    """Provide sample markdown for testing."""
    return "# Sample\n\nTest content"

@pytest.fixture
def temp_database():
    """Provide temporary test database."""
    # Setup
    db = create_test_db()
    yield db
    # Cleanup
    db.close()

Test Types and Guidelines

Unit Tests

• Scope: Single function/method
• Dependencies: Mocked/isolated
• Speed: Fast (<100ms)
• Location: tests/unit/

Integration Tests

• Scope: Component interaction
• Dependencies: Real dependencies within system
• Speed: Medium (100ms-2s)
• Location: tests/integration/

End-to-End Tests

• Scope: Full system workflows
• Dependencies: Complete system
• Speed: Slow (>2s)
• Location: tests/e2e/

Performance Testing

Benchmarking

# Run performance benchmarks
markitect perf-benchmark --test-type all

# Validate performance thresholds
markitect perf-validate --threshold-ops 100

Performance Tests in pytest

@pytest.mark.performance
def test_large_document_processing():
    """Ensure large documents process within time limits."""
    start_time = time.time()
    # ... test logic ...
    duration = time.time() - start_time
    assert duration < 5.0  # Max 5 seconds

Database Testing

Test Database Setup

• Uses temporary SQLite databases
• Automatic cleanup after tests
• Isolated transactions per test

@pytest.fixture
def test_db():
    """Provide isolated test database."""
    from markitect.database import DatabaseManager
    db = DatabaseManager(":memory:")  # In-memory database
    yield db
    db.close()

CLI Testing

Testing CLI Commands

from click.testing import CliRunner
from markitect.cli import cli

def test_cli_help():
    """Test CLI help command."""
    runner = CliRunner()
    result = runner.invoke(cli, ['--help'])
    assert result.exit_code == 0
    assert 'MarkiTect' in result.output

Continuous Integration

GitHub Actions

• Automatic test execution on push/PR
• Multiple Python versions tested
• Coverage reports generated
• Configuration: .github/workflows/test.yml

Quality Gates

• All tests must pass
• Coverage minimum: 80%
• No failing static analysis checks

Test Data Management

Fixtures Directory

tests/fixtures/
├── sample_documents/    # Test markdown files
├── expected_outputs/    # Expected test results
├── schemas/            # Test schemas
└── data/               # Test data files

Test Data Guidelines

• Keep test data minimal but representative
• Use meaningful names
• Include edge cases
• Document complex test scenarios

Debugging Tests

Common Debugging Commands

# Run single test with detailed output
pytest tests/test_module.py::test_function -vvv

# Drop into debugger on failure
pytest --pdb

# Stop on first failure
pytest -x

# Show local variables in tracebacks
pytest --tb=long -l

Logging in Tests

import logging
import pytest

def test_with_logging(caplog):
    """Test that captures log output."""
    with caplog.at_level(logging.INFO):
        # ... test code that logs ...
        assert "Expected message" in caplog.text

Best Practices

Test Organization

1. One concept per test: Each test should verify one specific behavior
2. Clear naming: Test names should describe what is being tested
3. Arrange-Act-Assert: Structure tests clearly
4. Independent tests: Tests should not depend on each other

Test Maintenance

1. Keep tests simple: Complex tests are hard to maintain
2. Regular cleanup: Remove obsolete tests
3. Update documentation: Keep this guide current
4. Review coverage: Aim for high but meaningful coverage

Performance Considerations

1. Fast feedback: Unit tests should be very fast
2. Parallel execution: Tests should support parallel running
3. Resource cleanup: Always clean up resources
4. Mocking: Mock external dependencies appropriately

Troubleshooting

Common Issues

Import Errors

# Ensure PYTHONPATH is set correctly
export PYTHONPATH=.
pytest

Database Conflicts

# Clean test database
rm -f test_markitect.db
pytest

Slow Tests

# Profile test execution
pytest --durations=0

Contributing

When contributing tests:

1. Follow naming conventions
2. Add appropriate markers
3. Include docstrings
4. Test edge cases
5. Update this documentation if needed

For more information about contributing, see the project's contribution guidelines.

Resources

• pytest Documentation
• Python Testing Best Practices
• Project Architecture Documentation
• Development Guidelines