Format Implementation Guide

Adding New Formats

Step-by-Step Process

1. Create format file: iterable/datatypes/<format>.py
2. Implement BaseIterable: Inherit from BaseIterable in iterable/base.py
3. Required methods: read(), write(), readbulk(), writebulk(), etc.
4. Add detection: Update iterable/helpers/detect.py
5. Create tests: tests/test_<format>.py
6. Update dependencies: Add optional dependency to pyproject.toml if needed
7. Update documentation: Add format to docs

Implementation Pattern

from iterable.base import BaseIterable

class NewFormatIterable(BaseIterable):
    def __init__(self, source, mode='r', **kwargs):
        super().__init__(source, mode, **kwargs)
        # Initialize format-specific resources
    
    def read(self):
        # Return iterator of dict objects
        pass
    
    def write(self, data):
        # Write dict objects to file
        pass
    
    def read_bulk(self, size=1000):
        # Bulk read for performance
        pass
    
    def write_bulk(self, data):
        # Bulk write for performance
        pass

Format Detection

Update iterable/helpers/detect.py:

1. Add file extension detection
2. Add magic number detection (for binary formats)
3. Add content-based heuristics if needed
4. Update detectfiletype() function

Example:

def detect_file_type(filename, content=None):
    # Check extension
    if filename.endswith('.newformat'):
        return 'newformat'
    
    # Check magic numbers
    if content and content.startswith(b'MAGIC'):
        return 'newformat'
    
    # ... existing detection logic

Adding New Codecs

Step-by-Step Process

1. Create codec file: iterable/codecs/<codec>codec.py
2. Implement codec class: read(), write(), close() methods
3. Add detection: Update iterable/helpers/detect.py
4. Add compression detection: Update format detection logic
5. Create tests: Add to relevant test file or create new one
6. Update dependencies: Add optional dependency to pyproject.toml

Codec Pattern

class NewCodec:
    def __init__(self, fileobj, mode='r'):
        self.fileobj = fileobj
        self.mode = mode
        # Initialize compression library
    
    def read(self, size=-1):
        # Decompress and return data
        pass
    
    def write(self, data):
        # Compress and write data
        pass
    
    def close(self):
        # Clean up resources
        pass

Testing Requirements

Test File Structure

import pytest
from iterable import open_iterable

class TestNewFormat:
    def test_read(self):
        # Test basic reading
        pass
    
    def test_write(self):
        # Test basic writing
        pass
    
    def test_read_bulk(self):
        # Test bulk operations
        pass
    
    def test_compressed(self):
        # Test with compression (.gz, .bz2, .zst, etc.)
        pass
    
    def test_edge_cases(self):
        # Empty files, malformed data, etc.
        pass

Test Coverage

• Basic read/write operations
• Bulk operations
• Compressed files (if supported)
• Various encodings (for text formats)
• Edge cases: empty files, malformed data
• Missing optional dependencies (should skip gracefully)

Dependencies

Optional Dependencies

Add to pyproject.toml:

[project.optional-dependencies]
newformat = ["newformat-library>=1.0.0"]

Import Handling

Handle missing dependencies gracefully:

try:
    import newformat_library
except ImportError:
    raise ImportError(
        "newformat support requires 'newformat-library'. "
        "Install with: pip install iterabledata[newformat]"
    )

Format Capabilities

Implement capability reporting:

def get_capabilities(self):
    return {
        'read': True,
        'write': True,
        'bulk': True,
        'totals': False,  # Can't count rows without reading
        'streaming': True,
        'tables': False,  # Single table format
    }

Examples

Look at existing implementations:

• iterable/datatypes/csv.py - Text format example
• iterable/datatypes/parquet.py - Binary format example
• iterable/codecs/gzipcodec.py - Compression codec example

Common Pitfalls

• Memory issues: Use streaming for large files
• Encoding: Handle various text encodings automatically
• Compression: Support common codecs (gzip, bz2, zstd, etc.)
• Error messages: Provide helpful errors for missing dependencies
• Context managers: Always support with statements
• Filename-only formats: If your format requires a file path and does not support stream or codec, validate in reset() (or init) and raise a clear error (e.g. ReadError) when filename is None.
• readbulk() exhaustion: When no more records are available, readbulk() MUST return an empty list []; do not raise StopIteration.