grepai-storage-gob

GrepAI Storage with GOB

This skill covers using GOB (Go Binary) as the storage backend for GrepAI, the default and simplest option.

When to Use This Skill

• Single developer projects
• Small to medium codebases
• Simple setup without external dependencies
• Local development environments

What is GOB Storage?

GOB is Go's native binary serialization format. GrepAI uses it to store:

• Vector embeddings
• File metadata
• Chunk information

Everything is stored in a single local file.

Advantages

Benefit	Description
🚀 Simple	No external services needed
⚡ Fast setup	Works immediately
📁 Portable	Single file, easy to backup
💰 Free	No infrastructure costs
🔒 Private	Data stays local

Limitations

Limitation	Description
📏 Scalability	Not ideal for very large codebases
👤 Single user	No concurrent access
🔄 No sharing	Can't share index across machines
💾 Memory	Loads into RAM for searches

Configuration

Default Configuration

GOB is the default backend. Minimal config:

# .grepai/config.yaml
store:
  backend: gob

Explicit Configuration

store:
  backend: gob
  # Index stored in .grepai/index.gob (automatic)

Storage Location

GOB storage creates files in your project's .grepai/ directory:

.grepai/
├── config.yaml    # Configuration
├── index.gob      # Vector embeddings
└── symbols.gob    # Symbol index for trace

File Sizes

Approximate .grepai/index.gob sizes:

Codebase	Files	Chunks	Index Size
Small	100	500	~5 MB
Medium	1,000	5,000	~50 MB
Large	10,000	50,000	~500 MB

Operations

Creating the Index

# Initialize project
grepai init

# Start indexing (creates index.gob)
grepai watch

Checking Index Status

grepai status

# Output:
# Index: .grepai/index.gob
# Files: 245
# Chunks: 1,234
# Size: 12.5 MB
# Last updated: 2025-01-28 10:30:00

Backing Up the Index

# Simple file copy
cp .grepai/index.gob .grepai/index.gob.backup

Clearing the Index

# Delete and re-index
rm .grepai/index.gob
grepai watch

Moving to a New Machine

# Copy entire .grepai directory
cp -r .grepai /path/to/new/location/

# Note: Only works if using same embedding model

Performance Considerations

Memory Usage

GOB loads the entire index into RAM for searches:

Index Size	RAM Usage
10 MB	~20 MB
50 MB	~100 MB
500 MB	~1 GB

Search Speed

GOB provides fast searches for typical codebases:

Codebase Size	Search Time
Small (100 files)	<50ms
Medium (1K files)	<200ms
Large (10K files)	<1s

When to Upgrade

Consider PostgreSQL or Qdrant when:

• Index exceeds 1 GB
• Need concurrent access
• Want to share index across team
• Codebase has 50K+ files

.gitignore Configuration

Add .grepai/ to your .gitignore:

# GrepAI (machine-specific index)
.grepai/

Why: The index is machine-specific because:

• Contains binary embeddings
• Tied to the embedding model used
• Each machine should generate its own

Sharing Index (Not Recommended)

While you can copy the index file, it's not recommended because:

1. Must use identical embedding model
2. File paths are absolute
3. Different machines may have different code versions

Better approach: Each developer runs their own grepai watch.

Migrating to Other Backends

To PostgreSQL

1. Update config:

store:
  backend: postgres
  postgres:
    dsn: postgres://user:pass@localhost:5432/grepai

2. Re-index:

rm .grepai/index.gob
grepai watch

To Qdrant

1. Update config:

store:
  backend: qdrant
  qdrant:
    endpoint: localhost
    port: 6334

2. Re-index:

rm .grepai/index.gob
grepai watch

Common Issues

❌ Problem: Index file too large ✅ Solution: Add more ignore patterns or migrate to PostgreSQL/Qdrant

❌ Problem: Slow searches on large codebase ✅ Solution: Migrate to Qdrant for better performance

❌ Problem: Corrupted index ✅ Solution: Delete and re-index:

rm .grepai/index.gob .grepai/symbols.gob
grepai watch

❌ Problem: "Index not found" error ✅ Solution: Run grepai watch to create the index

Best Practices

1. Use for small/medium projects: Up to ~10K files
2. Add to .gitignore: Don't commit the index
3. Backup before major changes: Copy index.gob before experiments
4. Re-index after model changes: If you change embedding models
5. Monitor file size: Migrate if index exceeds 1GB

Output Format

GOB storage status:

✅ GOB Storage Configured

   Backend: GOB (local file)
   Index: .grepai/index.gob
   Size: 12.5 MB

   Contents:
   - Files: 245
   - Chunks: 1,234
   - Vectors: 1,234 × 768 dimensions

   Performance:
   - Search latency: <100ms
   - Memory usage: ~25 MB