soundwave/docs/DATA_PERSISTENCE_FIX.md

Data Persistence & PWA Offline Fix

🎯 Issues Fixed

1. Database Persistence ✅

Problem: Downloaded playlists were lost on container rebuild because SQLite database was not persisted.

Solution:

• Created /app/data volume mount in Docker
• Updated Django settings to store db.sqlite3 in persistent /app/data directory
• Added data/ directory with proper .gitignore

2. Route Conflicts ✅

Problem: Playlist download routes conflicted with main playlist routes (both at root path '')

Solution:

• Moved download routes to downloads/ prefix
• Proper route ordering in backend/playlist/urls.py
• API endpoints now: /api/playlist/downloads/ instead of /api/playlist/

3. PWA Offline Playlist Caching ✅

Problem: No dedicated offline caching strategy for playlists

Solution:

• Added cachePlaylist() and removePlaylistCache() to PWA Manager
• Enhanced Service Worker with playlist-specific cache handlers
• Added playlist methods to IndexedDB storage:
  • savePlaylist()
  • getOfflinePlaylists()
  • updatePlaylistSyncStatus()
• Updated PWA Context to expose playlist caching functions

4. Security Audit ✅

Verified:

• ✅ All sensitive endpoints require authentication
• ✅ User isolation with IsOwnerOrAdmin permission
• ✅ Admin-only routes properly protected
• ✅ CORS and CSRF configured correctly
• ✅ Token authentication working

📁 Files Modified

Backend

1. docker-compose.yml - Added data and staticfiles volumes
2. backend/config/settings.py - Database path now /app/data/db.sqlite3
3. backend/playlist/urls.py - Fixed route conflicts

Frontend (PWA)

4. frontend/src/utils/offlineStorage.ts - Added playlist offline methods
5. frontend/src/utils/pwa.ts - Added cachePlaylist() and removePlaylistCache()
6. frontend/src/context/PWAContext.tsx - Exposed new playlist functions
7. frontend/public/service-worker.js - Added playlist cache handlers

Infrastructure

8. data/.gitignore - Created to exclude database from git

🚀 Migration Steps

For Existing Deployments

1. Stop containers:

   docker-compose down
   

2. Create data directory (if not exists):

   mkdir -p data
   

3. Migrate existing database (if you have one):

   # If you have an existing db.sqlite3 in backend/
   mv backend/db.sqlite3 data/db.sqlite3
   

4. Rebuild and restart:

   docker-compose build
   docker-compose up -d
   

5. Verify persistence:

   # Check database exists
   ls -lh data/db.sqlite3
   
   # Check it persists after rebuild
   docker-compose down
   docker-compose up -d
   ls -lh data/db.sqlite3  # Should still exist
   

🎨 PWA Offline Playlist Usage

In Your Components

import { usePWA } from '../context/PWAContext';
import { offlineStorage } from '../utils/offlineStorage';

function PlaylistComponent() {
  const { cachePlaylist, removePlaylistCache, isOnline } = usePWA();

  // Download playlist for offline use
  const downloadPlaylist = async (playlist) => {
    // 1. Cache audio files via Service Worker
    const audioUrls = playlist.items.map(item => item.audio_url);
    const cached = await cachePlaylist(playlist.id, audioUrls);
    
    // 2. Save metadata to IndexedDB
    if (cached) {
      await offlineStorage.savePlaylist({
        id: playlist.id,
        title: playlist.title,
        items: playlist.items,
        offline: true,
      });
    }
  };

  // Remove offline playlist
  const removeOfflinePlaylist = async (playlist) => {
    const audioUrls = playlist.items.map(item => item.audio_url);
    await removePlaylistCache(playlist.id, audioUrls);
    await offlineStorage.removePlaylist(playlist.id);
  };

  // Get offline playlists
  const loadOfflinePlaylists = async () => {
    const playlists = await offlineStorage.getOfflinePlaylists();
    return playlists;
  };
}

📊 Data Persistence Structure

soundwave/
├── audio/              # Persistent: Downloaded audio files
├── cache/              # Persistent: Application cache
├── data/               # ✨ NEW: Persistent database storage
│   ├── db.sqlite3      # Main database (persists between rebuilds)
│   └── .gitignore      # Excludes database from git
├── es/                 # Persistent: Elasticsearch data
├── redis/              # Persistent: Redis data
└── backend/
    └── staticfiles/    # Persistent: Collected static files

🔒 Security Verification

All endpoints verified for proper authentication and authorization:

Public Endpoints (No Auth Required)

• /api/user/login/ - User login
• /api/user/register/ - User registration

Authenticated Endpoints

• /api/playlist/* - User playlists (owner isolation)
• /api/playlist/downloads/* - Download management (owner isolation)
• /api/audio/* - Audio files (user-scoped)
• /api/channel/* - Channels (admin write, all read)

Admin-Only Endpoints

• /api/download/* - Download queue management
• /api/task/* - Task management
• /api/appsettings/* - System settings
• /admin/* - Django admin

Permission Classes Used

• IsAuthenticated - Must be logged in
• IsOwnerOrAdmin - Owner or admin access
• AdminOnly - Admin/superuser only
• AdminWriteOnly - Admin write, all read

🧪 Testing Checklist

• Database persists after docker-compose down && docker-compose up
• Downloaded playlists remain after container rebuild
• Audio files persist in /audio volume
• Static files persist in /staticfiles volume
• PWA offline playlist caching works
• Route conflicts resolved
• Security permissions verified
• Multi-user isolation working
• Full end-to-end test with rebuild

🎯 API Endpoint Changes

Before

/api/playlist/                    # List/Create playlists
/api/playlist/<id>/               # Playlist detail
/api/playlist/                    # ❌ CONFLICT: Downloads viewset

After

/api/playlist/                    # List/Create playlists
/api/playlist/<id>/               # Playlist detail
/api/playlist/downloads/          # ✅ Downloads viewset (no conflict)
/api/playlist/downloads/<id>/     # Download detail
/api/playlist/downloads/active/   # Active downloads
/api/playlist/downloads/completed/# Completed downloads

💡 Best Practices

1. Always use volumes for persistent data

   • Database files
   • User uploads
   • Application cache
   • Static files

2. Separate data from code

   • Code in container (rebuilt)
   • Data in volumes (persisted)

3. PWA offline strategy

   • Cache API responses for metadata
   • Cache audio files for playback
   • Store state in IndexedDB
   • Sync when online

4. Security layers

   • Authentication (token-based)
   • Authorization (permission classes)
   • User isolation (owner field checks)
   • Admin protection (admin-only views)

📝 Environment Variables

Optional configuration in .env or docker-compose:

# Data directory (default: /app/data)
DATA_DIR=/app/data

# Media directory (default: /app/audio)
MEDIA_ROOT=/app/audio

🔄 Future Enhancements

1. Database Backup

   • Add automated SQLite backup script
   • Volume snapshot strategy

2. Cache Management

   • PWA cache size limits
   • Auto-cleanup old cached playlists

3. Sync Strategy

   • Background sync for offline changes
   • Conflict resolution

4. Analytics

   • Track offline usage
   • Cache hit/miss ratios

❓ Troubleshooting

Database not persisting

# Check volume mount
docker inspect soundwave | grep -A 5 Mounts

# Verify data directory
docker exec soundwave ls -lh /app/data/

# Check database location
docker exec soundwave python manage.py shell -c "from django.conf import settings; print(settings.DATABASES['default']['NAME'])"

PWA cache not working

# Check service worker registration
# Open browser DevTools -> Application -> Service Workers

# Clear all caches
# DevTools -> Application -> Storage -> Clear site data

# Re-register service worker
# Navigate to app and check console

Route conflicts

# Test endpoints
curl http://localhost:8889/api/playlist/
curl http://localhost:8889/api/playlist/downloads/

🎉 Result

✅ Playlists now persist between container rebuilds ✅ PWA offline support for playlists ✅ No route conflicts ✅ Security verified ✅ All users (admin & managed) working