Streamlit Project Structure and Environment Setup: A Comprehensive Guide | Struktur Proyek Streamlit dan Pengaturan Lingkungan: Panduan Komprehensif -

Enhanced Guide: Streamlit Project Structure and Environment Setup

This comprehensive guide offers an optimized approach to organizing a Streamlit application, establishing separate environments for development and production, and maintaining an efficient deployment pipeline. By adhering to these best practices, you ensure a maintainable, scalable, and portable project structure that facilitates seamless development and deployment.

Enhanced Guide: Streamlit Project Structure and Environment Setup
Table of Contents
Project Structure Overview
Dependency Management
- Production Dependencies (requirements.txt)
- Development Dependencies (requirements-dev.txt)
Environment Configuration
- Using .env Files
- Securing Environment Variables
Docker for Development and Production
Local Development Setup
Testing and Quality Assurance
Deployment to Production
- Preparing for Deployment
- Deployment with Docker
Version Control Best Practices
Advanced Features
Final Thoughts
Additional Resources

Project Structure Overview

A well-organized project structure separates concerns, enhances maintainability, and facilitates collaboration. Below is an optimized project layout for a Streamlit application:

streamlit-app/
│
├── .env                  # Default environment variables (optional)
├── .python-version       # Specifies the Python version/virtualenv for pyenv
├── README.md             # Project documentation
├── requirements.txt      # Production dependencies
├── requirements-dev.txt  # Development dependencies (e.g., testing, linting)
├── Dockerfile            # Docker configuration for app deployment
├── docker-compose.yml    # Docker Compose configuration for local development
├── src/                  # Application source code
│   ├── __init__.py
│   ├── main.py           # Main Streamlit app entry point
│   ├── components/       # Reusable UI components
│   ├── utils/            # Utility functions and helpers
│   └── config.py         # Configuration management
├── tests/                # Unit and integration tests
│   ├── __init__.py
│   ├── test_main.py
│   └── fixtures/         # Test fixtures
├── .gitignore            # Files to ignore in version control
├── pyproject.toml        # Optional, for advanced packaging and dependency management
├── configs/              # Configuration files
│   ├── .env.prod         # Production-specific environment variables
│   └── .env.dev          # Development-specific environment variables
└── scripts/              # Utility scripts (e.g., deployment, setup)
    └── setup.sh

Key Enhancements:

docker-compose.yml: Facilitates managing multi-container Docker applications, useful for services like databases.
src/ Directory: Organized into subdirectories (components/, utils/) to promote modularity.
scripts/ Directory: Stores scripts for automation tasks, enhancing reproducibility.
tests/ Directory: Structured to include fixtures and organize test modules logically.
configs/ Directory: Separates configuration files for different environments, improving clarity and security.

Dependency Management

Efficient dependency management ensures consistency across different environments and simplifies the development workflow.

Production Dependencies (`requirements.txt`)

Purpose: Lists only the dependencies necessary to run the application in a production environment.
Location: Root of the project.

Example requirements.txt:

streamlit==1.25.0
pandas==1.5.1
requests==2.31.0
python-dotenv==1.0.0

Installation Command:

pip install -r requirements.txt

Best Practices:

Pin Versions: Specify exact versions to prevent discrepancies across environments.
Minimal Dependencies: Include only what’s necessary to reduce the application’s footprint and potential security vulnerabilities.
Security Audits: Regularly review dependencies for known vulnerabilities using tools like pip-audit or safety.

Development Dependencies (`requirements-dev.txt`)

Purpose: Includes additional packages required for development tasks such as testing, linting, and formatting.
Location: Root of the project.

Example requirements-dev.txt:

pytest==7.4.0
black==24.0
flake8==6.1.0
mypy==0.991
pre-commit==3.3.3
pytest-cov==4.0.0

Installation Command:

pip install -r requirements-dev.txt

Best Practices:

Isolate Development Tools: Keep development dependencies separate to ensure production environments remain lean.
Automate Code Quality: Utilize tools like pre-commit to enforce coding standards automatically.
Version Control: Track changes to development dependencies to maintain consistency across development environments.

Environment Configuration

Managing environment-specific settings securely and efficiently is crucial for application stability and security.

Using `.env` Files

Environment variables store sensitive information such as API keys, database URLs, and configuration settings. Organizing them into .env files allows for easy management across different environments.

.env.dev: Development-specific variables.
.env.prod: Production-specific variables.

Example .env.prod:

API_KEY=your_production_api_key
DATABASE_URL=postgresql://user:password@host:port/dbname
SECRET_KEY=your_secret_key
ENV=prod

Example .env.dev:

API_KEY=your_development_api_key
DATABASE_URL=postgresql://user:password@localhost:5432/streamlit_db
SECRET_KEY=your_dev_secret_key
ENV=dev

Securing Environment Variables

Do Not Commit .env Files: Ensure .env files are listed in .gitignore to prevent sensitive data from being pushed to version control.
Use Secrets Management: For production, consider using dedicated secrets management services like AWS Secrets Manager, Azure Key Vault, or HashiCorp Vault for enhanced security.
Environment Variable Hierarchy: Allow environment variables to override .env settings for flexibility in deployment.

Example .gitignore Entry:

# Environment variables
.env
.env.dev
.env.prod

Loading Environment Variables in Python:

Install python-dotenv:
```
pip install python-dotenv
```

Configure config.py in src/:

import os
from dotenv import load_dotenv
from pathlib import Path

# Determine the current environment
ENV = os.getenv('ENV', 'dev')  # Default to 'dev' if ENV not set

# Construct the path to the appropriate .env file
env_path = Path(__file__).resolve().parent.parent / 'configs' / f'.env.{ENV}'

# Load the environment variables from the .env file
load_dotenv(dotenv_path=env_path)

# Access environment variables
API_KEY = os.getenv('API_KEY')
DATABASE_URL = os.getenv('DATABASE_URL')
SECRET_KEY = os.getenv('SECRET_KEY')

Set Environment Variable Before Running:

export ENV=prod  # or 'dev' for development

Best Practices:

Default .env: Optionally include a .env for default or fallback settings, but ensure it does not contain sensitive information.
Validation: Implement validation to ensure all required environment variables are set, using libraries like pydantic or environs.
Documentation: Maintain documentation for required environment variables to aid onboarding and maintenance.

Docker for Development and Production

Docker ensures consistency across different environments by containerizing applications along with their dependencies.

Dockerfile with Multi-Stage Builds

Multi-stage builds optimize Docker images by separating the build environment from the runtime environment, resulting in smaller and more secure production images.

Example Dockerfile:

# Stage 1: Base Builder
FROM python:3.11-slim AS base
WORKDIR /app
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    && rm -rf /var/lib/apt/lists/*

# Stage 2: Development
FROM base AS dev
ENV ENV=dev
COPY requirements-dev.txt .
RUN pip install --upgrade pip
RUN pip install --no-cache-dir -r requirements-dev.txt
COPY . .
CMD ["streamlit", "run", "src/main.py", "--server.port=8501", "--server.address=0.0.0.0"]

# Stage 3: Production
FROM base AS prod
ENV ENV=prod
COPY requirements.txt .
RUN pip install --upgrade pip
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["streamlit", "run", "src/main.py", "--server.port=8501", "--server.address=0.0.0.0"]

Key Features:

Base Stage: Common setup for both development and production, including system dependencies.
Development Stage: Includes development dependencies and source code, facilitating an interactive development environment.
Production Stage: Strips out development dependencies, resulting in a leaner image optimized for deployment.
Environment Variables: Sets ENV to differentiate between development and production within the container.

Best Practices:

Minimize Layers: Combine commands where possible to reduce the number of layers and image size.
Use .dockerignore: Exclude unnecessary files from the Docker build context to speed up builds and enhance security.
Non-Root User: Run the application as a non-root user to enhance security.

Example .dockerignore:

__pycache__/
*.pyc
*.pyo
*.pyd
*.db
.env
.env.dev
.env.prod
venv/
env/
.git/
.gitignore
Dockerfile
docker-compose.yml

Docker Compose for Local Development

Using Docker Compose simplifies managing multi-container applications and orchestrates services like databases alongside your Streamlit app.

Example docker-compose.yml:

version: '3.8'

services:
  app:
    build:
      context: .
      target: dev
    ports:
      - "8501:8501"
    volumes:
      - ./src:/app/src
      - ./configs:/app/configs
      - ./scripts:/app/scripts
    env_file:
      - configs/.env.dev
    depends_on:
      - db
    environment:
      - ENV=dev
    command: ["streamlit", "run", "src/main.py", "--server.port=8501", "--server.address=0.0.0.0"]

  db:
    image: postgres:15
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
      POSTGRES_DB: streamlit_db
    volumes:
      - db_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

volumes:
  db_data:

Benefits:

Service Orchestration: Easily manage dependencies like databases, caches, and message brokers.
Volume Mounting: Enable live code reloading by mounting source code directories.
Environment Separation: Use different .env files for services as needed.

Best Practices:

Service Dependencies: Clearly define service dependencies using depends_on to ensure proper startup order.
Network Configuration: Utilize Docker networks for secure inter-service communication.
Health Checks: Implement health checks to ensure services are running correctly before dependent services start.

Build and Run Docker Image

Build the Docker Image:
```
docker build -t streamlit-app .
```

Run the Container (Production):

docker run -d \
  -p 8501:8501 \
  --env-file=configs/.env.prod \
  --name streamlit-app-prod \
  streamlit-app:prod

Run with Docker Compose (Development):
```
docker-compose up -d
```

Best Practices:

Tagging Images: Use semantic versioning for Docker images (e.g., streamlit-app:v1.0.0) to manage deployments effectively.
Automated Builds: Integrate Docker builds into your CI/CD pipeline for automated image creation.
Resource Limits: Define resource constraints (e.g., CPU, memory) to prevent containers from exhausting host resources.

Example Docker Run Command with Resource Limits:

docker run -d \
  -p 8501:8501 \
  --env-file=configs/.env.prod \
  --name streamlit-app-prod \
  --memory="512m" \
  --cpus="1.0" \
  streamlit-app:prod

Local Development Setup

Establishing a robust local development environment ensures productivity and minimizes environment-related issues.

Set Up a Python Virtual Environment:

Using pyenv and pyenv-virtualenv:

pyenv install 3.11.0
pyenv virtualenv 3.11.0 streamlit-app-env
pyenv local streamlit-app-env

Alternative Using venv:

python3.11 -m venv venv
source venv/bin/activate

Upgrade pip:
```
pip install --upgrade pip
```
Install Development Dependencies:
```
pip install -r requirements-dev.txt
```

Set Up Environment Variables:

Create .env.dev in configs/:

API_KEY=your_development_api_key
DATABASE_URL=postgresql://user:password@localhost:5432/streamlit_db
SECRET_KEY=your_dev_secret_key
ENV=dev

Run the Application:
```
streamlit run src/main.py
```
Run Tests:
```
pytest tests/
```
Format and Lint Code:
```
black src/ tests/
flake8 src/ tests/
```
Type Checking:
```
mypy src/ tests/
```

Best Practices:

Automate Environment Setup: Use scripts (e.g., scripts/setup.sh) to automate environment setup tasks.
Editor Integration: Configure your code editor to use the virtual environment and integrate linters and formatters for seamless development.
Consistent Environments: Ensure all developers use the same Python version and dependencies to prevent environment drift.

Example scripts/setup.sh:

#!/bin/bash

# Exit on any error
set -e

# Create virtual environment
python3.11 -m venv venv
source venv/bin/activate

# Upgrade pip
pip install --upgrade pip

# Install dependencies
pip install -r requirements.txt
pip install -r requirements-dev.txt

# Set up pre-commit hooks
pre-commit install

echo "Setup complete. Activate the virtual environment with 'source venv/bin/activate'."

Testing and Quality Assurance

Ensuring code quality and reliability through testing is vital for maintaining application integrity.

Unit Testing with pytest:

Example tests/test_main.py:

import pytest
from src.main import some_function

def test_some_function():
    assert some_function(2) == 4

Test Fixtures:

Example tests/fixtures/db.py:

import pytest
from src.utils.database import create_test_db

@pytest.fixture(scope='module')
def test_db():
    db = create_test_db()
    yield db
    db.close()

Code Coverage:

Install Coverage:
```
pip install coverage
```

Run Coverage:

coverage run -m pytest
coverage report
coverage html  # Generates an HTML report

Continuous Integration (CI):
- Integrate with CI Platforms: Automate testing using platforms like GitHub Actions, GitLab CI, or Travis CI.

Best Practices:

Test-Driven Development (TDD): Write tests before implementing features to ensure functionality aligns with requirements.
Maintain High Coverage: Strive for comprehensive test coverage to catch potential issues early.
Automate Testing: Ensure tests run automatically on every commit or pull request to maintain code quality.
Mock External Services: Use mocking frameworks to simulate external dependencies, making tests faster and more reliable.
Document Tests: Clearly document what each test covers to facilitate understanding and maintenance.

Example tests/test_utils.py:

import pytest
from src.utils.helper import calculate_sum

def test_calculate_sum():
    assert calculate_sum([1, 2, 3]) == 6
    assert calculate_sum([-1, 1]) == 0
    assert calculate_sum([]) == 0

Integrating with CI:

Ensure that your CI pipeline includes steps for installing dependencies, running linting, executing tests, and checking coverage. Here’s an example using GitHub Actions (detailed in the CI/CD Integration section).

Deployment to Production

Deploying your Streamlit application to a production environment involves preparing the application, ensuring security, and managing scalability.

Preparing for Deployment

Finalize Dependencies: Ensure requirements.txt includes all necessary production dependencies without redundant packages.
Optimize Configuration: Configure the application to use production-specific settings and environment variables (.env.prod).
Security Audits: Review code for vulnerabilities and ensure sensitive information is secured.
Performance Optimization: Profile the application to identify and optimize performance bottlenecks.
Documentation: Update and finalize documentation to reflect the production setup and usage instructions.

Deployment with Docker

Build and Tag the Docker Image:

docker build -t your-dockerhub-username/streamlit-app:latest .

Push the Image to a Docker Registry:

docker push your-dockerhub-username/streamlit-app:latest

Deploy to Production Server:

Using Docker Run:

docker run -d \
  -p 80:8501 \
  --env-file=configs/.env.prod \
  --name streamlit-app-prod \
  your-dockerhub-username/streamlit-app:latest

Using Docker Compose (Production Configuration):

Create docker-compose.prod.yml:

version: '3.8'

services:
  app:
    image: your-dockerhub-username/streamlit-app:latest
    ports:
      - "80:8501"
    env_file:
      - configs/.env.prod
    restart: unless-stopped
    environment:
      - ENV=prod

Deploy with Docker Compose:

docker-compose -f docker-compose.prod.yml up -d

Set Up Reverse Proxy (Optional but Recommended):

Use Nginx or Traefik to handle SSL termination, load balancing, and routing.

Example Nginx Configuration:

server {
    listen 80;
    server_name yourdomain.com;

    location / {
        proxy_pass http://localhost:8501;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Enable SSL with Let’s Encrypt:
- Install Certbot:
```
sudo apt-get update
sudo apt-get install certbot python3-certbot-nginx
```
- Obtain and Install SSL Certificate:
```
sudo certbot --nginx -d yourdomain.com
```
- Auto-Renew Certificates: Certbot sets up a cron job by default. Verify with:
```
sudo systemctl status certbot.timer
```

Best Practices:

Use HTTPS: Secure your application with SSL/TLS certificates using services like Let’s Encrypt.
Scalability: Consider container orchestration platforms like Kubernetes for managing multiple instances and scaling.
Monitoring and Logging: Implement monitoring tools (e.g., Prometheus, Grafana) and centralized logging (e.g., ELK Stack) to track application performance and issues.
Zero Downtime Deployments: Utilize rolling updates or blue-green deployments to ensure continuous availability during updates.
Environment Variables Management: Use environment variable management solutions to securely handle sensitive information during deployment.

Example Nginx Reverse Proxy with SSL:

server {
    listen 80;
    server_name yourdomain.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    location / {
        proxy_pass http://localhost:8501;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Version Control Best Practices

Effective version control practices enhance collaboration, maintain code history, and facilitate smooth project management.

`.gitignore` File

Prevent sensitive information and unnecessary files from being committed to the repository by configuring .gitignore appropriately.

Example .gitignore:

# Environment variables
.env
.env.dev
.env.prod

# Python cache and binaries
__pycache__/
*.pyc
*.pyo
*.pyd
*.db

# Virtual environments
venv/
env/
.python-version

# Streamlit configurations
.streamlit/

# Test coverage
.coverage
htmlcov/
.pytest_cache/

# IDEs and editors
.vscode/
.idea/
*.sublime-project
*.sublime-workspace

# OS files
.DS_Store
Thumbs.db

# Docker
*.dockerignore
docker-compose.override.yml

# Logs
*.log

# Build artifacts
build/
dist/
*.egg-info/

Best Practices:

Regular Updates: Update .gitignore as new files or directories that should be excluded are introduced.
Global Git Ignore: Configure a global .gitignore for patterns common across all projects on your machine.
```
git config --global core.excludesfile ~/.gitignore_global
```
Example ~/.gitignore_global:
```
# macOS
.DS_Store

# Windows
Thumbs.db
```

Commit Guidelines

Maintain a clear and meaningful commit history to track changes effectively.

Descriptive Messages: Clearly describe what each commit does.
- Good Example: Add user authentication module with OAuth support
- Bad Example: Update stuff
Atomic Commits: Each commit should represent a single logical change.
Use Present Tense: Write commit messages in the present tense (e.g., “Fix bug” instead of “Fixed bug”).
Reference Issues: Link commits to relevant issue numbers for traceability.

Commit Message Structure:

<type>(<scope>): <subject>

<body>

<footer>

Example:

feat(auth): implement OAuth2 authentication

Added OAuth2 support using Google and GitHub providers. Refactored authentication module to handle multiple providers.

Closes #42

Types of Commits:

feat: New feature
fix: Bug fix
docs: Documentation changes
style: Code style changes (formatting, missing semi-colons, etc.)
refactor: Code refactoring without adding features or fixing bugs
test: Adding or modifying tests
chore: Changes to the build process or auxiliary tools
ci: Continuous Integration related changes
perf: Performance improvements
build: Changes that affect the build system or external dependencies

Best Practices:

Consistent Formatting: Follow a consistent commit message format, possibly enforced by tools like commitlint.
Granular Commits: Avoid large commits that encompass multiple changes; instead, break them into smaller, manageable commits.
Commit Often: Make frequent commits to capture incremental changes and facilitate easier rollbacks if necessary.

Example: Enforcing Commit Message Standards with commitlint:

Install commitlint:

npm install --save-dev @commitlint/{config-conventional,cli}

Create commitlint.config.js:

module.exports = { extends: ['@commitlint/config-conventional'] };

Add to package.json:

{
  "scripts": {
    "commitmsg": "commitlint -E HUSKY_GIT_PARAMS"
  }
}

Set Up Husky Hooks:

npx husky install
npx husky add .husky/commit-msg 'npx --no-install commitlint --edit "$1"'

Branching Strategy

Adopt a consistent branching strategy to streamline development and collaboration.

Recommended Strategy: Git Flow

Main Branches:
- main: Always production-ready.
- develop: Integration branch for features.
Supporting Branches:
- feature/*: Develop new features.
- bugfix/*: Fix bugs.
- hotfix/*: Immediate fixes on production.
- release/*: Prepare for a new production release.

Example Workflow:

Create a Feature Branch:
```
git checkout -b feature/user-auth
```
Develop and Commit Changes:

Merge Back to develop:

git checkout develop
git merge feature/user-auth
git push origin develop

Create a Release Branch:
```
git checkout -b release/v1.0.0
```

Finalize Release and Merge to main and develop:

git checkout main
git merge release/v1.0.0
git tag -a v1.0.0 -m "Release version 1.0.0"

git checkout develop
git merge release/v1.0.0
git push origin main develop --tags

Hotfixes:

Create Hotfix Branch from main:

git checkout -b hotfix/fix-crash-on-login

Fix, Commit, and Merge to main and develop:

git commit -am "fix(login): resolve crash on login with invalid credentials"
git checkout main
git merge hotfix/fix-crash-on-login
git tag -a v1.0.1 -m "Hotfix: resolve crash on login"

git checkout develop
git merge hotfix/fix-crash-on-login
git push origin main develop --tags

Benefits:

Isolation: Keeps different types of work separate.
Stability: Ensures main is always deployable.
Traceability: Facilitates tracking of features, fixes, and releases.
Parallel Development: Allows multiple features and fixes to be developed simultaneously without interference.

Alternative Strategy: GitHub Flow

For simpler projects, consider using GitHub Flow, which involves creating feature branches off main and merging via pull requests after reviews.

Key Steps:

Create a Feature Branch from main:
```
git checkout -b feature/new-dashboard
```
Develop and Commit Changes:
Open a Pull Request:
- Submit a PR to merge the feature branch into main.
- Request code reviews and address feedback.
Merge and Deploy:
- Once approved, merge the PR.
- Deploy the updated main branch to production.

Benefits:

Simplicity: Easier to manage with fewer branches.
Continuous Deployment: Facilitates rapid deployment cycles.
Collaborative Reviews: Encourages code reviews and team collaboration.

Advanced Features

Enhance your Streamlit application with advanced practices to ensure robustness, scalability, and maintainability.

CI/CD Integration

Automate the testing and deployment process to increase efficiency and reduce human error.

Example: GitHub Actions Workflow (.github/workflows/ci.yml):

name: CI Pipeline

on:
  push:
    branches:
      - main
      - develop
  pull_request:
    branches:
      - main
      - develop

jobs:
  build:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_USER: user
          POSTGRES_PASSWORD: password
          POSTGRES_DB: streamlit_db
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5          

    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          pip install -r requirements-dev.txt          

      - name: Run Lint
        run: |
          flake8 src/ tests/          

      - name: Run Tests
        env:
          DATABASE_URL: postgres://user:password@localhost:5432/streamlit_db
          ENV: test
        run: |
          pytest --cov=src tests/          

      - name: Upload Coverage to Codecov
        uses: codecov/codecov-action@v3
        with:
          token: ${{ secrets.CODECOV_TOKEN }}

Key Steps:

Checkout Code: Retrieves the repository code.
Set Up Python: Specifies the Python version.
Install Dependencies: Installs both production and development dependencies.
Linting: Ensures code adheres to style guidelines.
Testing: Executes tests and generates coverage reports.
Coverage Reporting: Integrates with services like Codecov for visibility.
Security Scans: Optionally add steps to scan for vulnerabilities using tools like bandit or safety.

Best Practices:

Parallel Jobs: Utilize parallel jobs to speed up the CI pipeline.

Caching: Implement caching for dependencies to reduce build times.

- name: Cache pip
  uses: actions/cache@v3
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }}
    restore-keys: |
      ${{ runner.os }}-pip-

Environment Variables: Securely manage secrets and environment variables using GitHub Secrets.
Notifications: Set up notifications for pipeline failures or successes to keep the team informed.
Branch Protections: Enforce CI checks on critical branches (e.g., main, develop) to maintain code quality.

Extending the CI Pipeline:

Build Docker Images: Add steps to build and push Docker images upon successful tests.
Deploy to Staging: Automatically deploy to a staging environment for further testing.
Automated Rollbacks: Implement rollback mechanisms in case deployments fail.

Example: Building and Pushing Docker Image in CI:

- name: Log in to Docker Hub
  uses: docker/login-action@v2
  with:
    username: ${{ secrets.DOCKER_USERNAME }}
    password: ${{ secrets.DOCKER_PASSWORD }}

- name: Build and Push Docker Image
  uses: docker/build-push-action@v4
  with:
    context: .
    push: true
    tags: your-dockerhub-username/streamlit-app:latest

Advanced Dependency Management

Utilize modern tools for more sophisticated dependency management, enhancing reproducibility and version control.

Using pyproject.toml with Poetry:

Initialize Poetry:
```
poetry init
```
Follow the interactive prompts to set up your project.

Add Dependencies:

poetry add streamlit pandas requests python-dotenv
poetry add --dev pytest black flake8 mypy pre-commit pytest-cov

Example pyproject.toml:

[tool.poetry]
name = "streamlit-app"
version = "0.1.0"
description = "A Streamlit application with authentication"
authors = ["Your Name <youremail@example.com>"]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.11"
streamlit = "1.25.0"
pandas = "1.5.1"
requests = "2.31.0"
python-dotenv = "^1.0"

[tool.poetry.dev-dependencies]
pytest = "7.4.0"
black = "24.0"
flake8 = "6.1.0"
mypy = "0.991"
pre-commit = "3.3.3"
pytest-cov = "4.0.0"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Install Dependencies:
```
poetry install
```
Activate Virtual Environment:
```
poetry shell
```

Benefits:

Lock Files: Ensures consistent environments across different machines.
Simplified Commands: Manages dependencies, scripts, and packaging seamlessly.
Enhanced Metadata: Provides detailed project information for better management.
Integrated Environment Management: Handles virtual environments automatically.

Additional Tools:

pipenv: Another tool for dependency management and virtual environments.
conda: Useful for managing dependencies, especially for data science projects.

Best Practices:

Lock Files Maintenance: Regularly update and commit lock files to track dependency changes.
Dependency Audits: Periodically review dependencies for updates and security patches.
Semantic Versioning: Follow semantic versioning to manage dependency versions effectively.

Logging and Monitoring

Implement logging and monitoring to track application performance, diagnose issues, and ensure reliability.

Logging with logging Module:

import logging
import sys

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler("logs/app.log"),
        logging.StreamHandler(sys.stdout)
    ]
)
logger = logging.getLogger(__name__)

logger.info("Streamlit app started")
logger.error("An error occurred")

Integrate with Monitoring Tools:
- Prometheus and Grafana: For real-time metrics and visualization.
- Sentry: For error tracking and alerting.
- Datadog: Comprehensive monitoring and analytics platform.
- New Relic: Application performance monitoring.

Example: Integrating Sentry:

Install Sentry SDK:
```
pip install sentry-sdk
```

Configure Sentry in config.py:

import sentry_sdk
from sentry_sdk.integrations.logging import LoggingIntegration

# Configure Sentry logging integration
sentry_logging = LoggingIntegration(
    level=logging.INFO,        # Capture info and above as breadcrumbs
    event_level=logging.ERROR  # Send errors as events
)

sentry_sdk.init(
    dsn="your_sentry_dsn",
    integrations=[sentry_logging],
    traces_sample_rate=1.0,  # Adjust based on your needs
    environment=ENV
)

Use Logging in Application:

logger.info("Starting the application")
try:
    # Application logic
    pass
except Exception as e:
    logger.exception("An unexpected error occurred")
    raise e

Benefits:

Proactive Issue Detection: Identify and address issues before they impact users.
Performance Insights: Monitor application performance to optimize user experience.
Comprehensive Logging: Maintain detailed logs for auditing and troubleshooting.
Alerting Mechanisms: Receive notifications for critical issues to enable rapid response.

Best Practices:

Structured Logging: Use structured logging formats (e.g., JSON) for easier parsing and analysis.
Log Rotation: Implement log rotation to manage log file sizes and retention periods.
Sensitive Data Protection: Avoid logging sensitive information to prevent data leaks.
Correlation IDs: Use correlation IDs to trace requests across different services and logs.
Dashboards: Create dashboards in monitoring tools to visualize key metrics and trends.

Example: Prometheus Metrics in Python:

from prometheus_client import start_http_server, Summary

# Create a metric to track time spent and requests made.
REQUEST_TIME = Summary('request_processing_seconds', 'Time spent processing request')

@REQUEST_TIME.time()
def process_request():
    # Your request processing logic
    pass

if __name__ == "__main__":
    start_http_server(8000)  # Prometheus will scrape metrics from this port
    while True:
        process_request()

Final Thoughts

This enhanced guide provides a structured and detailed approach to developing, managing, and deploying a Streamlit application. By implementing these best practices, you ensure that your project is:

Maintainable: Organized codebase and clear documentation facilitate easy updates and collaboration.
Scalable: Modular structure and containerization support growth and adaptability.
Secure: Proper environment management and secrets handling protect sensitive information.
Reliable: Automated testing, CI/CD pipelines, and monitoring ensure consistent performance and rapid issue resolution.

Embracing these methodologies not only streamlines your development workflow but also positions your application for long-term success and scalability. Continuously iterate on your processes, stay updated with industry best practices, and leverage community resources to enhance your project’s capabilities.

Additional Resources

Streamlit Documentation: https://docs.streamlit.io/
Docker Documentation: https://docs.docker.com/
Pytest Documentation: https://docs.pytest.org/
Poetry Documentation: https://python-poetry.org/docs/
GitHub Actions Documentation: https://docs.github.com/en/actions
Sentry Documentation: https://docs.sentry.io/
Prometheus Documentation: https://prometheus.io/docs/introduction/overview/
Grafana Documentation: https://grafana.com/docs/grafana/latest/
Git Flow Documentation: https://nvie.com/posts/a-successful-git-branching-model/
Pre-commit Documentation: https://pre-commit.com/
Codecov Documentation: https://docs.codecov.com/
Husky Documentation: https://typicode.github.io/husky/#/

Leveraging these resources will further deepen your understanding and enhance your project’s capabilities. Engage with the community through forums, contribute to open-source projects, and stay informed about the latest advancements to continuously improve your Streamlit application.

This guide is continually updated to reflect the latest best practices and tools. For any questions or contributions, feel free to reach out through the project’s repository or community channels.

Tags: Streamlit Python Docker CI/CD Development Deployment Pengembangan Deploy Python Docker CI/CD

Enhanced Guide: Streamlit Project Structure and Environment Setup

Table of Contents

Project Structure Overview

Dependency Management

Production Dependencies (requirements.txt)

Development Dependencies (requirements-dev.txt)

Environment Configuration

Using .env Files

Securing Environment Variables

Docker for Development and Production

Dockerfile with Multi-Stage Builds

Docker Compose for Local Development

Build and Run Docker Image

Local Development Setup

Testing and Quality Assurance

Deployment to Production

Preparing for Deployment

Deployment with Docker

Version Control Best Practices

.gitignore File

Commit Guidelines

Branching Strategy

Advanced Features

CI/CD Integration

Advanced Dependency Management

Logging and Monitoring

Final Thoughts

Additional Resources

Production Dependencies (`requirements.txt`)

Development Dependencies (`requirements-dev.txt`)

Using `.env` Files

`.gitignore` File