Skip to main content

Repository Structure and Adding Features

This document explains the project structure and how to add new features to Docusaurus without polluting the documentation or breaking maintainability.

Project Structure Overview​

TechStackMain/
├── tools/                          # Source of truth for tool docs
│   ├── *.md                        # One file per tool with frontmatter metadata
│   └── README.md                   # Tools directory guide
├── scripts/                        # Workspace automation scripts
│   ├── build-downloadables.cjs     # Builds downloadable zips for the docs site
│   ├── sync-tools/                 # TwentyCRM sync system
│   └── resources/                  # Scripts and data files, organised by tool
│       ├── gtm/                    # GTM analyzer script + template container JSONs
│       ├── webflow/                # Webflow backup/export/sync scripts + JS snippets
│       ├── n8n/                    # n8n fetch scripts
│       ├── notifuse/               # Notifuse fetch scripts
│       ├── twilio/                 # Twilio fetch scripts
│       ├── hetzner/                # Hetzner check scripts
│       └── cloudflare/             # Cloudflare fetch scripts
│           # Each folder: README.md with frontmatter → syncs to Scripts section
│           # Extra *.md files with frontmatter title: → sync as sub-pages
├── docusaurus/                     # Docusaurus site
│   ├── config/                     # Configuration files
│   │   └── sections.ts             # Top-level section definitions (sidebar order)
│   ├── docs/                       # Documentation pages
│   │   ├── tools/                  # ⚠️ Auto-generated by sync-tools.cjs (DO NOT EDIT)
│   │   │   ├── website/            # Website category
│   │   │   ├── self-hosting/       # Self-hosting category
│   │   │   ├── operational-tools/  # Operational tools category
│   │   │   └── ...
│   │   ├── scripts/                # ⚠️ Auto-generated by sync-scripts.cjs (DO NOT EDIT)
│   │   │   ├── index.md            # Scripts section index
│   │   │   ├── gtm/                # GTM Analyzer docs
│   │   │   └── webflow/            # Webflow scripts docs (index + sub-pages)
│   │   ├── Guides/                 # Operational guides (edit directly)
│   │   ├── docusaurus/             # Docusaurus-specific docs
│   │   │   ├── features/           # Feature documentation
│   │   │   ├── index.md            # Docusaurus overview
│   │   │   ├── repo-structure.md   # This document
│   │   │   └── todo.md             # To-do list
│   │   └── *.md                    # Top-level pages (architecture, intro, etc.)
│   ├── src/                        # Source code
│   │   ├── theme/MDXComponents/    # Custom MDX components
│   │   │   ├── FAQStructuredData/
│   │   │   ├── ToolMetadata/
│   │   │   ├── CategorySummary/
│   │   │   ├── ToolsOverview/
│   │   │   └── SearchBar/
│   │   └── css/                    # Custom styles
│   ├── agents/                     # Custom Cursor agents (see /docs/agents/)
│   ├── scripts/                    # Build scripts
│   │   ├── sync-tools.cjs          # Syncs tools/*.md → docs/tools/ (categorised)
│   │   ├── sync-scripts.cjs        # Syncs scripts/resources/*/README.md → docs/scripts/
│   │   ├── sync-todo.cjs           # Syncs TODO.md → Docusaurus
│   │   ├── validate-mdx.cjs        # MDX validator (runs pre-build)
│   │   ├── check-links.cjs         # Broken link checker
│   │   ├── generate-llms-txt.cjs   # Generates /llms.txt with Gemini-enhanced summaries
│   │   ├── index-algolia.cjs       # Algolia indexing - keyword index + Ask AI markdown index
│   │   └── __tests__/              # Unit tests for all build scripts (run via npm test)
│   ├── static/downloads/           # Downloadable zip files (generated by scripts/build-downloadables.cjs)
│   │                               # ⚠️ Do not edit - regenerated on every prebuild with secret scanning
│   ├── sidebars.ts                 # Sidebar config (generated from sections.ts)
│   └── docusaurus.config.ts        # Docusaurus configuration
├── architecture.md                 # Tech stack architecture (workspace root copy)
└── README.md                       # Workspace overview and quick-start

Key Principles​

1. Separation of Concerns​

Source Files (tools/*.md):

  • Edit these directly
  • Include frontmatter with metadata (category, costs, renewals, account types)
  • Auto-synced to categorised Docusaurus structure on build
  • One file per tool
  • Metadata synced from TwentyCRM CSV export

Generated Files - Tools (docusaurus/docs/tools/):

  • ⚠️ NEVER EDIT - Auto-generated, overwritten on every build by sync-tools.cjs
  • Organised into category subdirectories (website/, self-hosting/, etc.)
  • Include Docusaurus frontmatter and ToolMetadata component
  • Category index pages auto-generated with CategorySummary component

Generated Files - Scripts (docusaurus/docs/scripts/):

  • ⚠️ NEVER EDIT - Auto-generated, overwritten on every build by sync-scripts.cjs
  • Source: scripts/resources/*/README.md with frontmatter title:
  • Extra .md files alongside the README (with frontmatter title:) become sub-pages
  • The source scripts/resources/<tool>/ folder is the right place to edit

Custom Pages (docusaurus/docs/*.md):

  • Standalone documentation pages
  • Not auto-synced
  • Manual frontmatter required
  • Examples: intro.md, architecture.md, tools.md

Feature Documentation (docusaurus/docs/docusaurus/features/):

  • Reusable components and features
  • Setup guides for new repos
  • Not tool-specific

2. Adding New Features​

When adding a new feature to Docusaurus (like the FAQ component):

  1. Create the component in docusaurus/src/theme/MDXComponents/
  2. Document the feature in docusaurus/docs/docusaurus/features/
  3. Update setup guide if it's a reusable feature
  4. Add to to-do list if it needs future work

Example: Adding FAQ Component

docusaurus/
├── src/theme/MDXComponents/
│   └── FAQStructuredData/          # Component code
│       ├── index.js
│       └── types.d.ts
└── docs/docusaurus/features/
    └── faq-structured-data.md      # Feature documentation

3. Adding Tool Documentation​

When documenting a new tool:

  1. Create tools/<tool-name>.md with frontmatter: 
    ---
    title: Tool Name
    category: WEBSITE
    categoryDisplay: Website
    monthlyCost: 25
    monthlyCurrency: GBP
    annualCost: 0
    annualCurrency: GBP
    renewalDate: null
    accountType:
      - INTERNAL
      - CLIENT_ACCESS
    accessLink: https://tool.com/dashboard
    moreInfo: Brief description
    ---
  2. Commit - Auto-sync runs on build, organises by category
  3. Verify - Check docusaurus/docs/tools/<category>/<tool-name>.md was created
  4. Never edit the generated file directly

See TOOL_CATEGORIZATION.md for full details on metadata fields and CSV integration.

4. Adding Custom Pages​

For standalone documentation pages:

  1. Create docusaurus/docs/<page-name>.md
  2. Add frontmatter: 
    ---
    id: page-name
    title: Page Title
    sidebar_position: 5
    ---
  3. Update sections config in docusaurus/config/sections.ts for top-level sections
  4. Update sidebar in docusaurus/sidebars.ts if needed (auto-generated from sections.ts)

5. Adding New Documentation Sections​

To add top-level sections like "Prompts", "Processes", or "Projects":

  1. Edit docusaurus/config/sections.ts
    • Add a new section entry or change status from 'planned' to 'active'
    • Set appropriate position (lower numbers appear first in sidebar)
    • Set landingPage to point to your index page (e.g., 'Projects/projects')
    • Set autogenerate: true to auto-generate sidebar items
  2. Create directory: docusaurus/docs/<section-name>/
  3. Create index page: docusaurus/docs/<section-name>/index.md
    • Add frontmatter with id matching the landingPage (without path)
    • Set sidebar_position: 0 to appear first in the section
  4. Create category config: docusaurus/docs/<section-name>/_category_.json
    • Use "type": "doc", "id": "<section-id>" to link to manual index
    • Or use "type": "generated-index" for auto-generated index
  5. Add content files to the directory
  6. Build - Sidebar automatically updates from sections config

Example from sections.ts:

{
  id: 'projects',
  label: 'Projects',
  dirName: 'Projects',
  status: 'active',
  position: 5,
  description: 'Internal projects and automation workflows',
  landingPage: 'Projects/projects',  // Points to docs/Projects/index.md with id: projects
  autogenerate: true,
}

Adding Sub-Projects to a Section:

For sections like "Projects" that contain multiple sub-projects:

  1. Create sub-project folder: docusaurus/docs/<section-name>/<project-name>/
  2. Create sub-project index: docusaurus/docs/<section-name>/<project-name>/index.md
    • Add frontmatter with unique id and sidebar_position (higher than section index)
  3. Create sub-category config: docusaurus/docs/<section-name>/<project-name>/_category_.json
    • Set "position" to match the sidebar_position in the index.md

Example structure for Projects section:

docusaurus/docs/Projects/
├── _category_.json                    # Section category config
├── index.md                            # Section landing page (id: projects, sidebar_position: 0)
└── cold-outreach-automation-flow/      # Sub-project
    ├── _category_.json                 # Sub-project category config (position: 2)
    └── index.md                        # Sub-project page (id: cold-outreach-automation-flow, sidebar_position: 2)

Feature Development Workflow​

Step 1: Plan the Feature​

  • Document the use case
  • Identify if it's reusable or one-off
  • Determine where it belongs (component, page, guide)

Step 2: Implement​

  • Create component/page in appropriate location
  • Follow existing patterns and conventions
  • Test locally with npm start

Step 3: Document​

  • Create feature documentation in docusaurus/docs/docusaurus/features/
  • Update setup guide if reusable
  • Add examples and usage instructions

Step 4: Integrate​

  • Update sidebar if needed
  • Add to relevant documentation
  • Test in production build

Step 5: Maintain​

  • Add to to-do list if future work needed
  • Document any gotchas or limitations
  • Keep examples up to date

Avoiding Documentation Pollution​

✅ Good Practices​

  • Tool docs go in tools/ (auto-synced to Tools section)
  • Script docs go in scripts/resources/<tool>/README.md (auto-synced to Scripts section); sub-pages go alongside the README as .md files with frontmatter title:
  • Guides go in docusaurus/docs/Guides/
  • Feature docs go in docusaurus/docs/docusaurus/features/
  • Architecture goes in top-level architecture.md or docusaurus/docs/architecture.md

❌ Avoid​

  • Don't bury reusable scripts in Guides - they belong in scripts/resources/ and will surface in the Scripts section automatically
  • Don't mix feature docs with tool docs
  • Don't create one-off pages for reusable features
  • Don't edit generated files (docusaurus/docs/tools/ or docusaurus/docs/scripts/)
  • Don't duplicate documentation across locations

Maintaining Project Structure​

Regular Tasks​

  1. Review generated files - Ensure auto-sync is working
  2. Check for duplicates - Search before creating new docs
  3. Update to-do list - Track planned features and tasks
  4. Clean up - Remove deprecated features and docs

Automation Opportunities​

  • Auto-sync (Tools) - Already automated ✅
  • Auto-sync (Scripts) - Already automated ✅
  • Script zip downloads - Already automated ✅ (with credential scanning)
  • MDX validation - Already automated ✅
  • Broken link checking - Already automated ✅
  • Build and deploy - Already automated ✅
  • Algolia indexing - Already automated ✅ (keyword + Ask AI markdown index)
  • llms.txt generation - Already automated ✅ (Gemini-enhanced, IP-scrubbed)

Future Automation Ideas​

  • Auto-update last verified dates on tool/script pages
  • Auto-format Markdown files

File Naming Conventions​

  • Tool docs: tools/<tool-name>.md (lowercase, hyphens)
  • Feature docs: docusaurus/docs/docusaurus/features/<feature-name>.md
  • Guides: docusaurus/docs/Guides/<guide-name>/
  • Components: docusaurus/src/theme/MDXComponents/<ComponentName>/

Version Control Strategy​

  • Main branch - Production-ready documentation
  • Feature branches - For new features or major updates
  • Pull requests - Review before merging
  • Auto-deploy - Cloudflare Pages builds on push to main