Motivation

Problem statement

Sphinx historically uses a dynamic configuration approach via the conf.py file. While this provides great flexibility, it poses a number of challenges

• Cannot be read outside of Python, for tools such as ubCode / ubc that require the configuration.

• Bigger projects tend to modularize their conf.py files. This means a simple AST readout of the conf.py is not possible as imports have to be followed.

• Quite often additional Python dependencies are required, to import conf.py. Those must be installed into a virtual environment before importing works. This also means you can’t just import the conf.py in a system Python interpreter.

• Using Python as a configuration language can sometimes lead to unintended complexity. For example, some projects may include time-consuming operations like network requests during configuration loading, which can slow down the documentation build process. A declarative configuration format helps encourage simpler, more maintainable practices.

• The final configuration is not directly visible to end users. They have to read through the entire conf.py (and its imports) to understand the final configuration.

• It’s often just unnecessary complexity for simple configuration needs.

Use cases

needs-config-writer aims to solve these issues by providing a way to export Sphinx-Needs configuration from conf.py into a static ubproject.toml file. This can be a one-time activity or can be done regularly to keep the file in sync.

This extension can be used in multiple use case.

Migration

Migrate configuration from conf.py to ubproject.toml. Just add this extension and copy over the configuration from _build/html/ubproject.toml to your own ubproject.toml. This file should commonly be located where conf.py is. Then use needs_from_toml to load configuration from the file for subsequent Sphinx-Needs builds.

Keep in sync

Keep ubproject.toml in sync with conf.py. This is useful if you want to keep using conf.py as main configuration source, but also want to provide declarative configuration ubproject.toml for tools like ubCode / ubc.

If the setting needscfg_outpath is set to ubproject.toml, the file will be written to the location of conf.py, keeping both files in sync. ubproject.toml should be version controlled, so tools like ubCode / ubc can access it without running any build system step. That leads to happy developers.

Complex setups

If you have a distributed documentation build setup that

• has multiple repositories involved in building documentation,

• stores configuration at a central place, maybe in a different repo,

• might require building dependencies such as needs.json files before building the docs or

• builds dependencies in sandboxes, with remote caching or remote execution,

then this use case might be worth looking into. Build system such as Bazel will work with this.

Multiple problems have to be solved in such a setup:

• How to provide Sphinx-Needs configuration to the build system?

• How to provide Sphinx-Needs configuration to ubCode / ubc?

• How to know which files go into a documentation build?

Generally the knowledge is hidden somewhere in build system files, and it needs to be made available to all tools involved, both build system and ubCode / ubc.

A full example can be found in the tests/complex_setups folder of this repository.

Imagine this file hierarchy:

.
├── generated_config
│   ├── schemas.json
│   └── ubproject.toml
├── generated_deps
│   └── needs_test.json
└── project
    ├── conf.py
    ├── index.rst
    └── ubproject.toml

The generated_* folders are generated by the build system (e.g., Bazel) before building the docs. They contain generated configuration and dependencies such as needs.json files.

Whenever a local or shared configuration changes, the build system runs Sphinx with needs-config-writer enabled, generating the ubproject.toml file in project/ubproject.toml. It’s a pure product of the assembled configuration, and can directly be used with tools like ubCode / ubc.

generated_config/schemas.json

{
  "$defs": {
    "type-feat": {
      "properties": {
        "type": { "const": "feat" }
      }
    },
    "safe-need": {
      "properties": {
        "asil": { "enum": ["A", "B", "C", "D"] }
      },
      "required": ["asil"]
    }
  },
  "schemas": [
    {
      "id": "feat",
      "select": {
        "$ref": "#/$defs/type-feat"
      },
      "validate": {
        "local": {
          "$ref": "#/$defs/safe-need"
        }
      }
    }
  ]
}

generated_config/ubproject.toml

"$schema" = "https://ubcode.useblocks.com/ubproject.schema.json"

[needs]
id_required = true
id_regex = "^[A-Z0-9_]{3,}"
build_json = true
schema_debug_active = true

[[needs.extra_options]]
name = "efforts"
description = "FTE days"
schema.type = "integer"
schema.minimum = 0

[[needs.extra_options]]
name = "priority"
description = "Priority level, 1-5 where 1 is highest and 5 is lowest"
schema.type = "integer"
schema.minimum = 1
schema.maximum = 5

[[needs.extra_options]]
name = "asil"
description = "Automotive Safety Integrity Level"
schema.type = "string"
schema.enum = ["QM", "A", "B", "C", "D"]

[[needs.types]]
directive = "feat"
title = "Feature"
prefix = "FEAT_"

[[needs.types]]
directive = "req"
title = "Requirement"
prefix = "REQ_"

[[needs.types]]
directive = "impl"
title = "Implementation"
prefix = "IMPL_"

generated_deps/needs_test.json

{
    "created": "2021-05-11T13:54:22.331741",
    "current_version": "1.0",
    "project": "needs test docs",
    "versions": {
        "1.0": {
            "created": "2021-05-11T13:54:22.331724",
            "filters": {},
            "filters_amount": 0,
            "needs": {
                "TEST_01": {
                    "avatar": "",
                    "closed_at": "",
                    "completion": "",
                    "created_at": "",
                    "description": "TEST_01",
                    "docname": "index",
                    "duration": "",
                    "external_css": "external_link",
                    "external_url": "file:///home/daniel/workspace/sphinx/sphinxcontrib-needs/tests/doc_test/external_doc/__error__#TEST_01",
                    "full_title": "TEST_01 DESCRIPTION",
                    "id": "TEST_01",
                    "id_complete": "TEST_01",
                    "id_parent": "TEST_01",
                    "id_prefix": "",
                    "is_need": true,
                    "is_part": false,
                    "layout": null,
                    "links": [],
                    "max_amount": "",
                    "max_content_lines": "",
                    "parent_need": null,
                    "parent_needs": [],
                    "parent_needs_back": [],
                    "parts": {},
                    "post_template": null,
                    "pre_template": null,
                    "query": "",
                    "section_name": "",
                    "sections": [],
                    "service": "",
                    "signature": "",
                    "specific": "",
                    "status": null,
                    "style": null,
                    "tags": [],
                    "template": null,
                    "title": "TEST_01 DESCRIPTION",
                    "type": "impl",
                    "type_name": "Implementation",
                    "updated_at": "",
                    "url": "",
                    "user": ""
                },
                "TEST_02": {
                    "avatar": "",
                    "closed_at": "",
                    "completion": "",
                    "created_at": "",
                    "description": "TEST_02",
                    "docname": "index",
                    "duration": "",
                    "external_css": "external_link",
                    "external_url": "file:///home/daniel/workspace/sphinx/sphinxcontrib-needs/tests/doc_test/external_doc/__error__#TEST_02",
                    "full_title": "TEST_02 DESCRIPTION",
                    "id": "TEST_02",
                    "id_complete": "TEST_02",
                    "id_parent": "TEST_02",
                    "id_prefix": "",
                    "is_external": true,
                    "is_need": true,
                    "is_part": false,
                    "layout": null,
                    "links": [
                        "TEST_01"
                    ],
                    "max_amount": "",
                    "max_content_lines": "",
                    "parent_need": null,
                    "parent_needs": [],
                    "parent_needs_back": [],
                    "parts": {},
                    "post_template": null,
                    "pre_template": null,
                    "query": "",
                    "section_name": "",
                    "sections": [],
                    "service": "",
                    "signature": "",
                    "specific": "",
                    "status": "open",
                    "style": null,
                    "tags": [
                        "test_02",
                        "test"
                    ],
                    "template": null,
                    "title": "TEST_02 DESCRIPTION",
                    "type": "req",
                    "type_name": "Requirement",
                    "updated_at": "",
                    "url": "",
                    "user": ""
                },
                "TEST_03": {
                    "avatar": "",
                    "closed_at": "",
                    "completion": "",
                    "created_at": "",
                    "description": "AAA",
                    "docname": "subpage_a/subpage_b/subpage",
                    "duration": "",
                    "external_css": "external_link",
                    "external_url": "file:///home/daniel/workspace/sphinx/sphinxcontrib-needs/tests/doc_test/external_doc/__error__#TEST_03",
                    "full_title": "AAA",
                    "id": "TEST_03",
                    "id_complete": "TEST_03",
                    "id_parent": "TEST_03",
                    "id_prefix": "",
                    "is_external": true,
                    "is_need": true,
                    "is_part": false,
                    "layout": null,
                    "links": [],
                    "max_amount": "",
                    "max_content_lines": "",
                    "parent_need": null,
                    "parent_needs": [],
                    "parent_needs_back": [],
                    "parts": {},
                    "post_template": null,
                    "pre_template": null,
                    "query": "",
                    "section_name": "",
                    "sections": [],
                    "service": "",
                    "signature": "",
                    "specific": "",
                    "status": "open",
                    "style": null,
                    "tags": [],
                    "template": null,
                    "title": "AAA",
                    "type": "req",
                    "type_name": "Requirement",
                    "updated_at": "",
                    "url": "",
                    "user": ""
                }
            },
            "needs_amount": 6
        }
    }
}

project/index.rst

basic test
==========

.. feat:: Feature
   :id: FEAT
   :asil: A
   :links: IMP_TEST_02

.. needimport:: dep1
   :id_prefix: IMP_

project/conf.py

project = "Basic Project"

extensions = ["sphinx_needs", "needs_config_writer"]

needs_from_toml = "../generated_config/ubproject.toml"
needs_schema_definitions_from_json = "../generated_config/schemas.json"
needs_import_keys = {"dep1": "../generated_deps/needs_test.json"}

needscfg_outpath = "ubproject.toml"
"""Write to this directory."""

needscfg_overwrite = True
"""Any changes to the shared/local configuration shall update the generated config file."""

needscfg_write_all = True
"""Write full config, so the final configuration is visible in one file."""

needscfg_warn_on_diff = True
"""Be sure to update this - running Sphinx with -W will fail the CI, that's wanted."""

project/ubproject.toml

# This file is auto-generated by needs-config-writer.
# It is a duplicate of shared and local configs to make tools like ubCode / ubc work.
# Do not manually modify it - changes will be overwritten.

[needs]
allow_unsafe_filters = false
build_json = true
build_json_per_id = false
build_json_per_id_path = "needs_id"
build_needumls = ""
builder_filter = "is_external==False"
completion_option = "completion"
constraints_failed_color = ""
css = "modern.css"
debug_filters = false
debug_measurement = false
default_layout = "clean"
diagram_template = "\n{%- if is_need -%}\n<size:12>{{type_name}}</size>\\n**{{title|wordwrap(15, wrapstring='**\\\\n**')}}**\\n<size:10>{{id}}</size>\n{%- else -%}\n<size:12>{{type_name}} (part)</size>\\n**{{content|wordwrap(15, wrapstring='**\\\\n**')}}**\\n<size:10>{{id_parent}}.**{{id}}**</size>\n{%- endif -%}\n"
duration_option = "duration"
external_needs = []
flow_engine = "plantuml"
flow_link_types = [
    "links",
]
flow_show_links = false
functions = []
id_from_title = false
id_length = 5
id_regex = "^[A-Z0-9_]{3,}"
id_required = true
include_needs = true
json_exclude_fields = [
    "collapse",
    "hide",
    "id_complete",
    "id_parent",
    "is_need",
    "is_part",
    "lineno_content",
    "type_color",
    "type_prefix",
    "type_style",
]
json_remove_defaults = false
max_title_length = -1
needextend_strict = false
part_prefix = "→ "
permalink_data = "needs.json"
permalink_file = "permalink.html"
report_dead_links = true
report_template = ""
reproducible_json = false
role_need_max_title_length = 30
role_need_template = "{title} ({id})"
schema_debug_active = true
schema_debug_ignore = [
    "extra_option_success",
    "extra_link_success",
    "select_success",
    "select_fail",
    "local_success",
    "network_local_success",
]
schema_debug_path = "schema_debug"
schema_severity = "info"
service_all_data = false
show_link_id = true
show_link_title = false
show_link_type = false
statuses = []
table_classes = []
table_columns = "ID;TITLE;STATUS;TYPE;OUTGOING;TAGS"
table_style = "DATATABLES"
tags = []
template_folder = "needs_templates/"
title_from_content = false
title_optional = false
types = [
    { directive = "feat", prefix = "FEAT_", title = "Feature" },
    { directive = "impl", prefix = "IMPL_", title = "Implementation" },
    { directive = "req", prefix = "REQ_", title = "Requirement" },
]
variant_options = []
warnings_always_warn = false

[needs.constraint_failed_options]

[needs.constraints]

[needs.extensions]

[[needs.extra_links]]
color = "#000000"
copy = false
incoming = "links incoming"
option = "links"
outgoing = "links outgoing"

[[needs.extra_links]]
color = "#333333"
copy = false
incoming = "child needs"
option = "parent_needs"
outgoing = "parent needs"

[[needs.extra_options]]
description = "Automotive Safety Integrity Level"
name = "asil"

[needs.extra_options.schema]
enum = [
    "QM",
    "A",
    "B",
    "C",
    "D",
]
type = "string"

[[needs.extra_options]]
description = "FTE days"
name = "efforts"

[needs.extra_options.schema]
minimum = 0
type = "integer"

[[needs.extra_options]]
description = "Priority level, 1-5 where 1 is highest and 5 is lowest"
name = "priority"

[needs.extra_options.schema]
maximum = 5
minimum = 1
type = "integer"

[needs.filter_data]

[needs.flow_configs]
cplant = "\n    ' CPLANT by AOKI (https://github.com/aoki/cplant)\n    !define BLACK   #363D5D\n    !define RED     #F6363F\n    !define PINK    #F6216E\n    !define MAGENTA #A54FBD\n    !define GREEN   #37A77C\n    !define YELLOW  #F97A00\n    !define BLUE    #1E98F2\n    !define CYAN    #25AFCA\n    !define WHITE   #FEF2DC\n\n    ' Base Setting\n    skinparam Shadowing false\n    skinparam BackgroundColor transparent\n    skinparam ComponentStyle uml2\n    skinparam Default {\n      FontName  'Hiragino Sans'\n      FontColor BLACK\n      FontSize  10\n      FontStyle plain\n    }\n\n    skinparam Sequence {\n      ArrowThickness 1\n      ArrowColor RED\n      ActorBorderThickness 1\n      LifeLineBorderColor GREEN\n      ParticipantBorderThickness 0\n    }\n    skinparam Participant {\n      BackgroundColor BLACK\n      BorderColor BLACK\n      FontColor #FFFFFF\n    }\n\n    skinparam Actor {\n      BackgroundColor BLACK\n      BorderColor BLACK\n    }\n    "
handwritten = "\n        skinparam handwritten true\n    "
lefttoright = "\n        left to right direction\n    "
mixing = "\n        allowmixing\n    "
monochrome = "\n        skinparam monochrome true\n    "
tne = "\n    ' Based on \"Tomorrow night eighties\" color theme (see https://github.com/chriskempson/tomorrow-theme)\n    ' Provided by gabrieljoelc (https://github.com/gabrieljoelc/plantuml-themes)\n    !define Background   #2d2d2d\n    !define CurrentLine  #393939\n    !define Selection    #515151\n    !define Foregound    #cccccc\n    !define Comment      #999999\n    !define Red          #f2777a\n    !define Orange       #f99157\n    !define Yellow       #ffcc66\n    !define Green        #99cc99\n    !define Aqua         #66cccc\n    !define Blue         #6699cc\n    !define Purple       #cc99cc\n\n    skinparam Shadowing false\n    skinparam backgroundColor #2d2d2d\n    skinparam Arrow {\n      Color Foregound\n      FontColor Foregound\n      FontStyle Bold\n    }\n    skinparam Default {\n      FontName Menlo\n      FontColor #fdfdfd\n    }\n    skinparam package {\n      FontColor Purple\n      BackgroundColor CurrentLine\n      BorderColor Selection\n    }\n    skinparam node {\n      FontColor Yellow\n      BackgroundColor CurrentLine\n      BorderColor Selection\n    }\n    skinparam component {\n      BackgroundColor Selection\n      BorderColor Blue\n      FontColor Blue\n      Style uml2\n    }\n    skinparam database {\n      BackgroundColor CurrentLine\n      BorderColor Selection\n      FontColor Orange\n    }\n\n    skinparam cloud {\n      BackgroundColor CurrentLine\n      BorderColor Selection\n    }\n\n    skinparam interface {\n      BackgroundColor CurrentLine\n      BorderColor Selection\n      FontColor Green\n    }\n    "
toptobottom = "\n        top to bottom direction\n    "
transparent = "\n    skinparam backgroundcolor transparent\n    "

[needs.global_options]

[needs.graphviz_styles.default.edge]
minlen = "2"

[needs.graphviz_styles.default.node]
margin = "0.21,0.11"

[needs.graphviz_styles.lefttoright.graph]
rankdir = "LR"

[needs.graphviz_styles.toptobottom.graph]
rankdir = "TB"

[needs.graphviz_styles.transparent.graph]
bgcolor = "transparent"

[needs.import_keys]
dep1 = "../generated_deps/needs_test.json"

[needs.layouts.clean]
grid = "simple"

[needs.layouts.clean.layout]
head = [
    "<<meta(\"type_name\")>>: **<<meta(\"title\")>>** <<meta_id()>>  <<collapse_button(\"meta\", collapsed=\"icon:arrow-down-circle\", visible=\"icon:arrow-right-circle\", initial=False)>> ",
]
meta = [
    "<<meta_all(no_links=True)>>",
    "<<meta_links_all()>>",
]

[needs.layouts.clean_l]
grid = "simple_side_left"

[needs.layouts.clean_l.layout]
head = [
    "<<meta(\"type_name\")>>: **<<meta(\"title\")>>** <<meta_id()>> <<collapse_button(\"meta\", collapsed=\"icon:arrow-down-circle\", visible=\"icon:arrow-right-circle\", initial=False)>> ",
]
meta = [
    "<<meta_all(no_links=True)>>",
    "<<meta_links_all()>>",
]
side = [
    "<<image(\"field:image\", align=\"center\")>>",
]

[needs.layouts.clean_lp]
grid = "simple_side_left_partial"

[needs.layouts.clean_lp.layout]
head = [
    "<<meta(\"type_name\")>>: **<<meta(\"title\")>>** <<meta_id()>> <<collapse_button(\"meta\", collapsed=\"icon:arrow-down-circle\", visible=\"icon:arrow-right-circle\", initial=False)>> ",
]
meta = [
    "<<meta_all(no_links=True)>>",
    "<<meta_links_all()>>",
]
side = [
    "<<image(\"field:image\", align=\"center\")>>",
]

[needs.layouts.clean_r]
grid = "simple_side_right"

[needs.layouts.clean_r.layout]
head = [
    "<<meta(\"type_name\")>>: **<<meta(\"title\")>>** <<meta_id()>> <<collapse_button(\"meta\", collapsed=\"icon:arrow-down-circle\", visible=\"icon:arrow-right-circle\", initial=False)>> ",
]
meta = [
    "<<meta_all(no_links=True)>>",
    "<<meta_links_all()>>",
]
side = [
    "<<image(\"field:image\", align=\"center\")>>",
]

[needs.layouts.clean_rp]
grid = "simple_side_right_partial"

[needs.layouts.clean_rp.layout]
head = [
    "<<meta(\"type_name\")>>: **<<meta(\"title\")>>** <<meta_id()>> <<collapse_button(\"meta\", collapsed=\"icon:arrow-down-circle\", visible=\"icon:arrow-right-circle\", initial=False)>> ",
]
meta = [
    "<<meta_all(no_links=True)>>",
    "<<meta_links_all()>>",
]
side = [
    "<<image(\"field:image\", align=\"center\")>>",
]

[needs.layouts.complete]
grid = "complex"

[needs.layouts.complete.layout]
footer = []
footer_left = [
    "layout: <<meta(\"layout\")>>",
]
footer_right = [
    "style: <<meta(\"style\")>>",
]
head = [
    "<<meta(\"title\")>>",
]
head_left = [
    "<<meta_id()>>",
]
head_right = [
    "<<meta(\"type_name\")>>",
]
meta_left = [
    "<<meta_all(no_links=True, exclude=[\"layout\",\"style\"])>>",
]
meta_right = [
    "<<meta_links_all()>>",
]

[needs.layouts.debug]
grid = "simple"

[needs.layouts.debug.layout]
head = [
    "<<meta_id()>> **<<meta(\"title\")>>**",
    "**<<collapse_button(\"meta\", collapsed=\"Debug view on\", visible=\"Debug view off\", initial=True)>>**",
]
meta = [
    "<<meta_all(exclude=[], defaults=False, show_empty=True)>>",
]

[needs.layouts.focus]
grid = "content"

[needs.layouts.focus.layout]

[needs.layouts.focus_f]
grid = "content_footer"

[needs.layouts.focus_f.layout]
footer = [
    "<<meta_id()>>",
]

[needs.layouts.focus_l]
grid = "content_side_left"

[needs.layouts.focus_l.layout]
side = [
    "<<meta_id()>>",
]

[needs.layouts.focus_r]
grid = "content_side_right"

[needs.layouts.focus_r.layout]
side = [
    "<<meta_id()>>",
]

[needs.layouts.test]
grid = "simple"

[needs.layouts.test.layout]
head = [
    "<<meta(\"type_name\")>>: **<<meta(\"title\")>>** <<meta_id()>>  <<collapse_button(\"meta\", collapsed=\"icon:arrow-down-circle\", visible=\"icon:arrow-right-circle\", initial=False)>> ",
]
meta = [
    "<<meta_all(no_links=True)>>",
    "<<meta_links_all()>>",
]

[needs.render_context]

[needs.schema_definitions."$defs".safe-need]
required = [
    "asil",
]

[needs.schema_definitions."$defs".safe-need.properties.asil]
enum = [
    "A",
    "B",
    "C",
    "D",
]

[needs.schema_definitions."$defs".type-feat.properties.type]
const = "feat"

[[needs.schema_definitions.schemas]]
id = "feat"
idx = 0
severity = "violation"

[needs.schema_definitions.schemas.select.properties.type]
const = "feat"

[needs.schema_definitions.schemas.validate.local]
required = [
    "asil",
]

[needs.schema_definitions.schemas.validate.local.properties.asil]
enum = [
    "A",
    "B",
    "C",
    "D",
]

[needs.services]

[needs.string_links]

[needs.variants]

[needs.warnings]

Ideally the configuration system is mostly based on TOML files. That would require local project configuration to refer to a shared configuration. While sharing configuration is already supported by ubCode / ubc, Sphinx-Needs does not support this yet. It is is a planned feature. The approach shown above therefore relies on generating a full configuration file from configuration given in conf.py locally and shared TOML configuration sources.

General advice

Irrespective of where you store your primary configuration:

If no dynamic configuration is used in conf.py, you should consider migrating over to a declarative configuration completely. It’s also possible to load all static configuration from ubproject.toml and only use conf.py for dynamic parts.