Core concepts

The following sections provide a more in-depth exploration of the core concepts underlying the toolbox. We delve into the fundamental assumptions and requirements, highlight key considerations to keep in mind, and present general examples to illustrate how to effectively utilize the toolbox in practice.

Datasets

The Congrads toolbox utilizes PyTorch’s Dataset classes, allowing seamless integration with custom data sources. PyTorch’s Dataset class provides a standardized way to represent datasets, making it easy to load, preprocess, and iterate over data efficiently. By implementing custom dataset classes that inherit from torch.utils.data.Dataset, users can define how data is accessed, transformed, and batched. This ensures compatibility with PyTorch’s DataLoader, enabling seamless integration into machine learning pipelines with minimal effort.

We provide two built-in datasets: Bias Correction and Family Income, both featuring automatic downloading and built-in preprocessing transformations. These datasets were initially used to evaluate the feasibility of the Constraint-Guided Gradient Descent (CGGD) technique (see the manuscript for details).

from congrads.datasets import FamilyIncome
from congrads.utils import preprocess_FamilyIncome

data = FamilyIncome("./datasets", preprocess_FamilyIncome, download=True)

The provided function `split_data_loaders` can be used to divide the data into three parts with configurable ratios: for training, validation and testing. The CongradsCore requires these loaders to function. You can also write your own function to divide the data into loaders.

from congrads.utils import split_data_loaders

loaders = split_data_loaders(
    data,
    loader_args={
        "batch_size": 100,
        "shuffle": True,
        "num_workers": 6,
        "prefetch_factor": 2,
    },
    valid_loader_args={"shuffle": False},
    test_loader_args={"shuffle": False},
)

Networks

You are free to define your own neural network (model) however you want. We only require it to extend the PyTorch torch.nn.Module class and it to output a dictionary structure. The network’s output must have the key “output”. Other layers, such as input or intermediary layers, can be named to your liking. When placing constraints on your network, you will use the layer names and an index to refer to specific neurons in the network.

class Model(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        a = F.relu(self.conv1(x))
        b = F.relu(self.conv2(x))
        return {"input": x, "middle": a, "output": b} # Output a dictionary

We also provide a basic Multi-Layer Perceptron (MLP) network using ReLU activations that can be easily configured to match your dataset format.

from congrads.networks import MLPNetwork

network = MLPNetwork(
    n_inputs=6, n_outputs=2, n_hidden_layers=3, hidden_dim=10
)

Descriptor

The descriptor class forms a translation manager that will attach human-readable names to specific points in your neural network. It represents your problem and has the structure of the neural network, while having names that relate to your dataset.

The descriptor.add(layer, index, name, ...) method is used to reference specific components of the neural network. The layer argument identifies a network section based on predefined dictionary keys, while index specifies a particular neuron within that layer. You must assign a name to each neuron (dataset column) that you plan to attach a constraints on so it can be referenced easily.

Note

The neural network receives data in the order it is defined in the dataset. For example, if your dataset has three columns—A (input) and B, C (outputs)—refer to A using index 0 and B, C using indices 0 and 1, respectively.

Certain parts of your network may be non-learnable and should be excluded from the CGGD (Congrads) algorithm. For example, network inputs are always fixed and cannot be optimized. To exclude a specific neuron from the learning process, set its attribute to `constant=True`.

from congrads.descriptor import Descriptor

descriptor = Descriptor()
descriptor.add("input", 0, "Date", constant=True)
descriptor.add("output", 0, "Minimum temperature")
descriptor.add("output", 1, "Maximum temperature")
descriptor.add("output", 2, "Normalized minimum sunshine")
descriptor.add("output", 3, "Normalized maximum sunshine")

Constraints

Constraints are a fundamental part of the Congrads toolbox, allowing you to define relationships between different sections of your dataset—or, in other words, between neurons in your network.

To set up a constraint, the names assigned in the descriptor are used to reference specific data. The relationship between them is defined using a comparator function, which can be one of PyTorch’s built-in comparison functions (`torch.lt`, `torch.gt`, `torch.le`, `torch.ge`).

Additionally, constraints can accept transformations instead of direct descriptor names. This enables you to apply a function or transformation to the data before evaluating whether the constraint is satisfied. This can be useful to undo an operation that was done in preprocessing for example.

Constraint.descriptor = descriptor
Constraint.device = device
constraints = [
    BinaryConstraint("Minimum temperature", le, "Maximum temperature"),
    ScalarConstraint("Normalized minimum sunshine", ge, 0),
    BinaryConstraint(
        DenormalizeMinMax(
            "Normalized minimum sunshine", min=12, max=32
        ),
        le,
        DenormalizeMinMax(
            "Normalized maximum sunshine", min=124, max=324
        ),
    ),
]

The above code translates into the following:

`Minimum temperature` \(\leq\) `Maximum temperature`
`Normalized minimum sunshine` \(\geq\) 0
`Minimum sunshine` \(\leq\) `Maximum sunshine`

Note

Constraints are always placed on data as it is present in the network. If the network outputs normalized predictions, the constraints also have to be formulated as such. A transformation can be used to apply a calculation before processing the constraints, such as a denormalization.

Warning

When placing a constraint between neurons in the network, you must make sure they can be compared sensibly. For example, you cannot directly compare two neurons that were normalized differently (e.g. with a different min/max range). You can use a transformation to first denormalize each neuron, to then make the comparison on original data.

It is possible to implement your own custom constraints to allow for additional complexity in your network. A new constraint must extend the base `Constraint` class and implement the check_constraint(...) calculate_direction(...) methods. Please refer to the built in constraints for examples how to achieve this.

MetricManager

To be able to track the Constraint Satisfaction Ratio (CSR), a score between 0 and 1 that indicates how well the constraint is satisfied, the MetricManager

All metric values can be calculated and retrieved using metric_manager.aggregate(group), after which they can be logged to TensorBoard, a CSV file or other storage methods using the hooks, such as the example below. Metrics having group “during_training” will update every epoch, having group “after_training” are calculated only when the training is finished. The metric_manager.reset(group) method will reset the metric for a specific group so that new values can be stored for a next iteration.

from torch.utils.tensorboard import SummaryWriter
from congrads.metrics import MetricManager
from congrads.utils import CSVLogger

# Initialize metric manager
metric_manager = MetricManager()

# Initialize data loggers
tensorboard_logger = SummaryWriter(log_dir="logs/FamilyIncome")
csv_logger = CSVLogger("logs/FamilyIncome.csv")

def on_epoch_end(epoch: int):
    # Log metric values to TensorBoard and CSV file
    for name, value in metric_manager.aggregate("during_training").items():
        tensorboard_logger.add_scalar(name, value.item(), epoch)
        csv_logger.add_value(name, value.item(), epoch)

    # Write changes to disk
    tensorboard_logger.flush()
    csv_logger.save()

    # Reset metric manager
    metric_manager.reset("during_training")

CongradsCore

The `CongradsCore` integrates all the previously discussed concepts, along with additional configuration settings such as the loss function, optimizer, dataloaders, network architecture, and more. It brings together these elements to form a cohesive framework that ensures smooth execution of the Congrads algorithm, guiding the neural network’s training process while adhering to constraints and other user-defined parameters.

core = CongradsCore(
    descriptor,
    constraints,
    loaders,
    network,
    criterion,
    optimizer,
    metric_manager,
    device,
)

You can then call core.fit(...) to start the training process. Attach functions via the hooks like `on_epoch_end`, `on_train_end` to execute your custom code at certain points in the training process.

core.fit(
    start_epoch=0,
    max_epochs=50,
    on_epoch_end=on_epoch_end,
    on_train_end=on_train_end,
)