paddlers/docs/dev/dev_guide_en.md

简体中文 | English

PaddleRS Development Guide

0 Catalog

• Add Remote Sensing Dedicated Models

• Add Data Preprocessing / Data Augmentation Functions or Operators

• Add Remote Sensing Image Processing Tools

1 Add Remote Sensing Dedicated Models

1.1 Write Model Definitions

First, find the subdirectory (package) corresponding to the task in paddlers/rs_models. The mapping between the task and the subdirectory is as follows:

• Change Detection: cd;
• Scene Classification: clas;
• Object Detection: det;
• Image Restoration: res;
• Image Segmentation: seg.

Create a new file in the subdirectory and name it {model name lowercase}.py. Write the complete model definition in the file.

The new model must be a subclass of paddle.nn.Layer. For the tasks of image segmentation, object detection, scene classification, and image restoration, relevant specifications formulated in the development kits PaddleSeg, PaddleDetection,PaddleClas, and PaddleGAN should be followed respectively. For change detection, scene classification and image segmentation tasks, the num_classes argument must be passed in the model construction to specify the number of output categories. For image restoration tasks, the rs_factor argument must be passed in during model construction to specify the super resolution scaling factor (for non-super-resolution models, this argument is set to None). For the change detection task, the model definition should follow the same specifications as the segmentation model, but with the following differences:

• The forward() method accepts three input parameters, namely self, t1, and t2, where t1 and t2 represent the input images of the first and second temporal phases, respectively.
• For a multi-task change detection model (for example, the model outputs both change detection results and building extraction results of two temporal phases), the class attribute USE_MULTITASK_DECODER needs to be specified as True. Also in the OUT_TYPES attribute set the label type for each element in the model forward output. See the definition of ChangeStar model as an example.

Note that if common components exist in a subdirectory (e.g., contents in paddlers/rs_models/cd/layers, paddlers/rs_models/cd/backbones and paddlers/rs_models/seg/layers), they should be reused as much as possible.

1.2 Add Docstrings

You have to add a docstring to the new model, with the references and links to the original paper (you don't have to be strict about the reference format, but consistency between different models of the same task is encouraged). For detailed annotation specifications, please refer to the document. An example is as follows:

"""
The ChangeStar implementation with a FarSeg encoder based on PaddlePaddle.

The original article refers to
    Z. Zheng, et al., "Change is Everywhere: Single-Temporal Supervised Object Change Detection in Remote Sensing Imagery"
    (https://arxiv.org/abs/2108.07002).

Note that this implementation differs from the original code in two aspects:
1. The encoder of the FarSeg model is ResNet50.
2. We use conv-bn-relu instead of conv-relu-bn.

Args:
    num_classes (int): Number of target classes.
    mid_channels (int, optional): Number of channels required by the ChangeMixin module. Default: 256.
    inner_channels (int, optional): Number of filters used in the convolutional layers in the ChangeMixin module.
        Default: 16.
    num_convs (int, optional): Number of convolutional layers used in the ChangeMixin module. Default: 4.
    scale_factor (float, optional): Scaling factor of the output upsampling layer. Default: 4.0.
"""

1.3 Write Trainer Definitions

Please follow these steps:

1. In __init__.py of paddlers/rs_models/{task subdirectories}, add from ... import.

2. Locate the trainer definition file corresponding to the task in the paddlers/tasks directory (for example, the change detection task corresponds to paddlers/tasks/change_detector.py).

3. Appends the new trainer definition to the end of the file. The trainer inherits from the related base class (such as BaseChangeDetector). Override __init__() and other methods according to your needs. The trainer's __init__() method is written with the following requirements:

   • For tasks such as change detection, scene classification, object detection, and image segmentation, the first input parameter of __init__() method is num_classes, which represents the number of model output classes. For the tasks of change detection, scene classification, and image segmentation, the second input parameter is use_mixed_loss, indicating whether to use a mixing loss. The third input parameter is losses, which represents the loss function used in training. For the image restoration task, the first parameter is losses, meaning the same as above; the second parameter is rs_factor, which represents the super resolution scaling factor; the third parameter is min_max, which represents the numeric range of the input and output images.
   • All input parameters of __init__() must have default values, and in the default case, the model receives 3-channel RGB input.
   • In __init__() you need to update the params dictionary, whose key-value pairs will be used as input parameters during model construction.

4. Add the class name of the new trainer to the global variable __all__.

It should be noted that for the image restoration task, the forward and backward of the model are implemented in the trainer definition. For GAN and other models that need to use multiple networks, please refer to the following specifications for the preparation of the trainer:

• Override the build_net() method to maintain all networks using GANAdapter. An GANAdapter object takes two lists as input when it is constructed: The first list contains all generators, where the first element is the main generator; the second list contains all discriminators.
• Override the default_loss() method to build the loss function. If more than one loss function is required in the training process, it is recommended to organize in the form of a dictionary.
• Override the default_optimizer() method to build one or more optimizers. When build_net() returns a value of type GANAdapter, parameters is a dictionary, where parameters['params_g'] is a list containing the state dicts of the various generators in order; parameters['params_d'] is a list that contains the state dicts of the individual discriminators in order. If you build more than one optimizer, you should use the OptimizerAdapter wrapper on return.
• Override the run_gan() method that accepts four parameters: net, inputs, mode, and gan_mode, for one of the subtasks in the training process, e.g. forward of generator, forward of discriminator, etc.
• Override train_step() method to define how a single training step goes. Usually, in a training step, we call run_gan() multiple times with different inputsand gan_mode, extract useful fields (e.g. losses) from the outputs dictionary returned each time, and summarize them into the final result.

See ESRGAN for a specific example of GAN trainers.

2 Add Data Preprocessing / Data Augmentation Functions or Operators

2.1 Add Data Preprocessing / Data Augmentation Functions

Define new function in paddlers/transforms/functions.py. If the function needs to be exposed and made available to users, you must add a docstring to it.

2.2 Add Data Preprocessing / Data Augmentation Operators

Define new operators in paddlers/transforms/operators.py. All operators inherit from paddlers.transforms.Transform. The operator's apply() method receives a dictionary sample as input, fetches objects stored in it, makes in-place modifications to the dictionary after processing, and finally returns the modified dictionary. Only in rare cases do you need to override the apply() method when defining an operator. In most cases, you just need to override the apply_im(), apply_mask(), apply_bbox(), and apply_segm() methods to handle the images, segmentation labels, bounding boxes, and target polygons, respectively.

If the operator has a complicated implementation, it is recommended to define functions in paddlers/transforms/functions.py and call them in apply*() of operators.

After writing the implementation of the operator, you must write docstring and add the class name in __all__.

3 Add Remote Sensing Image Processing Tools

Remote sensing image processing tools are stored in the tools/ directory. Each tool should be a relatively independent script, independent of the contents in the paddlers/ directory, which can be executed by the user without installing PaddleRS.

When writing the script, use the Python standard library argparse to process the command-line arguments. Also, we suggest using the if __name__ == '__main__': code block. If you have multiple tools that use the same function or class, please define these common components in tools/utils.