Figure 1: Image generation learned from a single training image. We propose SinGAN–a new unconditional generative model trained on a single natural image. Our model learns the image’s patch statistics across multiple scales, using a dedicated multi-scale adversarial training scheme; it can then be used to generate new realistic image samples that preserve the original patch distribution while creating new object configurations and structures.
先行研究と比べてどこがすごい？
Figure 2: Image manipulation. SinGAN can be used in various image manipulation tasks, including: transforming a paint (clipart) into a realistic photo, rearranging and editing objects in the image, harmonizing a new object into an image, image super-resolution and creating an animation from a single input. In all these cases, our model observes only the training image (first row) and is trained in the same manner for all applications, with no architectural changes or further tuning (see Sec. 4).
Figure 3: SinGAN vs. Single Image Texture Generation. Single image models for texture generation [3, 16] are not designed to deal with natural images. Our model can produce realistic image samples that consist of complex textures and non-reptititve global structures.
Figure 4: SinGAN’s multi-scale pipeline. Our model consists of a pyramid of GANs, where both training and inference are done in a coarse-to-fine fashion. At each scale, Gn learns to generate image samples in which all the overlapping patches cannot be distinguished from the patches in the down-sampled training image, xn, by the discriminator Dn; the effective patch size decreases as we go up the pyramid (marked in yellow on the original image for illustration). The input to Gn is a random noise image zn, and the generated image from the previous scale x˜n, upsampled to the current resolution (except for the coarsest level which is purely generative). The generation process at level n involves all generators {GN . . . Gn} and all noise maps {zN , . . . , zn} up to this level. See more details at Sec. 2.
Figure 6: Random image samples. After training SinGAN on a single image, our model can generate realistic random image samples that depict new structures and object configurations, yet preserve the patch distribution of the training image. Because our model is fully convolutional, the generated images may have arbitrary sizes and aspect ratios. Note that our goal is not image retargeting – our image samples are random and optimized to maintain the patch statistics, rather than preserving salient objects. See SM for more results and qualitative comparison to image retargeting methods.
以下は高解像度の生成結果。
Figure 7: High resolution image generation. A random sample produced by our model, trained on the 243 × 1024 image (upper right corner); new global structures as well as fine details are realistically generated. See 4Mpix examples in SM.
Figure 8: Generation from different scales (at inference). We show the effect of starting our hierarchical generation from a given level n. For our full generation scheme (n = N), the input at the coarsest level is random noise. For generation from a finer scale n, we plug in the downsampled original image, xn, as input to that scale. This allows us to control the scale of the generated structures, e.g., we can preserve the shape and pose of the Zebra and only change its stripe texture by starting the generation from n = N −1.
Figure 9: The effect of training with a different number of scales. The number of scales in SinGAN’s architecture strongly influences the results. A model with a small number of scales only captures textures. As the number of scales increases, SinGAN manages to capture larger structures as well as the global arrangement of objects in the scene.
Figure 10: Super-Resolution. When SinGAN is trained on a low resolution image, we are able to super resolve. This is done by iteratively upsampling the image and feeding it to SinGAN’s finest scale generator. As can be seen, SinGAN’s visual quality is better than the SOTA internal methods ZSSR [46] and DIP [51]. It is also better than EDSR [32] and comparable to SRGAN [30], external methods trained on large collections. Corresponding PSNR and NIQE [40] are shown in parentheses.
以下はペイント→写真変換の結果。
Figure 11: Paint-to-Image. We train SinGAN on a target image and inject a downsampled version of the paint into one of the coarse levels at test time. Our generated images preserve the layout and general structure of the clipart while generating realistic texture and fine details that match the training image. Well-known style transfer methods [17, 38] fail in this task.
以下は画像編集の結果。
Figure 12: Editing. We copy and paste a few patches from the original image (a), and input a downsampled version of the edited image (b) to an intermediate level of our model (pretrained on (a)). In the generated image (d), these local edits are translated into coherent and photo-realistic structures. (c) comparison to Photoshop content aware move.
以下は画像調和の結果。
Figure 13: Harmonization. Our model is able to preserve the structure of the pasted object, while adjusting its appearance and texture. The dedicated harmonization
図5 Precision-recall performance for near-duplicate detection using feature distance thresholding on deep ranking features.
図6 : Examples of near-duplicate image pairs that are robustly recognized as near-duplicates by stabilized features (small feature distance), but easily confuse un-stabilized features (large feature distance). Left group: using JPEG-50 compression corruptions. Right group: random cropping CROP-10 corruptions. For each image pair, we display the reference image x, the difference with its corrupted copy x − x 0 , and the distance in feature space D(f(x), f(x 0 )) for the un-stabilized (red) and stabilized features (green).
We hypothesize that with large mini-batches, the variance of the gradients is reduced, which makes NLCG method effective for DNN training.（大規模なミニバッチでは勾配の分散が減少し、それが非線形共役勾配法を深層学習に適用できると仮定した。
Second order methods have been previously used for DNN training … However achieving good convergence results on large scale DNN training problems like ImageNet classification has been difficult.（2次を用いたDNN学習は以前からあった … しかしImageNetを分類するような大規模なDNNの学習で良好な収束結果を達成することは困難だった。）
FIGURE 2 | Sample images from the three different versions of the PlantVillage dataset used in various experimental configurations. (A) Leaf 1 color, (B) Leaf 1 grayscale, (C) Leaf 1 segmented, (D) Leaf 2 color, (E) Leaf 2 gray-scale, (F) Leaf 2 segmented.
FIGURE 3 | Progression of mean F1 score and loss through the training period of 30 epochs across all experiments, grouped by experimental configuration parameters. The intensity of a particular class at any point is proportional to the corresponding uncertainty across all experiments with the particular configurations. (A) Comparison of progression of mean F1 score across all experiments, grouped by deep learning architecture, (B) Comparison of progression of mean F1 score across all experiments, grouped by training mechanism, (C) Comparison of progression of train-loss and test-loss across all experiments, (D) Comparison of progression of mean F1 score across all experiments, grouped by train-test set splits, (E) Comparison of progression of mean F1 score across all experiments, grouped by dataset type. A similar plot of all the observations, as it is, across all the experimental configurations can be found in the Supplementary Material.
もっともよいモデルはGoogLeNet(Transfer learning)のTRAIN : 80%, TEST : 20%、colorの場合であった。平均F1スコアは0.9934だった。
FIGURE4は(A)にような画像が与えられたときのAlexNet(TrainFromScratch, TRAIN : 80%, TEST : 20%, color)の1番最初の畳み込み層を可視化したもの(B)である。
FIGURE 4 | Visualization of activations in the initial layers of an AlexNet architecture demonstrating that the model has learnt to efficiently activate against the diseased spots on the example leaf. (A) Example image of a leaf suffering from Apple Cedar Rust, selected from the top-20 images returned by Bing Image search for the keywords “Apple Cedar Rust Leaves” on April 4th, 2016. Image Reference: Clemson University - USDA Cooperative Extension Slide Series, Bugwood. org. (B) Visualization of activations in the first convolution layer(conv1) of an AlexNet architecture trained using AlexNet:Color:TrainFromScratch:80–20 when doing a forward pass on the image in shown in panel b.
凄い、分かりやすいですね^^
validation accuracy : 0.78291
validation accuracy : 0.814062
validation accuracy : 0.833285
について見ていきます。
はglobal step lengthと呼ばれるもので以下の図のようなふるまいをします。
についてです。
