TY  - JOUR
AU  - Nikezić, Dušan P.
AU  - Radivojević, Dušan S.
AU  - Lazović, Ivan
AU  - Mirkov, Nikola S.
AU  - Marković, Zoran J.
PY  - 2024
UR  - https://vinar.vin.bg.ac.rs/handle/123456789/12988
AB  - In order to better predict the high aerosol concentrations associated with air pollution and climate change, a machine learning model was developed using transfer learning and the segmentation process of global satellite images. The main concept of transfer learning lies on convolutional neural networks and works by initializing the already trained model weights to better adapt the weights when the network is trained on a different dataset. The transfer learning technique was tested with the ResNet3D-101 model pre-trained from a 2D ImageNet dataset. This model has performed well for contrail detection to assess climate impact. Aerosol distributions can be monitored via satellite remote sensing. Satellites can monitor some aerosol optical properties like aerosol optical thickness. Aerosol optical thickness snapshots were the input dataset for the model and were obtained from NASA’s Terra-Modis satellite; the output images were segmented by comparing the pixel values with a threshold value of 0.8 for aerosol optical thickness. Hyperparameter optimization finds a tuple of hyperparameters that yields an optimal model that minimizes a predefined loss function on given independent data. The model structure was adjusted in order to improve the performance of the model by applying methods and hyperparameter optimization techniques such as grid search, batch size, threshold, and input length. According to the criteria defined by the authors, the distance domain criterion and time domain criterion, the developed model is capable of generating adequate data and finding patterns in the time domain. As observed from the comparison of relative coefficients for the criteria metrics proposed by the authors, ddc and dtc, the deep learning model based on ConvLSTM layers developed in our previous studies has better performance than the model developed in this study with transfer learning.
T2  - Mathematics
T1  - Transfer Learning with ResNet3D-101 for Global Prediction of High Aerosol Concentrations
VL  - 12
IS  - 6
SP  - 826
DO  - 10.3390/math12060826
ER  - 

@article{
author = "Nikezić, Dušan P. and Radivojević, Dušan S. and Lazović, Ivan and Mirkov, Nikola S. and Marković, Zoran J.",
year = "2024",
abstract = "In order to better predict the high aerosol concentrations associated with air pollution and climate change, a machine learning model was developed using transfer learning and the segmentation process of global satellite images. The main concept of transfer learning lies on convolutional neural networks and works by initializing the already trained model weights to better adapt the weights when the network is trained on a different dataset. The transfer learning technique was tested with the ResNet3D-101 model pre-trained from a 2D ImageNet dataset. This model has performed well for contrail detection to assess climate impact. Aerosol distributions can be monitored via satellite remote sensing. Satellites can monitor some aerosol optical properties like aerosol optical thickness. Aerosol optical thickness snapshots were the input dataset for the model and were obtained from NASA’s Terra-Modis satellite; the output images were segmented by comparing the pixel values with a threshold value of 0.8 for aerosol optical thickness. Hyperparameter optimization finds a tuple of hyperparameters that yields an optimal model that minimizes a predefined loss function on given independent data. The model structure was adjusted in order to improve the performance of the model by applying methods and hyperparameter optimization techniques such as grid search, batch size, threshold, and input length. According to the criteria defined by the authors, the distance domain criterion and time domain criterion, the developed model is capable of generating adequate data and finding patterns in the time domain. As observed from the comparison of relative coefficients for the criteria metrics proposed by the authors, ddc and dtc, the deep learning model based on ConvLSTM layers developed in our previous studies has better performance than the model developed in this study with transfer learning.",
journal = "Mathematics",
title = "Transfer Learning with ResNet3D-101 for Global Prediction of High Aerosol Concentrations",
volume = "12",
number = "6",
pages = "826",
doi = "10.3390/math12060826"
}

Nikezić, D. P., Radivojević, D. S., Lazović, I., Mirkov, N. S.,& Marković, Z. J.. (2024). Transfer Learning with ResNet3D-101 for Global Prediction of High Aerosol Concentrations. in Mathematics, 12(6), 826.
https://doi.org/10.3390/math12060826

Nikezić DP, Radivojević DS, Lazović I, Mirkov NS, Marković ZJ. Transfer Learning with ResNet3D-101 for Global Prediction of High Aerosol Concentrations. in Mathematics. 2024;12(6):826.
doi:10.3390/math12060826 .

Nikezić, Dušan P., Radivojević, Dušan S., Lazović, Ivan, Mirkov, Nikola S., Marković, Zoran J., "Transfer Learning with ResNet3D-101 for Global Prediction of High Aerosol Concentrations" in Mathematics, 12, no. 6 (2024):826,
https://doi.org/10.3390/math12060826 . .

TY  - JOUR
AU  - Radivojević, Dušan S.
AU  - Lazović, Ivan
AU  - Mirkov, Nikola S.
AU  - Ramadani, Uzahir
AU  - Nikezić, Dušan P.
PY  - 2023
UR  - https://vinar.vin.bg.ac.rs/handle/123456789/10871
AB  - The attention mechanism in natural language processing and self-attention mechanism in vision transformers improved many deep learning models. An implementation of the self-attention mechanism with the previously developed ConvLSTM sequence-to-one model was done in order to make a comparative evaluation with statistical testing. First, the new ConvLSTM sequence-to-one model with a self-attention mechanism was developed and then the self-attention layer was removed in order to make comparison. The hyperparameters optimization process was conducted by grid search for integer and string type parameters, and with particle swarm optimization for float type parameters. A cross validation technique was used for better evaluating models with a predefined ratio of train-validation-test subsets. Both models with and without a self-attention layer passed defined evaluation criteria that means that models are able to generate the image of the global aerosol thickness and able to find patterns for changes in the time domain. The model obtained by an ablation study on the self-attention layer achieved better outcomes for Root Mean Square Error and Euclidean Distance in regards to developed ConvLSTM-SA model. As part of the statistical test, a Kruskal–Wallis H Test was done since it was determined that the data did not belong to the normal distribution and the obtained results showed that both models, with and without the SA layer, predict similar images with patterns at the pixel level to the original dataset. However, the model without the SA layer was more similar to the original dataset especially in the time domain at the pixel level. Based on the comparative evaluation with statistical testing, it was concluded that the developed ConvLSTM-SA model better predicts without an SA layer.
T2  - Mathematics
T1  - A Comparative Evaluation of Self-Attention Mechanism with ConvLSTM Model for Global Aerosol Time Series Forecasting
VL  - 11
IS  - 7
SP  - 1744
DO  - 10.3390/math11071744
ER  - 

@article{
author = "Radivojević, Dušan S. and Lazović, Ivan and Mirkov, Nikola S. and Ramadani, Uzahir and Nikezić, Dušan P.",
year = "2023",
abstract = "The attention mechanism in natural language processing and self-attention mechanism in vision transformers improved many deep learning models. An implementation of the self-attention mechanism with the previously developed ConvLSTM sequence-to-one model was done in order to make a comparative evaluation with statistical testing. First, the new ConvLSTM sequence-to-one model with a self-attention mechanism was developed and then the self-attention layer was removed in order to make comparison. The hyperparameters optimization process was conducted by grid search for integer and string type parameters, and with particle swarm optimization for float type parameters. A cross validation technique was used for better evaluating models with a predefined ratio of train-validation-test subsets. Both models with and without a self-attention layer passed defined evaluation criteria that means that models are able to generate the image of the global aerosol thickness and able to find patterns for changes in the time domain. The model obtained by an ablation study on the self-attention layer achieved better outcomes for Root Mean Square Error and Euclidean Distance in regards to developed ConvLSTM-SA model. As part of the statistical test, a Kruskal–Wallis H Test was done since it was determined that the data did not belong to the normal distribution and the obtained results showed that both models, with and without the SA layer, predict similar images with patterns at the pixel level to the original dataset. However, the model without the SA layer was more similar to the original dataset especially in the time domain at the pixel level. Based on the comparative evaluation with statistical testing, it was concluded that the developed ConvLSTM-SA model better predicts without an SA layer.",
journal = "Mathematics",
title = "A Comparative Evaluation of Self-Attention Mechanism with ConvLSTM Model for Global Aerosol Time Series Forecasting",
volume = "11",
number = "7",
pages = "1744",
doi = "10.3390/math11071744"
}

Radivojević, D. S., Lazović, I., Mirkov, N. S., Ramadani, U.,& Nikezić, D. P.. (2023). A Comparative Evaluation of Self-Attention Mechanism with ConvLSTM Model for Global Aerosol Time Series Forecasting. in Mathematics, 11(7), 1744.
https://doi.org/10.3390/math11071744

Radivojević DS, Lazović I, Mirkov NS, Ramadani U, Nikezić DP. A Comparative Evaluation of Self-Attention Mechanism with ConvLSTM Model for Global Aerosol Time Series Forecasting. in Mathematics. 2023;11(7):1744.
doi:10.3390/math11071744 .

Radivojević, Dušan S., Lazović, Ivan, Mirkov, Nikola S., Ramadani, Uzahir, Nikezić, Dušan P., "A Comparative Evaluation of Self-Attention Mechanism with ConvLSTM Model for Global Aerosol Time Series Forecasting" in Mathematics, 11, no. 7 (2023):1744,
https://doi.org/10.3390/math11071744 . .